Joint top at the start of the match, Kilmarnock took the lead when Fraser Wright scored after some indecisive goalkeeping by Allan McGregor.
It took Rangers until nearly the hour mark to equalise when Kris Boyd scored from the penalty spot.
The much-criticised striker then scored the winner four minutes later with a crisp shot past Alan Combe.
It was a nervous final few minutes for Rangers when they were reduced to ten men after Sasa Papac was sent off for a second booking.
The visitors took an early lead as Garry Hay lobbed the ball back towards goal and McGregor seemed rooted to the spot as he allowed the ball to bounce and then Wright to out-jump him before the centre-back poked home the loose ball.
Kilmarnock were enjoying hitting on the break with the Mehdi Taouil and David Fernandez causing the home defence problems.
Kenny Miller fired over a great cross but Boyd was slow to react and then Miller skipped past the defenders before finding Boyd, whose shot was well blocked by Combe.
Rangers sent on Nacho Novo for the ineffectual DaMarcus Beasley at the start of the second half and it was the Spaniard who started the fightback for the home team when he was pulled down by Manuel Pascali in the box.
It was a clear penalty and Boyd made no mistake.
Steven Davis set up the winner for Rangers but his clever pass reached Boyd who turned and fired his shot home from eight yards.
Rangers faced several anxious moments before the end. Garry Hay's pass found Craig Bryson whose shot was blocked before Danny Invincibile mis-hit the rebound.
With Kilmarnock pushing forward, Rangers should have scored the third on the break but Combe produced marvellous saves to deny Miller, Novo and Boyd.