Rantie scored two brilliant individual first-half goals to put Shakes Mashaba's side in the driving seat, but Super Eagles substitute Sone Aluko pulled two goals back for the home side in the second half.
It would have been Bafana Bafana's first ever competitive win over the Super Eagles when the score was 2-1 in stoppage time, but Aluko's late strike denied the visitors at the death.
READ: Nigeria 2-2 SA minute by minute
The result sees South Africa end on top of the group on 12 points, and at the same time sent Nigeria, who needed three points, crashing out of Afcon.
Bafana ended the game a man down after Reneilwe Letsholonyane – playing his 51st cap for the national team – was sent off for his second bookable offence in the 71st minute.
It looked ominous for South Africa after just eight minutes when Dean Furman was caught in possession, allowing Ahmed Musa through on goal as he thundered a strike against the crossbar.
Then just minutes later goalkeeper Darren Keet had a major hand to play as he showed composure and experience to come out and thwart Musa in two dangerous one-on-one situations in quick succession, blocking with his legs on both occasions.
It was an open contest, although Nigeria were dominating possession. Six minutes before half-time Mashaba decided to withdraw Kermit Erasmus – who was a late change to the starting eleven – and brought on more traditional centre-forward Moeketsi Sekola for his second international cap.
Rantie then came up with a moment of magic just before the break, receiving the ball from Furman on the left touchline before outsprinting two Nigerian defenders, opening up his body and disguising a shot past Victor Enyeama at his near post. It was a quality finish from the Bournemouth forward to put his side into the ascendency.
Then just four minutes after the break the ever hard-working Rantie was at it again, stealing the ball of Nigeria's last defender before rounding the goalkeeper and slotting home on his left foot.
Aluko pulled a goal back on 68 minutes after Nwofor Uche was allowed too much space to turn and get a shot away on the edge of the area, the ball cannoning off the upright and falling fortunately for the substitute to tap home.
Bafana were still in control at this point but the game changed on 71 minutes when Letsholonyane – booked in the first half for a studs-up challenge on Raheem Lawal – was sent for an early shower after a cynical trip that resulted in his second caution.
It was all Nigeria in the final 20 minutes and although they were not creating too many clear-cut chances, they pushed right until the 92nd minute when Aluko dummied Furman on the edge of the box before trying his luck with his weaker right foot, beating Keet with a daisy-cutter into the bottom-right corner.
The result was disappointing for South Africa, but even more so for Nigeria as they failed to qualify for Equatorial Guinea 2015.