The home side were without coach Kosta Papic, who was still awaiting his work permit - a problem which also meant Geoffry Massa and Helder Pelembe couldn't feature for Celtic.
City took the lead after just seven minutes through Jabulani Maluleke, who sidefooted home despite a brilliant attempt by goalkeeper Patrick Tignyemb, with the assistant referee ruling the ball had crossed the line.
The hosts kept on pressing, but Celtic gradually came into the game and were gifted a goal on 35 minutes when Nazier Jacobs' backpass was intercepted and Thapelo Morena slotted home across debutant Pule Konopi.
However, the goalie was brilliant a few minutes later when he rushed out to save at Morena's feet.
The game swung from end to end after the break, but Celtic eventually took the lead on 74 minutes through Lerato Lamola, who fired into the roof of the net after an excellent flick-on from Musa Nyatama.
However, City were level a minute later through substitute Linda Shiba, who finished from close range, before Thobani Mncwango got the winner eight minutes from time after expertly controlling a perfectly timed through-ball from Man of the Match Maluleke.