The first chance of the match fell for Mame Niang in the second minute but his header produced a comfortable low save from Patrick Tignyemb.
However, Niang got it right a few minutes later, when he headed the ball into an open net, after Tignyemb had been forced to keep out a stinging shot from Aubrey Ngoma.
The Senegalese striker came closing to scoring a second just three minutes later when he arrived late to meet a cross from Ramahlwe Mphahlele, but his shot went wide of the near post.
The hosts then came close to leveling matters in the 39th minute when Joel Mogorosi was allowed a free header inside the box, but his effort went inches over the crossbar.
However, the equaliser did arrive four minutes later from a defensive error, as Celtic were awarded an indirect free-kick following Washington Arubi's failure to release the ball within six seconds.
The goalkeeper managed to block the initial shot from Auguston Leonard, but Madubanya was there to nudge the ball over the line and into the net.
The home side threatened to take the lead in the 58th minute when Madubanya found Lyle Lakay at the far post, but his first-time volley was kept out by Arubi.
AmaTuks then managed to carve Celtic open in the 71st minute, when Ngoma released Zungu on the right side of the box, who calmly tucked the ball past the goalkeeper.
Celtic had a few opportunities to claw back a point, but Arubi made up for his earlier blunder with two good saves to deny a header from Bacela in the 80th minute, and another from Madubanya in added time.