A Marawaan Bantam penalty and Clifford Mulenga goal put Wits into a commanding 2-0 lead, but a late Petrus Mahlatsi strike brought Celtic back into it with just minutes remaining on the clock.
Bantam was brought down in the penalty-area by Ditheko Mototo on 16 minutes and the Wits midfielder picked himself up to convert from the penalty-spot.
Celtic were creating more chances, but lacked a cutting edge in front of goal, leading to coach Mich D’Avray to make two changes at the break.
Off went Mark Mayambela and Moses Spandeel, to be replaced by Shaun Bartlett and Mahlatsi respectively.
But Wits grabbed a second when Fabricio Rodrigues found Mulenga in the box and the young Zambian showed good composure to fire home.
Celtic huffed and puffed their way through the second half, but looked like they would draw a blank, before Mahlatsi showed his predatory instincts on 89 minutes after being set up by Bartlett.
The home side poured forward in search of an equaliser, but Wits managed to hold firm and move up to third in the table. Celtic finish the weekend bottom, with relegation rivals Free State Stars and Bay United each picking up points.