The sides were deadlocked at 1-1 going into added time before Tshepo Rikhotso found the winner with literally only seconds remaining.
Earlier, Oupa Manyisa scored for the first time in almost two years to give the home side an 11th minute lead that was cancelled out in the second period as Lantshene Phalane scored from inside his own half.
Both those goals did, however, come as a result of truly awful mistakes - on both occasions the respective defence and goalkeeper combinations of the sides in action colluding to gift their opposition a goal.
Inside the opening ten minutes of the game Manyisa struck the crossbar with a free-kick out wide but the whistle blew for offside with the linesman deeming Ndoro to have been impeding Tignyemb.
Not long afterwards Shikweni sent Mxoyana's cross wide as he lunged forward to get his head on it and his side were made to pay as Pirates went straight up the other end to score.
A route one ball out from the back should have been dealt with easily but Letlabika, in his attempt to shield it back to Tignyemb, collided with his 'keeper and allowed Manyisa to bounce on the loose ball and roll it into an unmanned net.
Celtic attempted to hit back immediately but Masuku was denied by Mabokgwane who was forced to have words with a defence that opened up far too easily.
There were more chances for Siwelele to come but again they failed to convert, this time an unmarked Rikhotso powered his header over from a corner.
At the other end another ball into the box from Pirates was cushioned down brilliantly by Ndoro into the path of Manyisa but the captain was too casual with his attempt and Tignyemb turned the ball onto the upright.
The visitors continued to create decent chances and Mabaso, having been caught offside earlier, perhaps believed he had strayed once again when he broke the line and struck the ball with very little conviction or accuracy.
Towards the end of the half Ndengane was lucky to escape with only a yellow card after he frustratedly swung his leg in the direction of Makudubela and sent the slight winger crashing to the turf.
Early in the secon-half Pirates won a free-kick in a promising area and Pirates staged the set-piece cleverly before Matlaba was set up to fire his effort wide.
Heading towards the hour-mark Celtic had to content themselves with a shot from range via the boot of half-time substitute Dlamini but Mabokgwane was not troubled.
Pirates were able to put plenty of pressure on Celtic soon afterwards through rather curious circumstances - all of which stemmed from Tignyemb mistakes.
First the Cameroonian stopper illustrated a poor understanding of the rules as he scrambled to keep the ball in with his feet after it had come off a teammate but was clearly a miscued clearance and not a deliberate back pass.
With the ball having gone out for a corner Tignyemb plucked the resultant set-piece out of the air with ease but the inexplicably left the area before booting down field and afforded Pirates a free-kick opportunity just outside the area that was crashed into the wall much to the visitors' relief.
At the other end Hotto had a crack from range but his goal-bound effort was beaten away by Mabokgwane whilst back in front of the Celtic goal Gyimah had the ball in the back of the net with an acrobatic strike but was miles offside.
Entering the final 20 minutes of the match Nyatama should have had the equaliser but curled wide of the post after Hotto put it on a platter for him in a great central position inside the area.
The leveller was forthcoming not long afterwards as another howler from a goalkeeper resulted in a goal - this time Mabokgwane letting Phalane's innocuous long ball go over his head, perhaps anticiapting that the referee would blow his whistle for what he may have believed was Dlamini infringing on Gcaba.
There was to be more late drama in stoppage time as a free-kick was whipped in low and Rikhotso threw his body at the ball which found its way into the back of the net to hand Celtic the win.