Pirates, who trail leaders Mamelodi Sundowns by one point following their 2-1 victory over Golden Arrows prior to the international break, were looking to continue from where they left off.
They had a promising start with Thembinkosi Lorch going close from a free-kick, which forced Patrick Tignyemb to a save with Justin Shonga unable to bury the rebound.
Celtic opened them up at the other end following a counterattack, which forced the Bucs into desperate defending.
Following a spate of half-chances at both ends, the opener eventually came and it was the hosts who made it count as Musa Nyatama rose highest in between a cluster of defenders to head home following a corner kick on 17 minutes.
It was the second time he has come back to haunt his former club after grabbing the winner in the corresponding fixture this season.
The lead put the ascendancy firmly in favour of the hosts, and it was Lorch who threatened with another shot from range but Tignyemb had it safely guarded on that occasion.
The scores were leveled at the start of the second half with Lorenzo Gordinho propping up at the far post to poke the ball home with the opposing defence in sixes and sevens.
The closest Celtic came to going in front had Ndumiso Mabena breaking through the box where he unleashed a close-range shot that forced Siyabonga Mpontshane to an outstretched save.
The visitors were dealt a big blow after the man who found the equaliser, Gordinho, saw red for a professional foul on Luvuyo Memela.
That jeopardised their chances of a come-from-behind victory, and it came as no surprise when the hosts found the winner through a scorching strike from substitute Augustine Mulenga.