The match marked the debut of new Phunya Sele Sele coach Ernst Middendorp, and he looked on course for victory when Lerato Manzini put the hosts ahead, but the Birds fired back twice through veteran attacker 'Bhele' to claim the lead.
However, a late own goal from Roger da Costa salvaged a 2-2 draw for the hosts and ensured the German coach’s tenure did not begin with a loss.
Celtic threatened first, with youngster Keagan Buchanan shooting on target in the 13th minute, but Greg Etafia was able to punch the effort away from danger.
Etafia was called upon again in the 40th minute, parrying away a shot from Botswana international Joel Mogorosi as Phunya Sele Sele looked the likelier scorers.
And three minutes later the hosts did open the scoring when Manzini headed a cross beyond the reach of Etafia to make it 1-0.
Celtic would have been hoping to hold their lead until half-time, but just before the break the Birds levelled the game, when Luvhengo Mungomeni set up Nomvete, who finished the chance with a right-footed shot.
Four minutes into the second half Edward Manqele threatened to put Swallows ahead, but his shot was well saved by Patrick Tignyemb.
A few minutes later Manzini was upended in the 18-yard area by Etafia, and the referee awarded Celtic a penalty. Mogorosi stepped up to take the spot-kick, but saw his effort saved by Etafia.
Swallows threatened again in the 75th minute, with Giorgi Nergadze getting on the end of Lerato Chabangu’s corner-kick, but he was unable to direct his header on target.
Just over 10 minutes later Chabangu was in the thick of things again, when he shot wide of Tignyemb’s right-hand post.
The visitors continued to build up the pressure, and in the 89th minute Nomvete struck again, heading home from Chabangu’s cross to make it 2-1.
But just when it appeared Celtic would suffer another home defeat, Da Costa put the ball into his own net in injury time.