The Birds had opened the scoring through Georgi Nergadze on the half-hour mark.
However, an own-goal from Phumelele Bhengu and then a winner from Prince Hlela consigned Swallows to yet another defeat.
The home side started a little slowly and looked like a side struggling for form. Celtic, on the other hand, passed the ball around nicely in the opening stages and created some decent chances.
In the 18th minute only a fine finger-tip save from Greg Etafia denied Celtic the lead. Bennett Chenene picked up the ball about 22 yards from goal and unleashed a powerful shot that saw the keeper at full stretch to tip the effort around the post.
Celtic continued to control the half after that as they had the lion’s share of possession, but in the 31st minute they still found themselves a goal behind and somewhat against the run of play.
Nothing, however, can be taken away from the goal that is sure to end up as one of the goals of the 2010/11 PSL season. Nergadze pounced on a loose ball just outside the 18-yard box and produced a wonder goal with a thunderbolt shot that found the top right-hand corner of the net.
Celtic almost found the equaliser two minutes later, but another fine save from Etafia kept his side in the lead. The Swallows keeper did brilliantly diving low to his right to tip a deflected Hlela free-kick around the post for a corner.
From the resulting corner Lefa Tsutsulupa was forced to clear a Luyanda Bacela header off the goal line.
The second half started a little sloppy as neither side managed to find their rhythm from the first stanza and in the 66th minute Swallows were made to pay for some poor defending.
In an almost comical manner Bhengu scored one of the most bizarre own-goals which drew Celtic level. Cecil Oerson attacked down the right flank and whipped in a low cross which saw Bhengu fall over his own feet and then accidentally handle the ball into his own net.
Eleven minutes later and Swallows’ season got even worse as they went a goal behind. Again some bad defending saw them concede the goal when they failed to clear a long throw-in into the box. Hlela pounced on the loose ball and fired his side in front from about eight yards out.
That is the way the game ended as Celtic picked up their first away win of the league season and Swallows remain at the bottom of the log, yet to win a match.