Former Celtic players combined for Chiefs' opener as Keagan Buchanan's corner was converted by Erick Mathoho inside the final 10 minutes of the first half.
Parity was restored in the second half as Lorenzo Gordinho conceded a penalty and Atusaye Nyondo slotted home.
The home side enjoyed more time on the ball and better chances in the opening exchanges and should have gone ahead just before the 10-minute mark but Alfred Ndengane skied his effort from close range.
Both Buchanan and Celtic's Lucky Baloyi attempted efforts from range but the former's effort was terribly tame and the latter's wastefully high.
Twenty minutes into proceedings Deon Hotto's shot fell to Kabelo Mahlasela but the tricky midfielder couldn't fire off a shot and Amakhosi were able to scramble clear.
Celtic were completely in charge of the game at the midway point of the half but stuggled to find any real penetration.
Inside the closing quarter-hour of the stanza Chiefs gained a foothold in the match and eventually capitalised in the 37th minute.
A Buchanan corner was well met by Mathoho, powering his header past a helpless Patrick Tignyemb.
Coming out of the break Celtic almost levelled matters early but Roggert Nyundu's long-range effort was kept out by a diving Brilliant Khuzwayo.
Ndumiso Mabena, having wasted a free-kick in a promising area earlier in the second stanza, blasted wide as the game moved into the final 20 minutes.
Celtic eventually found their breakthrough in the 75th minute as Nyondo converted from the spot after Gordinho handled in the area.
In the 82nd minute Siphiwe Tshabalala attempted to restore his side's advantage but fired wide from a free-kick 20 yards out from goal.
Bloemfontein Celtic: Tignyemb, Letlabika, Rikhotso, Ndengane, Phalane, Motshegwa, Mahlasela (Masuku 83'), Baloyi (Mashinkinya 73'), Hotto, Nyundu (Nyondo 67'), Mabena
Kaizer Chiefs: Khuzwayo, Mphahlele, Gordinho, Mathoho (Cardoso 69'), Masilela, Buchanan, Katsande, Ekstein (Twala 77'), Tshabalala (Chirambadare 82'), Parker, Páez