Ninety minutes produced a 1-1 draw, with Amakhosi veteran Josta Dladla cancelling out Lerato Lamola's late opener at Free State Stadium.
Chiefs had the first attempt on goal, as Siyabonga Nkosi fired wide in the sixth minute, while Dumisani Zuma rattled an effort narrowly off target at the other end soon after.
Nkosi threatened again midway through the stanza, as his volley from the edge of the area brushed the top of the cross bar before his teammate Siphelele Mthembu dragged a shot wide.
Ovidy Karuru came close to breaking the deadlock in the 27th minute but the Chiefs newcomer's 30-yard strike was well kept out by Celtic goalkeeper Patrick Tignyemb, who made a neat save shortly before the break to foil a low shot from Nkosi.
Lamola had a great chance to score in the 50th minute but he was unable to get the better of Brilliant Khuzwayo in a one-on-one situation.
Karuru then looked to have scored on the hour when he turned in a right-wing cross but the goal was disallowed for off-side.
Lamola eventually got the breakthrough in the 73rd minute with a diving header but Dladla equalised from the spot a minute later after Alfred Ndengane was adjudged to have handled in the area.
The hosts, though, went on to pip the Absa Premiership leaders in the shoot-out for a much-needed morale-boosting win
Bloemfontein Celtic: Tignyemb, Nkausa, Fransman, Ndengane, Ritchie (Cloete 46’), Isaacks, Mulenga, Mogapi (Lakay 46’), Ntanga (Motshegwa 46), Zuma (Bilankulu 46’), Lamola.
Kaizer Chiefs: Pieterse (Khuzwayo 46’), Moleko, Gould, Gordinho, Mokgothu (Mtsweni 64’), Bukenya, Nkosi (Matombo 85’), Mngadi (Ekstein 63’), Rusike, Karuru (Dladla 63’), Mthembu.