In a thrilling game that saw three superb goals, fast-paced football and numerous scoring chances, the visitors will know that they should have headed back to Bloemfontein with a share of the points after Itumeleng Khune saved a last-gasp penalty by Geofrey Serunkuma.
The Chiefs goals were scored by Siphiwe Tshabalala and Kaizer Motaung Jnr while Mark Mayambela scored for Celtic.
The match got off to a cracking start, with two terrific goals within the first 15 minutes. Ironically, the first scoring opportunity came from Chiefs 'keeper Itumeleng Khune, as his long kick was allowed to bounce in the Celtic penalty area and Patrick Tignyemb was forced to make a good save, tipping the ball over his crossbar for a corner kick.
Mark Mayambela almost opened the scoring in the sixth minute, but his goal-bound shot was deflected onto the post by Khune. Instead it was Amakhosi who scored three minutes later, Siphiwe Tshabalala driving the ball low into the bottom corner after an excellent defence-splitting pass from Jose Torrealba.
Chiefs' lead lasted just four minutes before Mayambela rifled home a superb free kick to make it 1-1. The left footer struck the ball sweetly over the defensive wall and bulged the back of the net despite Khune making an acrobatic attempt at a save.
The sides then went on to trade scoring opportunities in a high-tempo, entertaining manner that had the crowd buzzing. Coaches Muhsin Ertugral and Owen da Gama were probably dismayed at the ease with which their defences were being stretched, but it made for compelling, end-to-end football.
Tshabalala curled a free kick just wide on 18 minutes, and then Petrus Mahlatsi powered a shot from an acute angle straight into the arms of Khune. Midway through the first half Abia Nale should have scored his first goal for the Soweto side after working some space in the penalty area, but his powerful shot was well saved by Tignyemb.
In the 26th minute Torrealba came within inches of putting the hosts back in the lead with a scorching drive that had the Celtic 'keeper beaten all ends up, but the ball thundered back off the upright.
The Venezuelan's near-miss seemed to galvanise the two defences and there were no further creditable scoring chances until the first minute of the second half when Celtic's Gert Schalkwyk – on loan at the club from Chiefs – shot wide of goal with only Khune to beat.
In the 50th minute Schalkwyk was desperately unlucky not to score his fifth goal for Phunya Sele Sele, hitting the base of the post with a low drive from the edge of the 18-yard area.
The visitors then needed Tignyemb to make a sharp low save to keep out Torrealba's downward header after a good overlapping run and cross from left-back Onismor Bhasera.
In the 58th minute Celtic were left to rue their bad luck, as Mahlatsi hit the woodwork after meeting Mayambela's teasing cross with a powerful header, and Chiefs almost immediately broke up the other end of the field and scored a well-worked goal to restore their lead.
Torrealba carried the ball forward with a strong run, played a low cross into the box which was neatly helped on by David Mathebula before Kaizer Motaung Jnr lifted it over the diving Tignyemb to make it 2-1.
On 67 minutes Mathebula drilled a powerful shot mere inches over the crossbar, and a short while later he set up Motaung Jr for a shooting opportunity but the striker blazed his effort well wide.
By this stage Chiefs were well in control of the game, though they did have a scare with 15 minutes to play when Schalkwyk wriggled his way to the byline before squaring for Tlou Segolela, but the substitute seemed not to be expecting the ball and he could only manage to poke it out for a goal kick.
Josta Dladla almost scored his first for Amakhosi in the 88th minute, running onto a bouncing ball with only Tignyemb to beat, but the 'keeper did enough to deflect the ball behind for a corner.
In the final minute of regulation time Celtic were gifted a way back into the game when Tinashe Nengomasha gave away a debatable penalty for handling the ball in the box, but Khune saved Geofrey Serunkuma's spot kick to ensure his team went back to Naturena with all three points.
Celtic remain second from bottom on 18 points, while Chiefs have moved to second on 33 points, four adrift of leaders SuperSport United.