The hosts held most of the possession and created more scoring chances, but it was Celtic who had the last word through a late goal from Thabo Nthethe.
Ruzaigh Gamildien continued to trouble the Arrows defence and in the 19th minute he latched onto a through-pass on the right, but again he was denied by goalkeeper Pavels Steinbors, who pushed the ball wide of the target.
Then it was the home team who started asking the questions with Bonginkosi Ntuli firing against the side netting in the 29th minute, while Bilankulu flashed his shot wide of the target two minutes later with the Celtic goalkeeper out of position.
In the second half Arrows had numerous chances to take the lead, but they lacked composure in front of goal.
Tignyemb also had a good game between the posts and in the 50th minute he blocked a low shot from Bilankulu and recovered quickly enough to deflect a secondary strike from Gabuza.
Twenty minutes later Ntuli broke through on the right before finding substitute Nkanyiso Mlotshwa in the centre of the box, but he rushed his shot over the target.
Lerato Manzini almost stole the lead for the visitors against the run of play in the 86th minute, but he could only direct his shot wide of the upright, while Gabuza shot straight at the goalkeeper at the other end moments later.
However, the home side were left stunned in the final minute when Nthethe nudged the ball over the line after the Arrows goalkeeper had failed to secure the ball from a corner kick.