Arrows' new signing Knox Mutizwa starred for the Durban-based club as his second half goal sealed victory for the hosts.
The two sides went into the half-time break locked at 0-0.
Three minutes after the restart, Zimbabwean attacker Kudakwashi Mahachi was brought down inside the box, prompting the referee to point to the spot.
Nkanyiso Cele then stepped up to slot the spot-kick beyond the reach of Celtic goalkeeper Ayanda Mtshali in the 54th minute.
Phunya Sele Sele then surged forward in search of an immediate response, and they got just that as Mpho Maruping netted from a corner in the 61st minute.
However, Mutizwa then stepped up to put his side back on the front foot with a well-taken header four minutes later.
Abafana Bes'thende, who had January recruit Chiukepo Msowoya sent off in the second half after receiving two cautions, held firmly onto their lead to see out a welcome victory following their humiliating defeat in midweek.
The result sees Arrows back up to sixth place, with 23 points from 17 games, while Celtic remain in 13th position, with the bottom three teams still to play this weekend.