Bafana Bafana goalkeeper Jackson Mabokgwane saved a penalty in the marathon shootout, but teammates Carl Lark and Kwanda Mngonyama were unsuccessful with their kicks, as was Gift Moyo for Peter Butler's outfit.
Shakes Mashaba's men will now face either Malawi or Mozambique in the Plate Section of the competition on Wednesday, while the Zebras joined Namibia in the next round following the Brave Warriors' defeat of holders Zambia.
West African guests Ghana play Madagascar and Malawi tackle Mozambique in the other last-eight fixtures on Monday.
Bafana dictated the pace early on, but were unable to make their dominance count against their neighbours, who had the first chance of the game in the 26th minute.
Mabokgwane dropped a cross at Omaatla Kebatho's feet, however, the Zebras man volleyed wide from the edge of the area.
At the other end, NFD top scorer Phumelele Bhengu came close in the 40th minute as he narrowly failed to get on the end of Marc van Heerden's dangerous cross.
Mabokgwane, who wore the armband, then came to Bafana's rescue when he kept out Lemponye Tshireletso's low drive from 20-yards out on the stroke of half-time.
Bafana threatened to break the deadlock in the latter stages, as Van Heerden's curling free-kick triggered a save from Platinum Stars' Kabelo Dambe in the 66th minute.
Soon after Siyabonga Nhlapo fired wide on the turn before Lark headed a Van Heerden set-piece over in the 80th minute, with the tie ultimately being decided from the spot in the visitors' favour.
Nato scores, Lark misses, Kgaswane scores, Daniels scores, Mogorosi scores, Van Heerden scores, Moyo misses, Motupa scores, Dambe scores, Morena scores, Legwaila scores, Segolela scores, Koobake scores, Modjeka scores, Gaolaolwe scores, Mngonyama misses.
Bafana XI: Mabokgwane (c), Nhlapo, Mngonyama, Daniels, Van Heerden, Lark, T. Sangweni (Madisha 87'), Morena, Segolela, Motupa, Bhengu (Nyanda 71').
Botswana XI: Dambe, Gadibolae, Gaolaolwe, Ncenga, Koobake, Legwaila, Nato, Boy, Kgaswane, Kebatho, Tshireletso.