Bafana defeated the Super Eagles for the first time in a competitive game with a 2-0 scoreline in Uyo in June 2017, which was also Baxter's first game on his return to the national team hotseat following the sacking of Shakes Mashaba.
Baxter's men next host Libya at Durban's Moses Mabhida Stadium on Saturday (15h00).
The Mediterranean Knights top Group E on goal difference thanks to their 5-1 rout of Seychelles.
"This is massive game for us, it's a massive game in many ways," Baxter said.
"We have to follow up the good result against Nigeria. It pales into insignificance if we don't get a good result in this game.
"So therefore it is very important to continue to take points, because if we continue to take points it may be done and dusted before the end of the group.
"So we do want that. We don't want to go all the way down to the wire.
"This is 90 minutes plus; it's in South Africa. We want to make sure we take control of it. If we can take control of this game we go a long way to taking control of the group."