Second-half goals from substitutes Marouane Fellaini and Dries Mertens gave Belgium the win after Sofiane Feghouli scored with a penalty in the first-half.
Belgium were favourites going into the match but Algeria looked far the brighter of the two.
Les Fennecs got their first big chance on 18-minutes when Riyad Mahrez sliced wide from an angle.
The Algerians continued to look the more determined and on 25 minutes took the lead.
The goal came via a penalty kick, scored by Feghouli, after he was pulled back by Jan Vetonghen to be awarded the spot-kick by the referee. He shot low to the left of goalkeeper Thibaut Courtois, 1-0.
Belgium were lacking in both penetration and ideas in the first-half. Alex Witsel has two good shots coming close for Belgium, crashing in 30-yard shots that on each occasion blocked by keeper Rais Mbolhi.
In the second half Algeria remained a threat and very nearly doubled their lead on 57 minutes when Carl Medjani's header flashed just wide of the post.
Belgium introduced Mertens, Fellaini and Divock Origi in the second half and that would prove the difference.
Fellaini scored with a looping a header in the 70th minute from Kevin de Bruyne's in-swinging cross, 1-1.
With 20 minutes to play and with Belgium tails up, it was no surprise when they went on to snatch a winner with ten minutes remaining.
The goal came in the 80th minute with Mertens racing through on to a measured through-ball from Eden Hazard and smashing an unstoppable right-foot shot into the roof of the Algerian net to make the final score 2-1.