Los Blancos manager Zinedine Zidane made wholesale changes, with Sergio Ramos out injured, Raphael Varane and Luka Modric rested and the likes of goalkeeper Thibaut Courtois, Karim Benzema, Toni Kroos and Marco Asensio starting from the bench.
Federico Valverde had the first chance for Madrid in the first quarter of an hour with a powerful low drive towards the bottom corner but shot-stopper Jose Juan Figueras made a great save.
Eder Militao finally broke the deadlock for Madrid on the stroke of half-time with a powerful header inside the box after a teasing cross by Marcelo.
Lucas Vazquez came close to doubling Los Blancos' lead ten minutes inside the second stanza with a powerful drive from range that produced an acrobatic save from Figueras.
Los Blancos were struggling to get their rhythm going forward after that and they finally brought on Benzema in the 67th minute in place of Mariano Diaz.
It was the hosts who got the next goal as Jose Solbes snatched an equaliser in the last ten minutes with a tap-in at the back post after a flick on by Ramon Lopez.
Vazquez came very close to netting the winner for Madrid in the last four minutes with a header towards the bottom corner from Marcelo's cross but Figueras made an outstanding save and the sides went to extra-time after a 1-1 draw in regulation time.
Zidane's men took time to get going in the early stages of extra-time and Marco Asensio came close to restoring their lead just before the first period of extra-time ended with a low curling shot that was saved by Figueras.
Alcoyano were reduced to ten men with ten minutes left of the first stanza after Ramon Lopez got a second booking for a lunging tackle on Casemiro and was sent off.
However, Juanan Casanova snatched the winner for the hosts in the last five minutes with a strike at the near post after a quick counter-attack, as Alcoyano booked their spot in the last 16 with a famous victory over Madrid.