Khedira penned a four-year deal in Turin last week after leaving Real on a free transfer.
The 28-year-old won seven trophies in his five seasons at the Santiago Bernabeu.
“It hurt me when people said that,” Khedira told Welt am Sonntag. “Real Madrid offered me a huge contract - one in which I could have eventually earned more. My decision to leave was not about money.
“You cannot accuse me of acting in an unfair or improper way. At one point I told my agent, Jorg Neubauer, who couldn't be a bigger Real Madrid fan, that I wanted a change. I was tired, because in the end it was all extremely stressful.
“However, I have to say they were five fantastic years.”
The midfielder admitted playing for Los Blancos came with heavy expectations and intense scrutiny.
“This obligation of having to win everything at all costs is something I have never experienced,” he said.
“Real Madrid has an image abroad that words cannot describe. The club is a magnet. No matter where you are there will be thousands of people who are with the club.
“I have experienced it myself. When I look for a holiday destination I always look for a place far away. But no matter how remote the place is, in any corner of the world, people always recognise me.”