Walters, who spent two hours at the prison with Madoff two weeks ago, also told ABC's "Good Morning America" program that Madoff said that while he had contemplated suicide during his early days behind bars, he lacked the courage and never thinks about killing himself now.
Madoff is serving a 150-year prison term for bilking investors out of billions of dollars in a decades-long Ponzi scheme that is considered the biggest financial fraud in U.S. history.
Madoff's wife, Ruth, said in an interview to be aired on CBS's "60 Minutes" program on Sunday that the couple actually tried to kill themselves by taking pills on Christmas Eve 2008 after the fraud was exposed.
"I don't know whose idea it was, but we decided to kill ourselves because it was so horrendous what was happening," Ruth Madoff said of the failed attempt.
Walters did not address the subject of suicide on Thursday. She said Madoff and his wife are now estranged.
The couple's elder son, Mark, 46, hanged himself in his New York apartment on December 11, the second anniversary of his father's arrest. Mark and Andrew Madoff turned in their father to authorities a day after he confessed to them.
Walters said Madoff, 73, was distraught over his son's suicide, and that his wife wanted to stop visiting him in prison after that and he agreed. He has not seen her since, Walters said.
"Ruth does not hate me. She has no one, and this is not fair to her," Walters quoted Madoff as saying.
"He has terrible remorse, he says he knows that he ruined his family," Walters said, adding that Madoff told her that with the help of therapy he does not think about what he has done, but "at night he says he has horrible nightmares."
The interview, one of several involving the Madoff family to surface in the past week, was not filmed because cameras are not allowed in the North Carolina facility where Madoff is serving time.
Walters said Madoff speaks of being happier now because for the first time in 20 years he has no fear of being arrested.
"I feel safer here than outside," Madoff told Walters.
"I have people to talk to, no decisions to make ... now I have no fear because I'm no longer in control" and "know that I will die in prison," she said he told her.
As for his crimes, Madoff said, "the average person thinks I robbed widows and orphans. I made wealthy people wealthier."
Walters said Madoff told her, "every once in a while I find myself smiling, and I'm horrified."
(Reporting by Chris Michaud; editing by Greg McCune)