"Jerry's lying," Granda told GMA's George Stephanopoulos. "That was his game plan from the beginning, to throw her under the bus which I think speaks a lot about who he is, about his character. He was aware from day one about our relationship."
Granda said he "lived through hell" when an attorney threatened to expose the nature of their relationship in 2014.
Granda, a former pool attendant, accused Falwell of knowing of his relationship with wife Becki and that Jerry Falwell participated in some of the encounters as a voyeur. Granda said the couple told him they were at a swingers club the night before.
Granda said he was talking to some guests at the Miami Beach hotel he worked at when he was 20 years old. Granda alleges Becki was noticeably drunk and flirtatious with him and invited him to a hotel where Jerry would watch their encounter.
Falwell Jr., the former president of Liberty University, was lying on the bed when the two went to the room, according to Granda.
"I don't want to go too much into the details, but he enjoyed watching," Granda said.
On Monday, Reuters published Granda's account of a sexual relationship between him and Becki Falwell, including what the news agency said was an audio recording of a phone call between both the Falwells and Granda. The relationship lasted nearly seven years.
The Falwells are accusing Granda of targeting them and other successful women.
"It's kind of ridiculous to think that this 20-year-old with not many resources, I don't come from a family with a lot of money, was targeting and preying upon this power couple who have all the political connections and all the money in the world. I feel like I was an ideal target for them," Granda said.
Overnight, a second allegation emerged from a LU student in a Politico report as a former student claimed they had a relationship with Becki.
Falwell said he was seeking mental health counseling after dealing with the fallout from the affair, which he claims he had no role in.
North Carolina GOP Rep. Mark Walker was one of several Liberty supporters to call for Falwell's resignation. Falwell resigned on Tuesday.
Granda, who did not return a call from AP on Tuesday, took on partial ownership of a hostel in Miami's party-friendly South Beach neighborhood with members of the Falwell family, according to court documents. Moving forward, Granda said he wants to sell his equity stake and cut business ties with the Falwells.
The Associated Press contributed to this story.