The veteran congressman said he'd rather be doing anything other than talking about Biden's fitness for office, or lack thereof.
"Four years ago Joe Biden did our country and our democracy a great service by beating Donald Trump. I'm not sure he can do it again this time," he said.
In the days since President Biden's halting debate, Quigley said the president hasn't given him confidence he can carry through another election.
MORE: ABC News anchor George Stephanopoulos' exclusive interview with Biden: Full transcript
"This isn't the time to be a cheerleader. this is the time to be a realist and be honest with ourselves of what the situation is, as unpleasant as it is," he said.
Quigley said the conversations are constant among House Democrats, dozens of whom are gearing up for their own tightly contested races. He said control of Congress hangs in the balance.
"Every vote counts. And if the top of the ticket doesn't carry, or at least help, in those races, you could have a red wave," he said.
RELATED: Biden doubles down on debate explanations in ABC News exclusive
U.S. Rep. Brad Schneider, a North Shore Democrat, isn't ready to publicly go as far as calling on Bide not back out.
"I will support the Democratic nominee and ensure Donald Trump is not elected president," he said.
At a rally in battleground Wisconsin Friday, President Biden made it clear he isn't taking the cue from some of his colleagues.
"Well, here's my answer. I am running and gonna win again," he said to cheers.
Quigley said he hasn't spoken to the president personally.
"I would tell him his legacy is what he did to save our democracy for the last four years and that he can cement that by making sure the person he replaced doesn't get a second shot at it," he said.
Quigley said he believes more of his colleagues will say the same as lawmakers return to Washington on Monday after the holiday break.