The friend also said Broadwell is trying to move forward. Broadwell and her family were seen returning to their home in North Carolina on Sunday after the 40-year-old biographer spent more than a week at her brother's home in Washington.
The family was greeted by a group of family and friends. The family declined to talk to the media waiting outside their home.
"At some point, I'll have a statement for you guys. It's just not right now," Broadwell's husband Scott said.
The FBI discovered the affair after tracking emails sent to Tampa socialite Jill Kelley, warning Kelley to stay away from Petraeus and Afghanistan war commander Gen. John Allen.
Broadwell is still being investigated by the FBI over classified documents found on her laptop and in her home. It's believed the author gathered the documents while researching her biography of Petraeus in Afghanistan.
Investigators say many of the documents are old and may no longer be classified despite their labels. Also, Broadwell had security clearance to review the documents as an intelligence officer in the U.S. Military Reserve.
The FBI found no reason to further investigate Petraeus, but now, the CIA is investigating whether Petraeus behaved inappropriately, such as possibly using agency resources to further the affair.
The Associated Press contributed to this report.