The college professor also argued to "Good Morning America's" Chris Cuomo Friday that the bombing campaign by the Weather Underground, the group he helped found, was not terrorism.
The Weather Underground bombed the Capitol, the Pentagon and the New York City Police Department in protest of the Vietnam War.
"It's not terrorism because it doesn't target people, to kill or injure," Ayers said.
Ayers became a boogeyman for Sen. John McCain and Gov. Sarah Palin, who demanded to know more about Obama's relationship with his Chicago neighbor. Palin accused Obama of "palling around ... with a terrorist."
Breaking his silence Friday, Ayers said that the GOP attack was a "dishonest narrative ... to demonize me."
"I don't buy the idea that guilt by association should have any part of our politics," he said.
Ayers scoffed at the Republican effort to make his ties to Obama appear suspicious.
"This idea that we need to know more, like there's some dark, hidden secret, some secret link," Ayers said. "It's a myth thrown up by people who want to exploit the politics of fear."
But he was unapologetic about his militant actions during the Vietnam War.
"What you call the violent past, that was a time when thousands of people were being murdered every month by our own government. ... We were on the right side," he told "GMA."
The co-founder of the Weather Underground was, as McCain has claimed, unrepentant about the the bombings his group committed during the 1960s.
"The content of the Vietnam protest is that there were despicable acts going on, but the despicable acts were being done by our government. ... I never hurt or killed anyone," Ayers said.
"Frankly, I don't think we did enough, just as Friday I don't think we've done enough to stop these wars," he said.
Ayers Says He Is a 'Family Friend' of Obama
Ayers softened his stand on violence during the "GMA" interview.
"We knew it was wrong. We knew it was illegal. We knew it was immoral," he said, but the group's members felt they "had to do more" to stop the Vietnam War.
He urged people Friday "to participate in resistance, in nonviolent, direct action" to stop the wars in Iraq and Afghanistan.
Ayers, 63, currently a distinguished professor of education at the University of Illinois at Chicago, became a political pinata for McCain and Palin during the presidential campaign.
Despite Obama's attempt to portray their relationship as a distant one, Ayers, in a new afterward to his book "Fugitive Days," describes Obama as a "neighbor and family friend."
On "GMA," Ayers again downplayed any close ties to Obama despite the reference to"family friend."
"I'm talking there about the fact that I became an issue, unwillingly and unwittingly," he said. "It was a profoundly dishonest narrative. ... I'm describing there how the blogosphere characterized the relationship."
"I would say, really, that we knew each other in a professional way on the same level of, say, thousands of other people," he said.
He added, echoing a phrase that Obama used to describe Ayers, "I am a guy around the neighborhood."
Ayers acknowledged that he held a reception in his home when Obama began his political run for state office.
"He was probably in 20 homes that day," Ayers said.
During the campaign, Obama tried to defuse the Ayers issue by condemning Ayers' past actions as "detestable."
"The notion that ... me knowing somebody who engaged in detestable acts 40 years ago, when I was 8 years old, somehow reflects on me and my values, doesn't make much sense," Obama said.
Palin Still Concerned About Ayers Tie
Ayers remained silent during the presidential race, but his proximity to Obama was highlighted on Election Day when the two men nearly ran into each other in the same polling place. As recently as Wednesday, Palin was still raising the Ayers' issue, telling NBC that she was still concerned about Obama's relationship to the former radical. Palin was the fiercest critic of the Obama-Ayers tie, accusing Obama of "palling around with a domestic terrorist." While he was a fugitive, he married Bernardine Dorhn, another member of the Weather Underground.
Obama and Ayers have several connections. The two men have also served on boards together, including the Woods Fund of Chicago and the Chicago Annenberg Challenge.
