It comes just a few days after a judge ruled the lawsuit involving the Robert E. Lee statue could more forward.
Police say no incidents happened at the rally.
Mayor Mike Signer tweeted at the protesters, saying "Another despicable video by Neo-Nazi cowards. You're not welcome here! Go home!"
The rally lasted under 10 minutes.
Police officers from the City of Charlottesville then escorted a tour bus carrying the protesters out of the city to ensure they left the area.
RELATED: A timeline of the deadly weekend in Charlottesville, Virginia