The ruling was made on appeals brought by owners of one-room pubs in two states, Baden Wuerttemburg and Berlin.
The court ordered all states to review their laws, giving state parliaments until the end of 2009 either to ban smoking entirely in all establishments or to revise exceptions written into their laws.
In the meantime, one-room pubs that have disallowed smoking since bans went into force can allow patrons to light up once again.
But the ruling could eventually force those same small businesses to go nonsmoking once again, if states opt to address the court's directive by banning smoking more uniformly -- which nonsmoking advocates hope.
"It could backfire on the entire hospitality industry," said Friedrich Wiebel, spokesman for German Smoke-Free Society. "We will fight hard over the next year and a half to push states to make laws that protect nonsmokers by allowing absolutely no exceptions."
Wiebel citing recent opinion polls suggesting two thirds of Germans support a broad ban on smoking in public places.
Germany has already banned smoking nationwide in government buildings.
But while each of the 16 states has enacted some curbs on smoking in bars, restaurants and other public places, the rules vary drastically.
In the southwestern state of Saarland, an exemption was made for establishments that offer water pipes. Bavaria touted laws it introduced in January as the nation's toughest, then quickly exempted beer tents -- and the millions of Oktoberfest visitors who fill them -- and also turned a blind eye as many bars to declared themselves private smoking clubs.
All 27 EU nations have rules limiting smoking in public places, but they vary from country to country. Germany's neighbors Italy and France have broad bans on smoking in public places.