The six-member village board has unanimously voted to withdraw from consideration for the site.
The vote follows months of organized opposition, which included a petition drive and a large protest march.
Opponents argued that the center would bring down property values and pose a potential security threat.