How far from the World Trade Center before it's OK to build a mosque?

I understand why many people don't want a mosque built two blocks from the World Trade Center site in New York City.

But if we start drawing that line, where do we stop?