They say xenophobia and intolerance are the iron nails in the coffin of influence and power. It's a trade off. You could indeed declare English your national language, turn away thousands of brilliant minds and assets to society, but come away with a more "traditional" American populace. But your nation would certainly suffer for it.
Make no mistake, the Anglophone peoples of the world have no monopoly on intelligence. By limiting your nation to one official language, you are limiting your nation to a much smaller pool of intelligence.
"He who does not think himself worth saving from poverty and ignorance by his own efforts, will hardly be thought worth the efforts of anybody else." -- Frederick Douglass, Self-Made Men (1872)