The problem isn't polygamist families. It is those who would use marriage to defraud the government and others, especially since this happens already with limiting people to only one spouse. We have a huge procedure, with a lot of already costly measures to check to ensure that when someone requests to gain citizenship for their foreign spouse, that there is no fraud involved in the case. Because there have been cases where people do this, and it is a very lucrative venture for them to do this if they are able to keep up the charade long enough to actually get through the process of being vetted and taken out of scrutiny. But if we simply knock down restrictions on how many spouses a person can have, this could cause huge issues in many areas, including immigration. What would prevent someone from marrying 20, 50, 100 people and applying for spousal citizenship for all of them? Even if all of them were eventually denied, our laws would require us to treat all of them the same as we would any other person looking to get citizenship for their spouse. With a limit, this type of fraud potential is limited as well as cost to society/government on investigating it, but without a limit on number of spouses, there is no limit on the fraud potential for these cases.
And this is just one of many areas where limiting fraud potential alone is a good idea and could easily be considered a legitimate state interest, for at the least not just striking down the bans.