1. That the mob (voters) know what they are doing (or are knowledgeable/intelligent in their actions).
2. That, according to exit polls, some Walker supporters were voting against the circumstances of the recall rather than voting to support Walker's actions.
3. The founding fathers were opposed to mob rule (direct democracy, which you are arguing supports Walker's actions).
As far as opinions of why voters voted the way they did, I suspect time heals all wounds; in other words, had the election been much sooner while the events that prompted the recall still be fresh in voters minds, then Walker may well have lost. That the attention span of the public is rather short I'd think is not something with which most would disagree.