The relentless federal push toward higher homeownership, the rock bottom interest rates, the overbuilding (and oversupply) of houses, foolish borrowing, foolish lending, securitization, insane leverage, the credit rating agencies, the general assumption that home prices would continue to rise forever, etc.
Wall street and the rest of the major banks weren't the only ones that played a role in the housing/economic crisis, they're just the most convenient ones to blame.
Who should be taxed to pay back the TARP money? The goonion members who's jobs were saved, for one. The bank employees who's jobs were saved, for another.
That sounds perfectly fair.