People should not be held accountable to their leaders actions because many leaders take actions against the wishes of their supporters. Examples: government shutdown, the Iraq war, keeping Guantanemo open, and so on. The leaders are then often punished by their supporters - the supporters themselves don't need to be punished as well.
If you can find a political system where all politicians are honest and keep their word, and furthermore are diverse enough that there is a politician availiable for each and every different kind of voter to be able to support wholeheartedly, with no policy disagreements whatsoever, and also a populace who are all educated enough to understand the political system and to have a rational debate on the pros and cons of the unforseen consequences of every slight shift in policy that their government might entail, without resorting to partisan tactics, then your idea might have half a hope.
TL;dr - no, because of partisans.