I agree with the sentiment, but let me try to step outside the box a bit of my own general view point and try to take a stab at answering from the realities of the federal world...
First, not every office place has a mail room or cafeteria or some such place you can easily just shuffle the person to. Additionally, when an active investigation is going on, sometimes the need is for that person NOT to be present for the investigation to actually function right. The person still being in the workplace can make other employees that need to be questioned perhaps feel less apt to speak truthfully. Or that person still being there while clearly under investigation can simply help foster a negative atmosphere and moral as they begin to spread their gripes all through the office. Additionally, IF you're giving them work that actually NEEDS to be done then you're essentially taking someone elses jobs and duties and giving it to this other person which can cause some problems in a whole variety of ways. Not to mention, if it NEEDS to get done then it sets up a way for that person to continue to cause trouble as they'd hardly be a motivated party. If you just gave them busy work, then you run into an issue of oversight in terms of waste of tax payers money. "Now that's foolish" you say, becuase we're wasting their money by sending them home...and yet, having them do "work" that isn't actually "work" cause bring rise to waste/fraud/abuse type issues as strange as it may sound.
Rationally, you'd say you'd want to send the person home without pay and go about everything because then nothings being harmed. But there's too many limitations on the feds to be able to do that. As such, strangely enough, during the investigation typically the least amount of harm and potential harm is done by keeping them AWAY even if you have to pay them.