Well, Kevin has a point. Overpenetration is a possible issue with almost any firearm.
The .44 magnum is a bit egregious on that point as a handgun though. The main objection most pro's have is that the .44 mag adds a lot of recoil and noise without an appreciable increase in "stopping power" (which is a bit of a misnomer where all handguns are concerned) but street stats indicate the .45 acp, 9mm +P and .357 jhp all to be slightly superior in "stopping power" than the 44mag, and WAY better in terms of rate of fire on repeat shots.
Resorting to personal anecdote, I find the 44mag hard to control while having no problems making a can dance with a 1911 in 45 cal. Now with sufficient practice you can train yourself to get used to the recoil and blast of the 44mag, sure. Most pros don't find the extra weight and bother worth it, considering its performance as a man-stopper is not improved over more easily managed guns.
You could carry .44 specials in it, as you mention. Again, street stats show a slight reduction in effectiveness compared to the 45, 9mm+P and .357mag. That's a generalization of course since so much of "stopping power" depends on shot placement, which is the Injun and not the arrow mostly.
When I was a cop I continued carrying a .357 revolver (Smith Wesson M13) when most were switching to Glocks, because I could drive tacks with my wheelgun and mistrusted the early polymer pistols as service sidearms.
So basically I'm saying, there's a certain validity to the criticism you're hearing, but if that's your personal choice of carry gun and you shoot well with it, do your thing.