No on children. Conditional on mentally impaired, if it's self impairment or outside their control matters IMO. The issue in all such cases is we pay a price as a society in making it illegal (see prohibition type reactions), in the black market, tax implications (least of the issues but it's real), the general idea that if we do this with most laws everyone is breaking some law at some point, there's blackmail and corruption that can result, etc.
What we may discover as a society is that if we allow certain freedoms to take their course, we may adjust favorably to them over time. It's definitely a complicated problem because it involves society, pros/cons on both sides, and is really hard to scientifically analyze.