Two points:
1. There are many reasons that people use drugs.
When a person is using drugs in a fashion to escape their problems or feel better about themselves, they are abusing drugs. It doesn't matter if the drug is legal or illegal.
This is going to have negative psychological ramifications regardless of the legality of the drug.
At the same time, there are also people who use drugs without abusing them. These people can be occasional drug users who do it for the enjoyable physical effects, but also use them in a self-controlled fashion.
One of the most important factors to look for when discussing drug use is whether the person uses the drugs as a coping mechanism. If they do this, they are abusing the drug.
2. Everyone has a different perspective regarding the potential physical harm of some behaviors. Some people will argue that tobacco use is stupid because it can cause cancer, emphysema, etc.
To explain, some people do not care if something may decrease their life by X amount of years for various reasons. For me, the lack of belief in an immortal aspect of the individual leads me to the conclusion that an infinite number of years of nonexistence is equal to an infinite number + 30 years of nonexistence.
If I die 30 years earlier because of some enjoyable behavior I engage in now, there is no discernible difference. In the grand scheme of things there is no difference between the possibility that I die tomorrow and the possibility that I die 60 years from now.
And the ultimate result is inevitable. So, while I don't engage in immediately risky behaviors, and I take steps to prevent immediate death, I do not worry about potentially shortening my life by a relatively small time span.
As such, I don't see the arguments about increased risks for cancer and other life-threatening illnesses years down the line as persuasive.
In fact, you could say that I find these arguments to be somewhat stupid because of their rigid adherence to a subjective perspective of time.
Essentially, the argument is that not engaging in a behavior can make you live longer, but this perspective fails to realize that nothing exists that can prevent you from being dead longer.
A person who died in 1000 AD will be dead for just as long as a person who will die in 2100 AD.
An infinity + 1100 = infinity no matter how you slice it.