When I was a child, I was surrounded by people on drugs, failures at life, rejects, and other similar people in a very poor part of the country. For a time, my parents did drugs (all the way up until I was 21 or so, based on stories from my brother), they split up twice and got back together, my dad was abusive up until the second split up, and our home culture wasn't exactly the best as you can imagine.
Who am I today?
1. I am quite well off (so are my parents actually, but it came to them very late in life and I had to help them with it) and am above the top 20 percent of households (I made 8 an hour about 10 years ago for comparison) while still being in my early 30s.
2. I still live in a poor part of the country (rural georgia)
3. I do not do drugs, I don't even drink or smoke.
4. My wife and I have had a very solid marriage with only one time it being so bad that we even considered not sleeping in the same bedroom
5. My child is special needs child with autism and somehow I resist my frustration from being overwhelming enough to act out (and trust me, some days are worse than others)
6. My kids don't even cuss and our home culture is quite good despite challenges from point 5.
What you are growing up may or may not have anything to do with who you are as an adult, this is the perspective on which I look at your paranoia.