Your body can cope with EXTERNAL radiation... how this disaster is different is that this radiation is contaminating the food chain.
First, the areas that have higher levels of background radiation : The people living there have been living in those areas for generations, if nothing else life has a tremendous capacity to adapt to this, still low-level higher radiation levels. In brick houses as you point out, the radiation is emanating from the brick itself, but contained WITHIN that brick, in other words, it acts like asbestos, which causes cancer, but once it's set then there's no danger unless you start shaking it around and getting it into the air.
Second, you're not taking into account the ways in which this contamination from these reactors works its way up through the food chain... If an area of the sea is contaminated, first small fish will be exposed and then eat contaminated algae or whatever... so they get a higher dose, THEN the bigger fish get a dose and eat more and more of these contaminated fish so they get the extra dose, and in this way the contamination works its way up the food chain.
Just like cows will eat contaminated grass, then they will process that into the milk that you drink.
Finally, there's also a difference between external exposure and internal exposure. Your body will absorb any radiation and it will radiate you from the inside until your body is able to process it, if your body can process the toxin at all.
It's not about alarmist panic, it's about realizing the reality of the situation that must be coped with...