the buildings around
how its structured
the concrete
All of these things DO add to the visual appeal.
For example, if you like a more traditional look than the new age glass monstrosities, the Empire State Building having a more visible concrete actually enhances its aesthetics by giving it a traditional look and an interesting dichotomy between the glass and concrete.
Its structure is a HUGE factor of it. For example, I don't mind the triangular shape of the NK building though I'm not a HUGE fan of it, but I hate the structure of the huge glass panel on the front. It seems out of place and makes it end up looking like a rocket ship rather than a building. Back to the Empire State Building, to me the structure of it gives the impression of a water fall, as you go farther down with each different piece jutting out it seems like the concrete is flowing over the building like water from a stream. The structure of the Empire State building adds to its aesthetics for me.
What buildings are around DOES matter. Things are beautiful, in part, due to the surrounding things. Sometimes, something that sticks out hugely from its surroundings can be appealing...other times it seems gaudy and over done. I like the Empire State Building; its size is enough to have a leg up over the other large buildings of New York's sky line. However, if it was in the down town area of my home city:
I'd probably say its FAR less attractive. Why? Because it would be SO out of place that it would just look weird, not beautiful. It'd tower nearly 60 stories over anything near by. That goes from being large enough to make an impact to just monstrous. That takes away from the appeal.
Talk to any good photographer and they will tell you, the area around what you're photographing DOES matter. Your surroundings play into whatever it is you're making, same goes for the buildings. The Empire State building is a majestically large building amongst other large ones. The one in NK is a gigantic monstrosity amongst run down fledgling smaller ones gathering at its foot like ants looking upon a person.
Additionally, where its built and how its built CAN affect peoples aesthetic views. For example, I know people who generally find designer clothes to be rather unpleasing to the eye. Not necessarily because they're not good to look at, but because for the price that they cost they are not as attractive enough to justify it, thus making it even uglier in their eye as it is wasteful. Now, you can whine, bitch, and complain that that's biased of them but that is part of human nature. We all view beauty differently, and all view it through different criteria's. There is no definite, universal, definition of "beauty" because its impossible to have one. Grossly over the top, wasteful things can be so offensive to the person viewing it that it can easily factor into whether they find something attractive or beautiful.
If it was simply "north korea" that was the issue, then people wouldn't be commenting positively on the next thing posted from North Korea, but its not. For those that have an issue with it for being wasteful, it would not have mattered if it was North Korea of South Africa, such huge decadence and waste in what is essentially a paper tiger of architecture to them is offensive to the senses and makes the thing look ugly.