Unofficial smoke breaks have driven me to the point of insanity at certain points of my life (I have never been a smoker). I worked in a pizza joint for 12 years, and the owner was a smoker as well as several employees. Let me tell you how happy I was to watch the store solo for 10-15 minutes so 2 other people could go chat and smoke, somewhere between 3 and 6 times in an 8 hour shift. I work in a hospital now, and it's not as bad, but its still there. If you smoke, you can squeeze an extra break or 3 into your day to indulge in your habit. How is that fair to non smokers, who have to work harder to pick up the slack while the smokers are taking their breaks, and who get no extra breaks themselves?
If you'd rather take 3 10 minute breaks to smoke, pray, floss, jog, or whatever instead of a 30 minute lunch, then sure, that's fair (assuming the work doesn't have to stop when 1 person takes a break, assembly line, certain parts of construction, etc.).
If you want those breaks on top of your normal lunch, that's simply not fair to non smokers.