The caffeine in coffee acts as a natural thermogenic. This boosts metabolism and helps attack fat. If it is the sugar which, as you've seen it is, than cut it out. Use Splenda, or Stevia, or even Sugar in the Raw in smaller amounts. Unless you are adding coffee to your sugar than sugar isn't the problem.
Now, what I've done with great success is took a strategic approach to my coffee drinking when I'm working on weight loss. Drink a cup before and a cup after exercising. Consuming sugar directly after exercising is crucial, as it replenishes lost glycogen levels in the body, the caffeine extends the increase in metabolic rate. Obviously drinking it before hand gives you an energy boost which will increase performance as well as allow for more calories to be burned.
IF you just want to get off it, than I'd suggest a green tea. There is one called Gunpowder that gives you a real bang without any of the jitters. Mancha is strong too but is a bit more expensive for the good stuff and takes a little longer to prepare.
As a mid day beverage I usually have a few cups of Earl Grey w/ honey or Sugar in the raw.
All in all I can assure you it isn't the coffee thats inhibiting you, not unless you're having a coffee cake to go along with it.