I think the key is finding someone at your level. Otherwise, the fun gets sucked out of it. I used to play at a club where a father and son duo would play once a week for hours. They only played each other, and were so close in ability. Because of that, they never improved, but that was fine with them. They'd play for a bit, chat, and then go back to playing, and chat some more. It was just as much as a social bonding thing as a way to get exercise. I always thought that was so cool.
Me, I have been playing for many years, and have just recently started taking it seriously once again. I used to play tournaments, but I only play a few select now days. There are many good juniors around here who hit the ball a ton, and play without fear. If I elected to compete in the same open tourneys, I'd so get my ass kicked. That would suck the interest for me-- so I avoid them--and keep tennis interesting and fun.
The best thing about the game, though, is that there is always room for improvement. The game is complex, and even the #1 player in the world has weaknesses.