Wearing google glass is the equivalent of camera that is always on. A cell phone might have a camera, but it becomes physically obvious when someone is using it. If someone sat in a restaurant videoing other customers with a camera, they'd probably be asked to leave as well.
Technology shouldn't be feared, but it should be used with respect for others. People have a reasonable right to sit down and have dinner in a privater restaurant without being recorded by random strangers without cause.