Yet none of those same people have an issue with taking their phone out of their pocket constantly and glancing at it which has... a camera on the front! But I could see the issue of never knowing whether or not the glass camera is recording and whether that recording is going to wind up on youtube.
The difference with a phone is generally people pull it out, do something, then put it away. You'd get a similar reaction from onlookers if you kept your phone front and center your entire day.
True, but Google Glass is more like camera pointed directly at you. If somebody kept pointing smartphone camera at me constantly, I'd sure get annoyed.