I’ve been following this thread with interest. As most are likely aware by now from my numerous comments on various threads, I like taking photos of people rather than things – I don’t necessarily use the term ‘street photographer’ in the same way I wouldn’t use the term ‘digital artist’ to discribe my humble attempts at design; however, ‘street photography’ would certainly best describe what I try to do.
Anywho, the reason I mention this is because I often get asked if random strangers often object to having their photos taken and if I’ve ever been confronted. My answer is no; not really – in all the time I’ve been doing it and all the different places I’ve been (well, there was this one occasion, but it was entirely my fault and I was completely in the wrong (I was drunk, it was at a party, and I was fooling around with a freind’s point-and-shoot, so I’m not sure that really even counts 
).
I find it interesting as, in my experience, most people (particularly non-photographers) seem to imagine I’d encounter such situations far more often (and are often surprised to learn that I don’t – as to why I don’t, there are a few reasons I can think of). I hardly ever take photos of places or buildings, or even landscape in general, as it’s just not my kind of thing; however, I have actually encountered ‘suspicious curiosity’ on a few of the rare occasions when I have. It’s interesting to read, here, the comments of others and the theories shared as to why these situations may sometimes happen.
EDIT: Of course, I live in the UK and my experiences are UK-based only – needless to say, laws and public tolerance to street photography can vary significantly around the world.