Odd. I occasionally end up having to fight, but usually it means I stopped while still in LOS or turned around for something. That only happened a couple of times during the course of the game. And early in the game (when I needed the money) I was walking through streets pickpocketing everyone in sight. Later it really wasn't worth the bother.