I like caravans too.
How about making it so you can see them on the map and attack them ? Maybe neutrals wouldn't be able to, or you would choose where to stop the caravan (but they would be quite fast)... And it could only move from 1 point to another, nowhere else.
I'm tired of the fact that when I steal an enemy castle, 1 or 2 caravans are about to arrive... If it was possible to see them or block them, I could just kill the creatures right away, instead of having some enemies wandering around my castle.