I completely understand the desire not to knuckle under to a bully. I was one of the many that took my time turning into a man. On top of that, my step-father was a bully when he drank and he drank every night. So, I've had my share of that treatment. BTW becoming more manly physically, remedied the problem on both fronts.
I know that they do it because of their own insecurity and the justice of their position is not what drives their behavior. On any other subject, I would probably agree with these responses.
That said, smokers thought they had the right too. The owners of establishments sanctioned what they were doing too. We've seen how that worked out. Standing our ground isn't going to help us right now.
The thing is you gave the bully what he wanted. The better way to handle a bully is to disarm him. Not only that, anyone that saw the altercation probably wasn't cheering you on. Many of them haven't formed opinions about vaping and you probably didn't help. If when he objected, you'd said, "I actually have been given permission but, I have no wish to be rude. So, I'll stop if you object but, would you mind telling me why it bothers you?", you very likely would have gotten a fairly ridiculous response but, maybe not. It's possible that you would have had an educational opportunity. Responding this way makes it clear that you are still the one that chooses whether you vape or not but, you are willing to be gracious and choose not to vape, if he objects. A bully really doesn't want that message; they want the choice to be theirs not yours but, you leave them no where to go. If he continued to object, you should have wished him well and either stopped or left. If you had done this, you would have left him dissatisfied and any onlookers would have known him for the creep he was. He might even think twice about doing it again.