C.K. wrote: What was I supposed to say ? "yes sir, you're right ... ."
Beyond expressing your disappointment with the situation, you should say nothing. That's the professional
approach. Your response made this a personal issue.
Note that this in no way
implies any agreement with his assessment. You know what the truth is, and you're not going to change either what he thinks or what he tells others. There is no way your little tirade was going to improve the situation - and every likelihood that it would make it worse.
When you're in a "can't win" situation like this, your objective should be to salvage what you can - not ensure that you lose.