Courtesy or Cowardice?

This week I attended a meeting (that had no legal connection) to hear a speaker who was to talk about a subject that interests me.
The meeting was in a charming old church hall in a beautiful Chiltern village. Those attending were what you might call countryside types, many of them active farmers. There was a Lord, and a Baronet. Everyone was charming, wine and sandwiches were handed out, and I had a super evening. Except for one thing.
I was introduced to a largeish, blazered, florid man of about my age who greeted me with a firm handshake. We were soon in conversation. He was an old Africa hand, having spent many years on a continent that I have never visited. As his confidence in me seemed to grow he launched without warning into an appalling diatribe about the dreadfulness of the blacks, how they can never govern themselves and how any attempt to civilise them must end in failure. He had been told that I was a JP, so he went on to lament the loss of the death penalty and to complain that nowadays the mere act of a white shooting a black might be treated as murder. Like so many right wingers he assumed as a fact that I 'must be so frustrated' at being unable to dish out really severe punishments. He was in favour of summary street-corner shooting of thieves, assuming that such would always, of course be black. He was looking forward to a trip to South Africa when he would do a bit of hunting - conveniently his guns were already there for him.
Now this man represents much of what I find most loathsome:- racism, contempt for anyone he does not understand, slaughtering great beasts for fun; the whole deal.
And I wimped out and said nothing, merely disentangling myself at the first decent opportunity and breathing in good fresh air outside.
As an Englishman I hope that I display courtesy and good manners. Nevertheless I feel a bit ashamed that I just stood and let him pour out his bile. To challenge him would have embarrassed my hosts, and besides, I could not have been so crass on a social occasion. I still feel a bit grubby though.