That aside, I dont think they are going to flip out on him. They may censure him- perhaps even ban him from appearing at games (if he even goes anyways- a certain number of owners have little interest in sports).
But... I doubt they are going to try to ban him. At the end of the day, this is an isolated comment made in an apparently private conversation by an elderly person. I think it will end as that.