As a white guy I wouldn't go to a restaurant run by racists either.
Would you as a white person buy products made by companies that hire child workers, gather information on their employees for the authoritarian regimes, destroy essential farmland and water sources and poison people with pollutants that could be controlled for an affordable price? Because I'll bet you dollars to yuan that in the past month you have bought products that were at least made in China in exactly that fashion. And I don't blame you, such products are difficult for you as a consumer to avoid. The right thing to do is for the government to ban such practices and for our government. Boycotts can be effective but only on particularly salient points because boycotts don't spring from nothing.
If the government hadn't made white's only restaurants illegal back in the 60's, you would eat at them unless it was a particularly high profile social issue or you were particularly committed. People in the past fought and won these battles and you dismiss them because they are already won.
So when people say this...
If I was a black guy, I wouldn't really want to go to a restaurant managed by racists in the first place
They are just being ignorant as to where our modern tolerance came from. Black people don't need to patronize racists anymore because people fought those battles already.