That's... strange. SF's black population has definitely been on the decline (but they're certainly here, all over!), but you must have been actively hiding if your only exposure to brown people was people "kept back in the kitchen".
I'm not going to claim that SF is some amazingly perfect bastion of racial diversity, but if you're mainly only seeing and interacting with white and Asian people here, that speaks more to the circles you spend your time in than the city's diversity.
The city of SF is approximately 40% non-hispanic white, 34% asian, 15% hispanic, 5% black, 5% mixed, and 1% other.
Other Bay Area cities have different distributions. For example Oakland is approximately 29% non-hispanic white, 26% hispanic, 23% black, 15% asian, 5% mixed, 1% other.
I'm not going to claim that SF is some amazingly perfect bastion of racial diversity, but if you're mainly only seeing and interacting with white and Asian people here, that speaks more to the circles you spend your time in than the city's diversity.