Not sure were you live in SF and what circles you frequent. I live in The Mission and most times I go somewhere in the neighborhood there's lots of "brown" people. There's not much black people in SF but black people from Oakland hang out in some places I frequent.

Maybe it's your neighborhood, some islands in the city look like a white suburb, like Noe Valley for example.