Why can’t there be some semi-public place where only men, or only women, or only whomever you choose, is allowed? Why can’t that exist?

What you described is unrelated. Yes, people can and do go out and do stuff.