"So their use of the drugs would interfere with my freedom of choice."

You are not entitled to particular choices, but are free to choose among what others may have made available.

It's no violation of rights that you don't get the choice you might prefer. If I get to the movie's five minutes before you, and take a seat you may have wanted, I'm not violating any of your rights of free choice because I parked my ass in a chair before you could.