Downvoting affects directly the visibility of a message and this should not be based on opinion.
You seem to be splitting hairs here. The OP complained about the bias of others but didn't like it when he had his own biases pointed out.
I'm suspicious of any claim that says "votes" somehow have more "power" than words. Words, opinions and ideas all have power too.