Regardless of whether you actually should* be able to, "should" and "can" don't always match.
* I'm sure Signal would object to you redistributing binaries under their name, even if you claim they are unmodified, but they can't verify that fact. And honestly such an objection seem pretty reasonable.
Modification of binaries of open source projects is a common and perhaps the only issue from a trademark standpoint. Firefox and debian used to have this too until they resolved it at some point.
As far as I understood the argument, you're free to distribute binaries, you just can't call them "Signal", since it's a trademarked name (?).