Sure, but that whole deal of "publisher signed an agreement with the author" only exists because publishers used to be necessary. They're not now.
Yes, you shouldn't arbitrarily decide anything. You can today, however, meaningfully conclude that "publishers are kind of useless, how are they still in business when much of what they do isn't necessary?" You'll find the answer is probably old law.
So, yes -- bring on the lawsuits. Let's see what happens.
ok, so don't bypass publishers for existing works, but set it up in a way so that authors can publish books in this library themselves if they so desire (and pay for editing, marketing, etc. themselves).
You can't just arbitrarily decide that only one party "deserves" to make money from the book, and doing so would probably lead to a lawsuit.