US slavery was not on the way out. It was other way round, actually. It was super profitable and powerful. It was also gaining legal protections. There was no active threat to it and the actual dispute and conflict was about new territories.

Also majority of abolitionists were pacifists to a fault. Literally to a fault.

Yet also, slavery was holding only due to violence. Both violence against white abolitionists and against blacks - slaves and free. Violence here includes murder and torture.