Finance companies were doing distributed systems even before Internet existed. Yes, finance has a lot of numerical work but it has a lot more data pipeline work. The reason for this is to move market data from one geographical location to another. To make profitable investments, finance companies need accurate market data from all over the world available and processed at its datacenters as quickly as possible. For this reason in the early 1980s finance companies were laying down their own cables and writing their own link layer protocols to move data from one location to another.

Now in the 21st century this has only got more intense. There is much more competition in this space. So finance companies that can make distributed systems right and reliable stand a good chance to the most money from the markets.

Finance is a huge field. I'd say the need for mathematical modelling, in one form or another, is pretty universal no matter what you do in Finance. But there are also tons of other stuff you might need on top of that depending on what specific field of finance you work in. Distributed systems can provide the data that your model relies on, or even the platform where the model itself needs to run on.

Downtime is death, and performance is important. Distributed systems usually can help you avoid some of the types of downtime, and can help you manage throughput, although straight up latency tends to be become slightly worse.