I've been on a lot of ML teams and outside of Finance and a few other sensitive topics explainability has always been irrelevant.

What happens, when your model exhibits a discriminating bias? How do you find out, what is going wrong? Knowing, what the model pays attention to can be pretty helpful.

Not aware of any court cases where someone successfully sued because they were shown one product recommendation or Ad on a webpage instead of another.