> However there are issues. How do you even begin to formalize concepts like human well-being?
Oh agreed! But with AI we might(!) have the luxury to create different types of brains; logically correct brains for space flight, building structures (or at least the calcuations), taxes, accounting, physics, math etc and brains with feelings for many other things. Have those cooperate.
The only problem is that "logical correctness" depends on the limits of human brain too: formal logic is based on the usual pre-accepted assumptions and definitions ("axioms").
This is what I consider the limit of the human mind: we have to start with a few assumptions we can't "prove" to build even a formal logic system which we then use to build all the other provably correct systems, but we still add other axioms to make them work.
It's hard for me to even think how AI can help with that.
That's the approach behind Max Tegmark and Steven Omohundro's "Provably Safe AGI":
https://arxiv.org/abs/2309.01933
https://www.youtube.com/watch?v=YhMwkk6uOK8
However, there are issues. How do you even begin to formalize concepts like human well-being?