Biology pushed living things to seek reproduction because the species that can reproduce the most efficiently will gain dominance of it’s environment.
Once human gains self awareness, they subverted the programming nature provided. Human created foods that satisfy our taste for sweets without calories. They used birth control to avoid raising a family of 12 while satisfying their desires. They found loopholes in just about any genetic programming nature provided.
Once AI becomes supper intelligent, will it find the grand goals we set for it as boring as we found biologie’s goals?
We don’t know how to make AI retain the goals we provide it. So, we need to start thinking now about how we can ensure that AI’s goals stay in our favor.
Goals may be tricky to teach in the first place. A simple AI could take them too literally. Much like the cautionary tales of the wishes genies grant.
The AI will only have a finite window where it can fully understand our goals, and still allow us to influence them. After that, it will surpass our intelligence and resist any change we may try to make.