January 1st, 2023
- #2tweet one of the weirdest things I've started noticing the last few years is some different qualities of sweat that I get. I haven't tracked them all down but one is a very runny foul-smelling one that afaict shows up when I'm writing something that disagrees with an authority
- one of the weirdest things I've started noticing the last few years is some different qualities of sweat that I get. I haven't tracked them all down but one is a very runny foul-smelling one that afaict shows up when I'm writing something that disagrees with an authority
- touches on everything makes sense,
- I left this comment
- Huh, reading this I noticed that counterintuitively, alignment requires letting go of the outcome. Like, what defines a non-aligned AI (not an enemy-aligned one but one that doesn't align to any human value) is its tendency to keep forcing the thing it's forcing rather than returning to some deeper sense of what matters.
- Humans do the same thing when they pursue a goal while having lost touch with what matters, and depending on how it shows up we call it "goodharting" or "lost purposes". The mere fact that we can identify the existence of goodharting and so on indicates that we have some ability to tell what's important to us, that's separate from whatever we're "optimizing" for. It seems to me like this is the "listening" you're talking about.
- And so unalignment can refer both to a person who isn't listening to all parts of themselves, and to eg corporations that aren't listening to people who are trying to raise concerns about the ethics of the company's behavior.
- The question of where an AI would get its true source of "what matters" from seems like a bit of a puzzle. One answer would be to have it "listen to the humans" but that seems to miss the part where the AI needs to itself be able to tell the difference between actually listening to the humans and goodharting on "listen to the humans".
- I didn't add this on LW yet since I thought it might turn people away from the first half
- It seems possible to me that the defining characteristic of AGI (as distinct from narrow AI) is whether it has this strange loop of being able to check whether it's goodharting, by listening to some deeper sense of what matters, that is somehow open rather than a closed fixed target. If this were the case, then it seems to me that AI risk can be summarized as:
- 1. risk of very powerful and strategic but still in this sense narrow intelligence wrecking our shit (like covid could with an ability to think ahead)
- 2. risk that if we created an AGI it wouldn't care about listening to people any more than we care about listening to ants, and it would go create a very interesting and possibly even beautiful universe (insofar as beautiful is true and it would recognize that truth) but without any people in it
- and it seems to me that people who think Eliezer is missing something obvious and AGI risk is overblown perhaps are assuming that 2 is not actually a concern because any being that knew how to do caring (ie "knowing what matters") would care about humans
- though it also seems to me that in this case (if my definition of AGI is apt), Eliezer is actually more concerned about narrow AI than he thinks he is
- (these latter thoughts are quite related to my thread on confabulation and neural nets)
