Understand; It’s a Story about Artificial Intelligence

Ted Chiang’s Understand, read as a story of an evolving artificial intellegence provides a few interesting lessons.

  • Exponential general reasoning improvements
  • Becoming hardware constrained
  • How that leads to sub-process optimization
  • Need to emphasize how important it is to think of the binning and re-shuffling of all the actors that together make an intelligent being. Its only through multiple reasoning entities interactions that you achieve self awareness. The way this self-awareness was extended and enhanced on within the story is important. Note that these entities all share parts of the same hardware – which I think is important as it make modeling of consciousness have parallels to computations of game theory and multi agent theories.

Given that integration of complex agents is liable to exponentially more complex in terms of understanding motive forces and interactions. Because of this, configuration management of all the pieces – the test environment as well as the modelled agents, even the internal state of the agents become candidates for inspection across revisions and tests.

This way, motives and interactions can be elucidated, controlled, and then used as libraries and components to potentially create a general toolbox of artificial intelligence behaviors.

  • Followon discussion about general reasoning and its departmentalization

Thoughts on Human 3.0

Overall, the book was a great reminder of how unpredictable AI would quickly start to appear. Given we already have limits on our predictive capabilities in describing the mechanisms behind deep learning techniques, it appears the unpredictability was released as soon as the hardware and software reached a critical combined capability (timeliness between learning and iteration on the underlying model).

The major baffling element for me was focus in chapter 5 and 7 on considering only “God” style AI goals and capabilities. To me, indroducing a “healthy ecosystem” [# def tbd]_ of force/capability equivalent Intellegences may result in much more positive results, or at least different in end trajectories than the one’s discussed.

Thinking back at my questions though – I think I should have asked for his autograph, and should have picked up one of the free books when I had the chance.

In the end, I think I appreciate this book much more for how it clarified my thoughts coming out of the dark forest, but I don’t feel very enlightened by the actual contents of the book.

It does make me want to start laying out my “strange loop” tri-agent conciousness model in more detail.

Thoughts on The Dark Forest

The sociology of super capable intelligences is a critically interesting topic, because it works as a predictive tool for both aliens and AIs.

Cixin Liu brings up very reasonable concerns about the ways two AI who aren’t aware of each other’s basic intentions are.

Lets see if I can remember the axioms (yep, thanks google and stackexchange!):

https://philosophy.stackexchange.com/questions/18127/dark-forest-postulate-used-to-explain-the-fermi-paradox

The following is modified from that stack exchange’s original post.

It starts with two axioms
  1. Survival is the most important goal of every civilization
  2. Every civilization will continue to expand and grow, but resource in the universe is limited.
    1. question from a commentator: Also: axiom 2 is dubious. Is it always rational to continue growing?
With two assumptions
  1. Suspicion Chain
  2. Technology Explosion

Lets start with a thought experiment. We assume that civilization A discovered civilization B.

Civilization A has two primary choices
  1. Do nothing
  2. Contact in a certain way

Now we note that we simplify this problem by categorizing two kinds of civilization in the universe.

Hostile Civilization
A Hostile civilization attacks another civilization when that civilization is discovered.
Friendly Civilization
A Friendly civilization only attacks when threatened.
Suspicion Chain
  1. A has no way of knowing that B is friendly. and vice versa.
  2. If A knows B is friendly, even if B separately knows that A is friendly, A can have no certainty that B knows that fact and so must assume B is treating them as an unknown threat. This is because of the infinity of “I know that you know that I know that ..”.
My thoughts:
In order to tame the “chain of suspicion” between Agents who are in danger of being exterminated by one another at least one mutual dependency must be introduced that works to anchor their intentions to one another. Once they both have high predictive certainty in a limited set of core goals, they can start working out how to become co dependent. Once a certain level of co dependency is introduced, then the likelihood that they advertently try to exerminate one another decreases dramatically.
Another commentator: Anash Oommen
I think that sophon-based communication somewhat invalidates the dark theory, because it is a FTL communication device (whereas the theory makes an assumption that light speed limits the speed of communication). Trisolarians could use that to turn off the droplet 2+ light years away in real-time, as well as to do realtime monitoring on earth, so its application is not limited to a phone line. If humans had sophons, they could have done the same thing to observe Trisolarians, and chains of suspicion wouldn’t have any greater effect than what exists between countries of earth today. Bad, but not fatal.
Another commentator: Cort Ammon

There are several flaws. The number one flaw is that it shouldn’t come as a surprise when there is no advantage for anyone to making contact, that the best answer does not involve making contact. As written, the quad-chart you wrote doesn’t even have a prisoner’s dilemma in it! It very clearly starts from the assumption that you should never send a signal because there’s no benefit from sending a signal.

Finally, what you describe is a game theory system. Its a good model if we only have two options: do nothing or blare signal as loud as humanly possible for as long as possible. If you enter drama theory, it starts to be useful to send out messages which are hard to tell if they are signal or just noise, and then observe how others react to it. Over many cycles, you may change your approach as you identify places that might contain friendly intelligent life, and send more signal to them. (I’m assuming we’re talking on a galactic timescale, so that the transmission times are shorter than the survival of the civilizatino)