The author, Baldur Bjarnason, was trying to understand why people feel LLMs are intelligent. LLMs do not have properties of brains. Either current AI tech is some sort of unprecedented and unexplainable new type of intelligence. Or that intelligence is an illusion in the mind of the user.
Baldur goes on to compare how LLMs work to the way that psychics work.
- Cold reading is a particular set of techniques psychics use to guess statistically likely facts in a specific feeling way about marks
- Humans have a subjective validation bias - we feel things that we identify with are correct
- The more things we identify as correct from the psychic the more likely we are to believe other things from them
- Psychics then repeatedly hone in on a mark with a variety of “validation statements”, which encompasses a variety of techniques
- Forer or Barnum statements - “you tend to be too hard on yourself”
- Vanishing negative - add a negative to a question that the psychic can interpret positively or negatively based on your answer. “you don’t play the piano?” → “didn’t think so” or “that’s what I thought”
- Rainbow ruse - associate mark with a trait and its opposite, “You’re a very calm person, but if provoked you can get very angry.”
- Statistical guesses - “you have, or used to have, a scar on your left leg or knee”
- Demographic guesses
- Unverifiable predictions
- Shotgunning - psychic fires of a barrage of statements and the mark will only fixate on/recall the one that lands
An interesting note, apparently many psychics fool themselves with their own readings. Very interesting parallel to the AI stuff going on right now.
LLMs are trained with Reinforcement Learning through Human Feedback (RLHF). This means that output data is rated by a human. The volume of training means that these are unlikely to have been thoroughly reviewed and rigorously fact checked by experts. Most likely they are reviewed en masse by low paid workers. They’ll be rated successful if they pass a cursory examination as valid. This means all the cold reading techniques above are very likely to be rated highly.
It is easy to fall for this. It is your intelligence/brain working against you.