I swear that I set out to make this short. C’est la vie! This thing is nine interrelated dives into the role of doubting our own feelings, morals, and cultural norms as part of the way that we grow more complex and intelligent about deception & seduction. The next article will be a tiny little morsel that is easy to consume!
As you know, there are early access, audio versions, and other perks for paying subscribers. This article particularly benefits from the vocal emphasis. And to help you frame the philosophical forest hike that is coming, here are the section headings:
I. An Introduction to Ambivalence
II. Platforming Manipulators or “The Content Problem”
III. The Flaw in Normal
IV. Les Non-Dupes
V. The Postures of Transmission
VI. The Girls of KINO
VII. Serial Killers and Cognitive Biases
IX. The AI Hole & CultworldTM.
I. AN INTRODUCTION TO AMBIVALENCE
We do not like to be deceived.
Or do we?
How often do our complaints about “seducers” boil down to the fact that they were insufficiently seductive? Are we agitated that they attempted to trick us, or are we irked to have seen so readily through their tricks? After all, we are the kinds of beings who will pay to be enchanted by a stage magician and then feel vaguely out of sorts if she reveals how the trick was done. And we do, sort of, like the way that our eyes and brains (and value systems) are constantly (falsely) simplifying the world for us. It feels empowering. Perhaps too much truth would disturb, destroy us, or paralyze our capacity for creative action.
We do not want to be dangerously tricked. That much seems clear. Yet our hearts seem to yearnj for a simplifying grand narrative that will set us in motion.
We even feel enthusiastic about any emotive pop song that is potent enough to hack our suspicious hearts — even though they are performing, pretending and embellishing.
We speak highly of any philosopher, saint, or political leader who makes the complications of life seem straightforward enough that we can take satisfaction in having heard a “convincing” explanation. Convictions, they say, make convicts. Yet would we still like someone to convince us…
Our relationship to the whole situation of manipulators, seducers, and deceivers is deeply ambiguous. We both love & hate manipulation. On the one hand (which is a joke about the etymological prefix in the word mani-pulation), we get to talk ourselves into delightful or dangerous foolishness by having an external “charismatic agent” to blame it on. The cult leader made me do it! We exploit the idea of exploiters to meet our own needs. And yet, on the other hand, exploiters bear responsibility not only for their influence upon others but also for their desire to influence.
Internal and external causality are both in play. It’s complex. And I’m going to make it even worse.
The whole “liminal web” of overlapping transformational, developmental, and regenerative networks is plagued by a two-headed monster. One head consists of narcissists, sociopaths, subtle abusers, appropriators, cult-like dynamics, etc.
The other head consists of the narcissists-who-denounce-others-as-narcissists, the fragility worshippers, guru-phobic pseudo-egalitarians, self-important “protectors,” and various neurotics who experience free-floating anxiety about the very possibility of manipulation, gaslighting, and insensitivity.
All of that, all together, forms the situation. In a sense, we are not yet even inquiring about it until we feel comfortable (or appropriately uncomfortable) asking questions such as:
“When are people detecting the possibility of inappropriate influence, and when are those ‘detections’ actually a projection of the detector’s own unacknowledged desire to inappropriately influence others?”
And even that does not go far enough.
Sociopathy is a spectrum. Influence is all-pervasive. Narcissism can be contextual. And we are all co-enacting realities with each other in ways that make ‘blame’ blurry and make the attribution of danger into a much more nuanced affair. (Plus, do we really think that there are no undercover intelligence agents at our weird gatherings?)
Then add in the even weirder dynamics of postmodernism hypermodernism that K. Wilber explored years ago in Trump and Post-Truth World. Plus the technology-driven elimination of the consensus-based public square, coupled with the rise of deep fakes, non-human cultural agents, and the general erosion of trust in information.
What would it take for us to be better meta-participants in all of this?
I’ll start from a very personal position…
II. PLATFORMING MANIPULATORS, or “The Content Problem”
I do not believe in platforming people. That’s probably an ambiguous statement. Who really knows what it means to “believe?” And it probably sounds like I am against platforming problematic persons and dangerous ideas. Actually, it is that very concept that I am skeptical about.
I remain half-seriously unconvinced that memetic contagion is a significant driver of world events. Yes, we are impacted by ideas and the content of media, but there are soooooooo many other mitigating and additional factors. And as a good McLuhanist, I tend to think that the medium is the message. That means that our major psychological, social, and moral influences come from the communication technology itself — rather than from the particular people or specific content that we are exposed to via that technology.
Our family life, behavioral habits, paychecks, subconscious desires, and the hidden dynamics of collective intelligence and deep neurobiology could exert considerably more persuasion upon our attitudes than the relatively trivial matter of who appears on what stage. I also suspect that our deep anxiety about “exposure to the wrong people and ideas” is part of the ideological mechanism by which the dominant social control system, with both its left and right hands, keeps us reactively plugged into the relatively impotent engagement layer of the surface chatter world.
Don’t THEY want you to worry about everyone’s exposure to misleading material?
This kind of arch-skeptical attitude informs my work as a (mostly amateur) podcast host and frontline interviewer in the liminal networks. I have frequently faced, socially and personally, the question of whether or not I am amplifying or legitimating people by simply talking convivially with them, exploring their worldviews, being humanly kind, or tracking their perspectival journeys with curiosity. And although I listen closely to counterarguments, my own hard-won conclusion, over and over again, is: No. Obviously, that is pretty self-serving. Would I talk with despots and mass murderers if they had an informed opinion about Metatheory? Almost certainly.
However, this does not mean that I do not worry about cultural contamination, deception, manipulation, and bad faith. These problems perturb me intensely. It is simply that I do not see that leverage over these issues operates primarily at the level of controlling the possibility of manipulative “exposure.”
It seems to me that our tools, our homes, and our schools are either cultivating children with organic self-esteem, mutual sensitivity, high skepticism, and self-critique of their own responses to fads, authorities, and hasty interpretations — OR we are not. In the former case, “content” is not that dangerous. In the latter case, even seemingly benign content can generate bad outcomes.
The human brain is an exquisite imitation machine that largely operates by sampling reality through performative duplication. Do you know anyone who starts tapping their toe to a piece of music before determining whether they approve of it? We are duplicators but that fact can mean many different things. For example, we might imitate something in order to refuse it. Many developmental theories suggest that we model various phenomena in our nervous system in order to externalize them. By performing it we might turn it into an object of inspection rather than part of our subjectivity. Yet we collaboratively generate a cultural attitude of anxiety that behaves as though we are all constantly at risk of becoming any bad thing to which we are occasionally exposed. Although there may be some truth and some falsehood to this attitude, it also clearly serves the Empire of Marketing. The sociological regime that wants you to keep participating breathlessly in a world of persuasion dynamics that never cease to obsessively inspect the current cultural surface.
Remember Ghostbusters?
At the climax of the film, instead of metacognitively emptying his mind, Dan Akroyd tried to think of the safest and nicest possible idea. But then the Sumerian demon Gozer the Destructor simply uses that same benevolent image (the Stay-Puft Marshmallow Man) to begin destroying New York. He should have been working on the skill of emptying his mind, but instead, he focused on trying to choose good content over bad content.
III. THE FLAW IN NORMAL
And remember Charles Manson? Sure you do. He was a high-powered, emotionally broken manipulator (potentially backed by even larger networks of manipulators) at the center of a malignant 1960s American whirlwind. A pretty unreliable narrator. Yet we cannot blithely dismiss his comments, at trial, and afterwards, about the preconditions that enabled his murderous criminal sex-cult to thrive. The whole tragic enterprise, he said, would not have worked even slightly if the conventional American family were not already producing large numbers of needy, traumatized, unheard & gullible children. Normal was breaking people. The monster was just exploiting their brokenness.
Normal has a deep flaw.
That flaw conditions the way we emotionally approach the problems of manipulation, seduction, and deception. Without neglecting the problem of real predators, we must also recognize that we are a profoundly scapegoating species.
Under distress, all things being equal, we tend to lynch the stranger. The aggrieved and bereaved often gather together to target whomever is “easy to detect” rather than waiting for the results of a nuanced investigation.
This is often tragic. It is also heartbreakingly understandable. But what it prevents is not only the cold rationality of slow justice but also any real critique of the role played by the background structure of our normal social patterns.
In village life, as well as in the global village of social media and “the news,” we tend to viscerally blame the person who is on stage — rather than the whole community that taught us to think that people on stages are significant.
That too is tragic but heartbreakingly understandable. Conventional culture cannot function without irrationally valuing people who seem either prominent or official. That is how its internal self-organization operates. It needs to raise unnecessarily gullible, unwise, and seducible children to become self-sacrificial believers and obedient post-tribal cooperators who perform the potency of official symbolism.
People who feel that something is being encouraged if it is acknowledged. If that mythical entity known as the “Government” officially recognizes gay marriage, then we all will be forced to get gay married! Any fairness protection applied to a particular group is helping them overcome the rest of us!
Notices = Confirms = Promotes. This is the profound anxiety within the conventional layer of our own hearts. These feelings, which are deeply entangled with a social system that has predominated on the Earth for several thousand years, inform our old-fashioned anxiety that “platforming” is a way of “endorsing and encouraging.” It feels like a failure to lock the mythic gate.
Conventional culture has its appropriate place historically and psychologically. It is a particular kind of social operating system that works in tandem with symbolic thinking (what the dandy fops call “concrete operational cognition”). And for this symbiotic psycho/social system to function properly, it needs several things:
a viable life world; if the farms are absorbed by multinationals, the local habit food is poisoned and addictive, the churches are predatory, the town gossip is algorithmically mediated manipulation, then traditionalism curdles
a good war; conventional culture uses the idea of a moral battle against “evil” rivals in order to invest their collective libido in re/creating the great ethnic and hereditary pyramid; our team has all good attributes (at least in public) and our rivals have all negative qualities
overcoming subcultures; the “people” or the “nation” must constantly prevent its natural regression into regional tribes by fighting against gangs, denouncing cults, shaming minority communities; and teaching the doctrine of patriotism — the pleasure of suppressing yourself in favor of the imposed symbols and caricatures of the mass.
lastly, of course, it requires raising gullible children…
This need is why conventional systems must be reactively focused on rogue manipulators. They are a threat to the standard manipulation. Sociopaths, actors, foreign spies, cult leaders, etc., take advantage of the very mechanism by which conventional culture reproduces its family loyalties, church beliefs, and political identities. These unauthorized seducers exploit the fact that people are trained to ignore contradictory feelings and avoid critiquing their own moralistic emotions. Without this kind of training, the whole thing erodes.
Of course, real spies, sociopaths, narcissists, and dangerous cult leaders exist, but to what degree can we organically trust our quick and strong feelings about these possibilities? And to what degree are feelings enslaved to the reproduction of a particular social regime whose other forms (racism, xenophobia, authoritarianism, illiteracy, etc.) we might strongly condemn?
There is truth. There is goodness. However, that does not mean that our strongest and most convincing emotions about protecting truth & goodness have actually evolved to perform that function.
We are aware that our passionate feelings of righteous retribution cannot serve as the basis of a moral and pragmatic criminal justice system. The ethics of justice must exceed and constrain the moral fire of retribution IF we want a society that does more than just regenerate tribalism. Likewise, we might need to undermine some very natural anxieties about “bad influence,” if we are to grow beyond the socially encouraged gullibility that reproduces mass mythic ethnocentrism.
Think about Santa Claus.
Left to their own devices, young kids naturally start shrugging off the Santa Claus myth after a few years. It is not because they are philosophically persuaded toward a different view. It happens as part of their standard cognitive, social, and emotional growth. We end up with no large populations of Santa Claus believers. However, we do have large adult populations that perform “certainty” about Muhammed, Buddha, Jesus, the Left, the Right, Race, Gender, Identity, etc.
Would most of these also erode naturally if it were not for a special form of widely distributed cultural training that both creates general gullibility and cultivates a consensus around which to deploy it? Could “normal” or “conventional” or “mythic” or “folksy” or “ethnocentric” societies function without securing a population that can readily fall under the sway? And doesn’t our so-called Modernity need the same thing in order to get people to work in the factories, believe in political propaganda, or be moved by commercial marketing?
My contention throughout this essay is that the problem of manipulators and deceivers, even in apparently advanced cultural networks, cannot be sanely investigated without a deep challenge to the hereditary instincts that have been bred into us. Instincts which automatically seek out and react anxiously or righteously toward unauthorized sources of seduction to secure the influence of authorized sources of seduction.
Here’s what part of that general background cultural training looks like:
A lack of the cognitive practice of doubting things without recoiling.
Lack of casual awareness that there is a seduction, marketing, and propaganda element to every interaction. The Influence Matrix.
Naive identification with the impulse to “be good” or to “get rewarded” as a result of believing.
Not taking seriously the possibility that our enthusiasms might conceal a self-sabotaging or nihilistic urge to throw ourselves away.
Presence of unmet emotional needs connected to pleasing authority, joining a group, conforming, or even rebelling.
Rebelling?
IV. LES NON-DUPES
I often quote my own translation of a phrase from the weird French psychoanalyst and philosopher Jacques Lacan, “The non-dupes go astray.” Regular readers will recognize it. If that is you — give yourself ten fictional points and a little shot of dopamine. Confetti and colorful balloons drop from the ceiling. Today, I just want to mention it briefly as a specific form of manipulation: anti-gullible gullibility.
Have you ever automatically believed someone who told you that someone else was lying? Have you ever felt the seduction of a worldview that just happens to explain why the “masses” are brainwashed dupes? Ever found yourself nodding along to a video that explains a dark conspiracy, government cover-up, insidious ideology or the utter toxicity and ignorance of half the population?
Most smart people are vulnerable to this. We typically and unthinkingly trust anything that explains how not to be one of the dupes. It is easy to get clicks by promising to reveal what THEY DON’T WANT YOU TO KNOW!
Rebelling against normative authority strikes our adolescent emotions as a very viable alternative to our low opinion of our fellow citizens. It is only when we notice that the entire authoritarian hegemony believes it is a persecuted minority in rebellion against a dominant social ideology that we must begin to suspect that our own counter-impulse might be part of the way that power structures reproduce themselves — by making us gullible.
That’s a big idea. Let’s repeat it:
It is only when we notice that the entire authoritarian hegemony believes it is a persecuted minority in rebellion against a dominant social ideology that we must begin to suspect that our own counter-impulse might be part of the way that power structures reproduce themselves — by making us gullible through the promise of an informed contrarianism.
The exploitative hegemony consists almost entirely of people who think they are fighting back against the exploitative hegemony. Rebellion is a kind of conformity. It is a kind of conformity that can sell almost any candidate, product or idea. We should not forget about this when we feel a strong defiant surge of emotion to protect other people from a “dangerous influence.”
Every conspiracy theory begins with the feeling that you have (suspiciously easily) seen through the trickery that is duping everyone else. Dismissing conspiracy theorists as foolish dupes operates in pretty much the same way. And it is, to some non-trivial degree, our feeling of wanting to save people from deception that makes us ready to buy into almost anything. Our easiest moral emotions are a profound gateway to manipulation. Both unauthorized and authorized influence-peddlers tend to lean into moral messaging.
The weaponization of “ought.”
Have you ever been in a situation where Person A suggests it would be immoral NOT to side with them, and a little later Person B insists that it would be deeply out of integrity for you to side with Person A?
Happens to me all the time. Regardless of the situation, both parties act like they are uniquely sensitive, moral agents who have seen through a popular deception and wish to take a stand against it. Gives the whole thing a bad name. And none of them seem to notice how unpleasant and exhausting it is to have someone trying to persuade you.
Although we must find moral clarity and utilize our influence in particular directions, we ALSO cannot neglect that it is often deeply disrespectful, unproductive, and soul-wearying to “try to convince or influence.”
Once, in my late teens, I lost a last-night chess game (in which I initially had a significant advantage) to a megalomaniacal drug dealer who kept trying to “psych me out.” Although I could see him doing this in a very clumsy way, I was also so wearied by the unpalatable process of persuasion (alliteration!) that I stopped caring about the game at all. He probably thinks it worked. Maybe he’s right. How many of us buy things just to shut the salesman up?
This draining quality is common in encounters with narcissists, psychic vampires, cognitively impaired persons, and concerned moral parties who want to “help you” and “prevent public deception.”
For example, as much as we need to solve the problem of a degraded information landscape in which misinformation flourishes, the answer cannot be that governments and corporations are constantly intervening in our information to protect us from their definition of wrong and bad ideas. They have too often been wrong about what is wrong and frequently badly intentioned in their attempt to keep our fragile little minds safe from the toxic, bad people.
Who watches the Watchmen? And who watches the people (like me, just now) who say, “Who watches the Watchmen?” What are those weirdos up to? I mean to say only that the risk of manipulation does not decrease with the presence of those who purport to help us avoid being manipulated.
It has even become a popular contemporary joke that those who tell you to “do your own research,” “think for yourself,” and “question everything” are usually trying to reinforce simplistic emotions that lead you toward gullibility and indoctrination into that person’s preferred reality tunnel.
It is not that there is no difference between liberation and deception, but that such an important difference is largely unrelated to whether or not people are regarded as liberators or deceivers. It is certainly not strongly coupled to how hard people try to convince you that they are liberators rather than deceivers. Beyond a certain threshold, such claims invert their own significance.
V. THE POSTURES OF TRANSMISSION
I grew up fascinated by (apparently) “transformed” spiritual leaders. Wonderfully, I have gotten to meet and hang out with lots of these people — even sometimes in a peer-to-peer kind of way. So clearly I like them. Different ones in different ways. But the whole scene of gurus, teachers, satsang, communion, existential contagion, aura, etc., is filled with deep transmission ambiguity.
Suppose you take seriously the possibility that, by attunement to another human being, you may be able to resonantly absorb some of their being, consciousness, or subtle conductivity. AND ALSO SUPPOSE that during this process, you observe that they deploy the manners of trance induction, rapport building, suggestive excitement, and deliberate bonding. What is going on?
Here are some possibilities:
(a) They are tricking you into feeling “transmission” by the intentional use of attention-controlling language, evocative ideas, micro gestures, etc.
(b) They are unintentionally tricking you because their own traumas and habits produce these same seductive maneuvers without their conscious knowledge.
(c) They really are transmitting something interesting (that could be good or bad for you, regardless of how it feels in the moment), and their body, in spontaneous relational flow, automatically exhibits gestures of conductivity that superficially resemble manipulation tactics.
(d) You, and or the community, generate a real transformative energy, but you do so by tricking yourselves into believing that it is being transmitted to you.
If we want a scene in which people are free to experimentally participate in kinds of transformational communion that are highly praised by the esoteric traditions of our ancestors, then we cannot simply wait until all purported “transmitters” are verified as perfectly true, honest, and safe.
We have to simultaneously build a community ethos in which spiritual leaders are generally more respectful and open, yes, but also in which people take intelligent agentic risks framed by their awareness of a range of interpretive possibilities applicable to events of phenomenological resonance.
Personally, I have more trust in people whose energies, or even their “general condition of being,” give me lots of space to have my own experience and my own ideas about that experience. Trying to get me to agree with them, or their tradition, about the phenomena I am experiencing, feels hasty, desperate, unreliable.
Of course, there is a counterargument that suggests that the performance, intensity, and urgency of the teacher are necessary to help overcome the egoic resistance of the self-serving personality. Let yourself be manipulated or seduced beyond your own self-imposed boundary! That too is something people should be able to explore (as safely as possible) if they are moved non-naively toward it, but, in general, we should be skeptical about whether it is a standard necessary component of even transcendental spiritual work.
I call the 20th century the Age of Tantric Patriarchs. Eastern-informed, educated, nondual spiritual teachers (mostly but not exclusively male) known for tremendously powerful shaktipat or intense transmission energy. Today, in a more therapeutically informed and intersubjective era, people sometimes ask me why we do not see as many of these overwhelming flow events.
Perhaps those guys were adding a little something extra into the flow? Perhaps their own conscious or subconscious desire to “penetrate and overwhelm” gave an extra kick to the process? And, if so, was that a useful way to provide a radical intensity or a dubious additive that might even end up traumatizing people with its wondrousness?
There are many natural, and a few seemingly supernatural, ways to be charismatic, but the problem with “deliberately charismatic individuals” (those who believe in the importance of persuasion) is that it becomes almost impossible for them to discern between healthy spiritual transmission & compulsive, tactical, interpersonal gameplaying.
PRO-TIP - Edging.
We are all always vulnerable to, and perhaps even needful of, certain kinds of deception and suggestion. Nonetheless, we can increase our intelligence, integrity, and tolerance to unnecessary manipulation by experiencing it at threshold conditions. Both “being persuaded” and “avoiding persuasion” might limit our ability to see, understand, and participatorily feel the dynamics that are involved. A solution is to spend more time intentionally in the space between. An immune system is weak if it experiences no invaders. It is even weaker if it is damaged by too many or too strong an invader. Our bodies grow stronger and smarter under partial conditions of activation. Thus, we might seek out opportunities at the edge of persuasive ideas or individuals. Not too close, not too far away. Feel the condition of being pulled, sold, entranced, etc., without going fully in. An entire organic curriculum may be produced, bit by bit, in this manner.
VII. THE GIRLS OF KINO
“Kino” refers to the process of safe touch & escalation. “Peacocking” refers to making a distinct visual impression. “Negging” involves a combination of compliments and verbal undermining so that a person feels that the value of your attention is greater than their own objective value. And speaking of value… demonstrate any capacity, competence or ability to provide worth.
These are standard (i.e., extremely well-known) parts of the PUA. The Pick-Up Artist community. It is a masculinist, sex-oriented cousin of other subtle manipulation communities, such as Neuro-Associative Conditioning and Conversational Hypnosis. If you have any children, perhaps especially if you have daughters, make sure there is an open conversation about discourses devoted to tricking their intuitions with behavior, speech, and even chemicals. People were putting “willingness drugs” in drinks and “trust hormones” in body spray decades ago already. What do you suppose they are doing now?
Of course, many people on dates actively want to be seduced, but the possibility of abuse is rampant. And it cannot be controlled merely by denouncing and forbidding manipulators. That must be supplemented by a good-natured skepticism toward one’s own intuitive judgments.
In an environment that clearly contains external practices meant specifically to hack your “gut feelings,” there must be counter-practices just to even the playing field. But this would require an attitude and a training that replaces belief in our own gut feelings and moral emotions with a kind of good-humored participatory skepticism.
And this is not just about protecting girls from bad sex guys. Last night, I saw a Google News article suggesting that Canadians in 2024 alone reported almost $700 million in losses to scams, impersonations, tricks, bad links, etc. The article said (and I just believed it) that authorities estimate that only 10% of these embarrassing events get reported. So multiply $700 000 000 by ten. Just in Canada. Just in 2024. That’s a lot.
Hard to believe that someone would respond to an “urgent warning” about their computer security by giving unknown-but-official-sounding people access to their computer. Actual officials always say not to do that. Purporting to protect you from the virus is how you get the virus.
Often, the elderly are targeted. Not just because of their undiagnosed brain plaque and notorious lack of internet savvy, but because they have spent an entire lifetime practicing being socially gullible. Their whole life was spent assuming that what was said-or-read was probably the truth. And if the person who scammed them gets caught, if people are not getting exposed to that one bad manipulator, then they might happily return to simply believing.
That is how most cultures have been run since the establishment of the Mesopotamian city-states.
PRO-TIP. Cool it down.
Anxious haste activates parts of the brain much simpler than our critical reasoning. Slow down. Maybe you get an email telling you to click on a link, or a text asking you to respond to a number that seems like it’s from the police, the tax department, etc. Maybe you even get a phone call from a voice you recognize. Do not act immediately. Check it with your own search first. Take some time. Cross-reference it with something else. Tell yourself it’s okay to decide about this in a little while. There is a risk in hesitating, but often a greater risk in not hesitating. And you're really getting excited at how amazing your date smells… wait till the second date before doing anything about it.
VIII. COGNITIVE BIAS & SERIAL KILLERS
I have been casually studying the pathology of serial killers since I was an adolescent. In the 1990s, there was a wild chasm between the data collected in interviews with these monsters and the general psychological assessments of “experts” like Dr. Robert Hare. Today, there are many richer, saner, and more daring books like My Life Among the Serial Killers and The Wisdom of Psychopaths. Generally speaking, I would distinguish a specific pathological condition of serial killers (whom I sometimes call necromancers) that is not the same as psychopathy or sociopathy, although these may frequently overlap.
The key feature of the serial killer, for me, is a kind of limbic fusion. What I mean is their brain’s inability to clearly distinguish between different basic animal drives. The impulses to attack, eat, fuck, flee, skulk, and hoard are activated as a single mass. A trigger in one circuit brings all the others along.
A healthy brain has layers of delicate capacity that can distinguish between these different drives. We typically do not copulate with corpses nor cause our sexual partners to become corpses and then eat bits of them. We do not lurk around crime scenes, hoard sadistic trophies, or interpret pornography as permission to rape and murder. These are signs that something has gone terribly wrong.
And at the heart of this terribly wrong thing is an inability to viscerally distinguish between distinct valuable functions. If we extend the basic architecture of that problem, we find something very interesting about our naive minds...
In Kahneman’s Thinking Fast and Slow, System 1 is a name given to an old form of consciousness that does very fast, very calorie-efficient thinking. It just throws you a convincing idea for free. In some areas of life, this is great. Gladwell’s book Blink explores situations in which this quick knowing is excellent.
When circumstances remain fairly constant and you have had lots of experiences in which you got immediate feedback about success or failure, you probably have a finely tuned System 1. A skateboarder, sculptor, or automobile driver develops fast and mostly accurate responses that do not require any pondering. But novel situations, delayed feedback, and complicated sequences reveal horrible flaws in System 1.
Here’s the classic example:
If a bat and a ball together cost $1.10 and the bat itself costs one dollar more than the ball, then what does the ball cost?
10 cents?
Of course not. If the ball costs 10 cents, then the bat (which costs one dollar more than the ball) would cost $1.10 on its own. The ball has to cost 5 cents. But why do most of our brains quickly and easily get the wrong answer?
Actually, in my opinion, we do not get the wrong answer. We get the correct answer to the wrong question. Our old, fast thinking interprets the problem incorrectly. It accurately answered the question, “What is the difference between $1.10 and a dollar?” It produces the correct answer: $0.10. But why did it answer the wrong question?
Why — given a jar containing an unknown number of jellybeans — would your brain ignore the prompt to “guess the number of beans” and instead answer the question, “What number have you seen recently?” This is the anchoring fallacy and the recency bias. We tend to assume that any complex, unknown situation is probably pretty close to the last specific amount we heard about. Dumb. Weird. Natural.
All of these cognitive biases, which neuropsychology has now mapped out in a pretty thorough way, seem to rely on a shared assumption that leads to misinterpreting the task. The assumption is that all values are the same value. True. Good. Recent. Easy. Familiar. Famous. Tall. Official.
The old mind does not know the difference.
A piece of information that is easier for the brain to find feels more true. Taller people, on average, get higher salaries when they are clearly not more competent. We think food falling on our own kitchen floor is safer to eat than food falling on someone else’s floor. It is more familiar. Local. We also act like a person who “looks presidential” is more qualified to be president. Rationally, these variables are not related. Yet our old brain treats all these different values as if they were just versions of the same One Value. A hit in one zone translates to a hit in every other zone.
This is the opposite of what Integral Philosophy calls the “different lines of intelligence” (an attempt to separate types of value and capacity) but, interestingly, it IS very similar to the pathological brains of serial killers.
Trusting and respecting someone who is famous (i.e., easy to think of) is analogous to killing and eating someone to whom you are sexually attracted. It is a disgusting conflation of variables that have nothing to do with each other. It is a failure of the meta-intelligence that is supposed to exert energy to differentiate these spheres of value.
How could Marianne be a serial killer? She was so nice, quiet, familiar, well-dressed, etc. Well, these variables are NOT related. Beautiful things can be false. Good things can be ugly. True things can be bad for us. This was Nietzsche’s critique of populist Platonism. While there is contemplative value in attuning to the phenomenological quality of valuableness itself, we cannot treat the types of value as interchangeable or mutually validating. If we do, that counts as “cognitive bias.”
And just as we can be manipulated by serial killers and psychopaths, we manipulate ourselves by answering one kind of value question in place of another. Instead of solving the problem, “Is this safe?” our brain solved the problem, “Is this person flattering me?” Why not? After all, safety and praise are both… good things.
IX. THE AI HOLE & CULTWORLD TM
People fall off the social chessboard. In the 1960s, my dad had a friend who got into Scientology. He vanished. Nobody from his hometown ever heard from him again. This is pretty common. Especially with communities that urge you to cut ties with “problematic people” in your life. “Problematic” often includes family and friends. Just ask around. But this is not just about cults.
As the great Radiohead song told us, “We do it to ourselves. That’s why it really hurts.” We now have terminology for people who go down rabbitholes. People who get radicalized by their own YouTube algorithm or drop out of human space through obsessive interaction with the pseudo-linguistic responses of an AI interface. Headlines about people falling in love with an automated app or getting brainwashed into a religion that their responsive chatbot just invented are going to get way more frequent.
This is a tricky problem.
It clearly requires a finessed form of mutual concern and awareness. We have to be able to intervene when people are sliding into a place of no return, or in which their temporary dalliance might destroy their relationships, resources, and integrity. Yet this has to be done with the realization that people have the inalienable right to explore strange material, take cultural and emotional risks, and even to place themselves in jeopardy.
With some legal and moral constraints, we are ethically obligated to honor the sovereign agency of adult citizens by allowing them to make bad decisions. And despite our strong feelings on the subject, we honestly do not know all the relevant factors in a complex world, nor do we clearly grasp the effects of things over longer ranges of time. We congratulate ourselves on giving desert or bread to hungry children with no thought about the long-term effects of processed carbohydrates or the function of fasting.
We can and should tell people our concerns about their behavior, if they are willing to listen, but we also have to let them do things we think might doom them or disgrace them. In addition to asking whether they are joining a good or bad cult, we might also need to ask whether it is or is not okay for them to throw themselves away. This is akin to the problem of deception, seduction, and manipulation in ourselves. Is it okay that we get duped or misled or drawn into something?
This is where it gets personal for me again.
I have said many things in this article about the need to produce a more benignly suspicious education because we cannot solve the “manipulation and bad exposure” problem merely by denouncing and blocking the bad people and bad ideas. The whole thrust of this piece is an invitation to wander around attentively in the complexity of our own complicity with persuasion and suggestion. This piece attempts to playfully and philosophically normalize a stronger awareness of manipulative tactics, INCLUDING the tactics that our own minds use against us.
We have to be less suggestible. That makes sense to me. I have often been unsurprised by subtle manipulation tactics. I have also grown a better conscience, on purpose, about being misled, misunderstood, seduced, and deceived. It does not have to be as threatening as it initially seems. Okay. But why am I like this?
Perhaps I grew up with a mindful distance from the assumed value of social communication around me, but perhaps I also had a deep anxiety about being led, misled, and deceived.
Can I treat myself as a responsible agent who might want to be misled? As one who enjoys the simplicity of trust? Who has sometimes taken a risky human pleasure in jumping to conclusions, or blaming the obvious, or not questioning the context of a piece of information, or welcoming the deceiver into my heart? What kind of self-compassion does that point toward?
In our journey toward personal and collective sensemaking, in a field that will inevitably be entangled with various degrees of acceptable and unacceptable manipulations, can we — without ceding our vigilance, protective instincts and increasing casual knowledge of suggestive tactics — still honor each other and ourselves as beings who make dangerous cognitive gambles to live in a world simple enough to get turned on by and take action within?
Let us be curious about, astonished by, and kind toward minds and hearts that do not always opt for accuracy and security.
And, in conclusion, to quote Nietzsche from his Prelude to a Philosophy of the Future:
O sancta simplicitas! In what strange simplification and falsification man lives! One can never cease wondering once one has got eyes for beholding this marvel! How we have made everything around us clear and free and easy and simple! How we have been able to give our senses a passport to everything superficial, our thoughts a godlike desire for wanton pranks and wrong inferences! How from the beginning, we have contrived to retain our ignorance in order to enjoy an almost inconceivable freedom, thoughtlessness, imprudence, heartiness, and gaiety—in order to enjoy life! And only on this solidified, granite-like foundation of ignorance could knowledge rear itself hitherto, the will to knowledge on the foundation of a far more powerful will, the will to ignorance, to the uncertain, to the untrue! Not as its opposite, but—as its refinement!
It is to be hoped, indeed, that LANGUAGE, here as elsewhere, will not get over its awkwardness, and that it will continue to talk of opposites where there are only degrees and many refinements of gradation; it is equally to be hoped that the incarnated Tartuffery of morals, which now belongs to our unconquerable "flesh and blood," will turn the words round in the mouths of us discerning ones. Here and there we understand it, and laugh at the way in which precisely the best knowledge seeks most to retain us in this SIMPLIFIED, thoroughly artificial, suitably imagined, and suitably falsified world: at the way in which, whether it will or not, it loves error, because, as living itself, it loves life!
After such a cheerful commencement, a serious word would fain be heard…
