I initially published this post in September 2021, with some additions or editing every month or two since then.
If you don't have time for a longer
read, I encourage skipping ahead to the end, where I discuss ideas about what
we can do about the problem of vaccine refusal.
Introduction
Political polarization, propaganda,
and conspiracy theories have caused the world great harm in the past few years.
A related problem has come up recently, with a significant minority of
people refusing COVID-19 vaccination, leading to the pandemic lasting much
longer, claiming many more lives, depleting and exhausting workers in the
health care system, and causing much more economic damage.
Another round of this came up in Februrary 2022, with convoys of protesters rallying supporters to challenge vaccine mandates and other public health measures. Protests of this sort are demoralizing and infuriating to those who are trying to help. It is like a town threatened by a giant forest fire, with burning embers blowing into the neighbourhood, but protesters tired of restrictions intimidating the firefighters, surrounding the fire stations with horns blaring (waking tired families and children trying to sleep nearby), and demanding to reclaim their right to have open fires.
In this post, I will explore the
psychological and social factors contributing to these problems, with
suggestions of things that individuals, community organizations, companies,
church groups, and governments can do to help.
For a brief video introduction to this topic, I recommend a recent short Netflix documentary, The Mind: Explained, the episode called "Brainwashing."
A highly recommended book about how to communicate effectively with people having extremely polarized or conspiratorial beliefs is How Minds Change by David McRaney (2022).
Polarization
It has become more common for people to hold extreme political views. There are increasingly hateful and intolerant attitudes towards political opponents. Many of us are familiar with the 2014 study done by the Pew Research Center, showing this polarization gradually worsening in the U.S. since 1994.
Propaganda
Propaganda is false, exaggerated, or
misleading information spread for political or manipulative purposes. Many
large news organizations in the U.S. support a particular political party,
leading to unprecedented exposure to biased information consumed by nearly half
the population. Social media sites such as Twitter and Facebook often lead
people to obtain information only from like-minded others. Not only does this
lead to extreme bias, it also builds a community of online friends or followers
who "egg each other on," ideologically or personally, while denigrating
opponents often in a mocking or hostile way.
Beginning in March 2022, the world has once again seen the most horrific propaganda of all, that which persuades an entire nation to support a war against its neighbour, while preventing its citizens from seeing or understanding the atrocities being committed.
Conspiracy Theories
Conspiracy theories have become more
common and more bizarre, often associated with ideological positions or a
particular political party. While most of us have had a sometimes amused
tolerance for people holding these beliefs, conspiracy theorists are now more
organized, can magnify and spread their ideas using social media, and have
managed to influence public policy to some degree. I am aware of people in
public positions who seriously believe that COVID-19 vaccines contain
microchips used to track people, and that Bill Gates is somehow responsible for
this. Others believe the moon landing was faked, or even that the earth is
flat.
Ideas of this sort are now very prevalent, with social media and other news sources contributing to their spread. Such misinformation can often be presented in a professional manner, as though it is valid documentary reporting. This attracts many followers who then continue to spread these ideas.
Anti-Vaxxers
We are all weary of the COVID-19 pandemic.
Millions have died or suffered severe disease, and many others have had severe
financial losses. Many more are going to die. Most COVID patients will recover
fully, but a significant minority of survivors will have long-term health
consequences (so-called "long covid"), with symptoms such as chronic fatigue and respiratory problems.
We have vaccines that can cause large reductions in pandemic-related severe disease and death. The rapid development and mass distribution of vaccines since 2020 is one of the most outstanding scientific achievements in
history. We also have other knowledge about control of viral respiratory
disease, such as about mask usage, ventilation improvement, frequent home
testing, etc. which, together with vaccination, could have brought our countries out
of the pandemic much more quickly, with much less economic hardship, with much less psychological or social hardship, and with much less loss of life.
No vaccine is perfect. There are small risks of side effects, including some rare serious problems; but these risks are much lower than risks caused by COVID itself. Also, very few vaccines lead to perfect "sterilizing" immunity. The main impact of COVID vaccines has been to reduce the probability of severe disease, hospitalization, and death from COVID, while having a much smaller effect to reduce milder disease. This protective effect also gradually fades over 6-12 months, though probably does not disappear entirely, requiring booster vaccinations at this point, while new COVID waves are continuing. There is accumulating evidence that the vaccines also reduce the probability of developing "long covid." Because of high vaccination rates in Canada, especially in older cohorts, we narrowly averted the disastrous hospital and critical care overflow that would otherwise have occurred in the waves before 2023. Vaccine development, like any other human process, is never perfect either; there may be many steps of the COVID vaccine development story, such as political or economic issues, to criticize in some way. But this should not distract us from appreciating how these vaccines have saved more lives, prevented more years of life lost, than almost any health intervention in our history.
But a significant minority of people refuse to
be vaccinated, refuse to use masks, and even refuse to acknowledge that the
pandemic is a serious problem. Those who refuse are more likely to belong to particular political or religious groups, are more likely to watch particular news channels, and are more likely to have less education.
Unvaccinated people from 2020-2022 were much, much more likely to require hospitalization, including intensive care. At this point, recipients of recent vaccine boosters continue to have a much lower hospitalization rate. From 2020 to early 2022, hospitals all over the world were filled to capacity, and beyond, by unvaccinated COVID patients. If everyone had been vaccinated, there would still have been hospitalized patients (since the vaccines are not perfect), but there would never have been overflow or a major strain on the health care system. Because of inadequate vaccination rates, everyone else (vaccinated or not) had much more limited access to medical services, including elective surgery and ICU. Because of people who refused vaccination, it became much more dangerous for all of us to have a heart attack or a case of appendicitis. Furthermore, the staff in the hospitals were overworked to the point of exhaustion, the horror of the situation magnified further by the needlessness of it. Vaccine refusal caused extreme harm to health care workers all over the world, and led the health care system itself to the brink of total breakdown.
A new horrific development following all of this, is the increasing prevalence of refusal to get vaccines against previously eradicated diseases such as polio and measles. If this continues, it is likely that we will once again see children needlessly paralyzed or killed from preventable infectious diseases, for the first time in decades.
Anti-vax beliefs and other bizarre
beliefs about COVID can be shockingly extreme and unchangeable: we have many
examples of people remaining convinced that COVID is a hoax, right up to the
moment of their death from respiratory failure in an ICU bed. Medical
colleagues of mine, working in rural areas with low vaccine uptake, have
described many stories like this during the pandemic. There are horrifying
examples of hospital workers being threatened or attacked by people convinced
that the medical care is somehow harmful. There are examples of public health officials who are afraid to advocate for vaccines, due to having received intense harassment and even violent threats.
Alternative Medicine
In many cases, bizarre beliefs about
COVID are an extension of unusual ideas about health care. The alternative medicine industry has a market size of about $100
billion per year. Parts of this industry harmlessly promote healthy lifestyle
habits, nutrition, or evidence-based care, but there are a lot of
exaggerated or false claims made in the sales of alternative medical services
and products. Alternative health care can involve bizarre or even delusional
beliefs about illness. There can be distrust
for evidence-based medical science, and loyal allegiance to the alternative
practitioners despite harmful practices.
I realize it is excessive and unfair to simply condemn alternative medicine. Many people have had bad experiences of conventional health care. And many people have experienced kindness, generosity, support, and helpful guidance from alternative practitioners. Many alternative practitioners have wisdom in particular rehabilitative techniques, and some knowledge of conventional medicine. Some supplements or alternative remedies do have a reasonable evidence base, sometimes on par with standard treatments. Conversely, conventional physicians sometimes recommend treatments including prescription drugs or surgery which in some cases have a questionable foundation in evidence.
Problems in modern medicine, including expense, access problems, or brief, impersonal clinical encounters, can feed some frustrated people's pursuit of alternative health care providers who may have more time for empathic support or apparent understanding. Unfortunately, this apparent understanding is often based on fictional beliefs couched in pseudoscientific language that can sound impressive or convincing.
There are frequent examples of misguided beliefs in alternative health care practice, particularly when the practitioners are marketing expensive products, pushing bizarre theories of causation using expensive, unnecessary pseudoscientific testing procedures, fomenting distrust in conventional medicine, or discouraging patients or clients from seeking proven therapies, including medications or vaccines.
Con Artistry & Fraud
Many people with
strong opinions opposing vaccines, supporting quack treatments for COVID, or
supporting particular political leaders since 2016, have been conned -- that
is, they have been victims of fraud. They have been sold something that seemed
very attractive to them at the time, but the goods they've obtained are
worthless or harmful to themselves and others. But many people would feel an
embarrassing or humiliating injury to their pride to admit that they were
conned; so instead, they double down on their support for con artists
(including particular politicians) or quack remedies.
There is a fascinating
research literature on this subject. I would start with Maria Konnikova's book, The Confidence Game: why we fall for it...every time. Her book is a series
of case studies of various types of spectacular con artistry & fraud, with
some discussion of the psychology underlying this. The next scholar to be
acquainted with is Brooke Harrington, a Dartmouth College sociologist. I'm in the midst of reading through
this work. One of her questions has to do with justice: when should a person
who has been conned into doing something harmful be considered an offender requiring management in the criminal justice system, instead of only a victim
requiring compassionate care?
I see these issues as closely tied
together, fed by the same underlying causes. Together they are driving people
and nations apart; they have caused needless suffering, death, and economic
hardship during the COVID pandemic, and have led to an unprecedented threat to
democracy in some parts of the world.
These are not new problems: they have
been with us throughout history. Many of us associate propaganda with World War
II or the Soviet Union, not modern-day western democracies. Many of us
associate bizarre or erroneous beliefs about health with previous centuries, in
which people attributed disease to evil spirits, "excesses of bile,"
or an excess of blood in the body requiring treatment by bloodletting.
Unfortunately, bizarre beliefs about health are alive and well in modern
society.
It is crucial to understand and study
these problems, to know why they happen and what can be done to improve the
situation. A thorough analysis requires input from many fields, including from
historians, political scientists, sociologists, public health experts, and
psychologists.
Myside Bias and Motivated Reasoning
The most powerful factor, in my opinion, driving extreme political difference, extreme contrarian views about the pandemic, and "anti-vax" ideas, is what Steven Pinker, in Rationality: What it is, why it seems scarce, why it matters (2021), calls "myside bias," which then gives rise to so-called "motivated reasoning." This is the tendency to selectively attend to evidence which supports your pre-existing opinion, and to discount or ignore evidence which refutes your opinion. We are all prone to this. Your opinions, ideas, and beliefs about almost any issue can become associated with your identity, or your group, or your "side," with differences of opinion or contrary evidence representing some kind of a threat to your identity or values. Contrary evidence could even bolster your initial opinion further, almost as though your ideas are like a family or village being attacked by outsiders, leading to the community rallying and strengthening itself in response. Myside bias and motivated reasoning are greatly magnified in groups, and are caused or fueled by many of the factors which I will describe below.
Haidt: The Righteous Mind
I recommend reading Jonathan Haidt's book, The Righteous Mind: Why Good People are Divided by Politics and Religion--an excellent introduction to the
psychological factors which drive ideological differences. Haidt presents
himself as a moderate, or even a right-leaning moderate, which I think at the
very least should increase the readership and acceptance of this book across a
wider swath of the political spectrum.
Group Loyalty, Tribalism & Ingroup Bias
Haidt concludes that there is a human
trait of feeling loyal to groups; those groups with stronger or more frequent
loyalty traits among members will have advantages in survival and prosperity.
These groups will be more cohesive and better able to defend themselves against
outsiders. Some individuals value group loyalty above all other values; this is
partially a heritable trait. While loyalty is a virtue, it can also lead to
group members continuing or even fanatically increasing their loyal devotion
when the group is engaging in destructive or corrupt behaviours, even when such
behaviours are causing suffering to the group members themselves. The most
extreme examples of fanatical group loyalty are seen in cults, but variations
of this phenomenon are seen in daily life--in our families, our communities,
our sports teams, our religions, our political groups, and our nations.
We have seen groups with extreme
opposition to COVID vaccination harassing exhausted health care workers outside
hospitals. Other
groups participated in 2021 in an unprecedented mob attack on a world
capital. Yet members of these groups previously may have valued ethical
principles, such as fairness, hospitality, helpfulness, and the rule of law.
Fanatical group allegiance can cause members to stray towards behaviour that is
contrary to the group's previous fundamental values.
Groups containing devoutly loyal
individuals are likely to have higher hostility to outsiders. Loyalty is a good
thing, but in the setting of polarization, propaganda, conspiracy theories, and
vaccine hesitancy, such unthinking, rigid loyalty is destructive to others and
destructive to the group members themselves.
One of the suggestions Haidt has about
improving the problem of polarization is to maintain open dialogue, value the
principle of respectful debate, and foster friendships between people and
groups with different views. This would involve cultivating friendships between
those on the "left" and "right" of the political spectrum,
rather than devolving into hostility and becoming "enemies." But this
approach is not very helpful for dealing with fanatical or extremist groups; at
that point, friendly debate and social warmth will not be possible.
Unfortunately, many people holding
anti-vax beliefs and other strongly polarized positions have become too extreme
to allow respectful social connection. Yet there are many others whose
positions are moderate or ambivalent on these issues, including friends,
relatives, and neighbors of extremists. These are the people most amenable to
friendly engagement.
The psychology of Conspiracy Theories
Lack of feeling in control, need for certainty
According to psychologists studying
this area, such as Van Prooijen and Douglas, conspiracy theorists often feel a
lack of agency or control, a need to make sense of complex or confusing
situations going on in life or in the world, a desire for being respected (but
not feeling that such respect is being given), and a need for certainty. Like
other delusions or overvalued ideas, conspiratorial thinking can give rise to a
feeling of relief, since there is an explanation about why a problem is
happening, even though the beliefs are fictional. The explanation, and the excitement
of being part of a select group of fellow believers, can give back some feeling
of control or certainty, a new sense of purpose. Other people's skepticism
could be perceived as a noble challenge to be faced.
In helping this problem therapeutically, people trapped in conspiracy beliefs need to be shown personal respect and empathy, before any attempt to challenge or refute the false beliefs.
Past Psychological Adversity or Trauma
Prior psychological
hardship can sometimes drive people into a fearful, angry, hateful,
distrustful, or even paranoid state, with relief of ongoing psychological
stress found in narrow or rigid ideologies. Others, including refugees, may
have understandable reasons not to trust authorities or the government. Neuroscientist Nafees Hamid has shown that experiences of social exclusion or discrimination contribute to radicalization. In some cases, people with a history
of trauma or social rejection will find comfort, support, and belonging in
groups, such as churches and other community organizations, or extremist fringe
groups, even if these organizations are engaging in extreme polarization or
conspiracy beliefs. Members of these groups will naturally feel protective and
loyal towards the group and the group's beliefs, even if these beliefs are
causing harm to others. Therefore, some people develop anti-vax beliefs as a
result of their past trauma.
The possibility of past trauma should always be
kept in mind when dealing with someone who is trapped in a conspiracy theory
mindset, with compassionate support offered, while always challenging the false beliefs.
Personality Disorders
Personality disorders are common,
affecting several percent of the population, with milder symptoms affecting
many more. They cause lifelong disruption in relationships, behaviour, and
emotional stability; people with personality disorders often lack insight that
they have a problem. They are caused by a combination of hereditary factors and
long-term environmental adversity, such as childhood abuse.
Many conspiracy theorists have
narcissistic personality: they believe they are better, more insightful, more
informed, and more intelligent than other people, and that other people's
skepticism or rational arguments are signs of stupidity or inferiority. They
are unable to tolerate critical feedback. A softer type of narcissism is due to
unmet psychological needs to feel unique. When extreme narcissism is present in
a major world leader (as was the case starting in 2016) the entire group of
followers can adopt a narcissistic attitude, even if these traits would
normally be abhorrent, or entirely at odds with the group's previous religious
or ethical standards.
Another factor is obsessive-compulsive
personality. Here, there is a rigid understanding of moral issues, a tendency
to be quickly and firmly judgmental, and a tendency to favour a polarized view
of issues. Again, such character traits would generally be difficult to tolerate,
but when present in a charismatic leader, they become endorsed by the group
itself.
Schizotypal and paranoid personality
disorders can also lead to conspiracy theory beliefs. With these personality
variants, people have low-grade delusional beliefs, magical thinking,
superstitions, and mild paranoia.
Finally, there is antisocial
personality, which leads to criminal behaviour, a lack of empathy, callous
disregard for others' suffering, manipulative behaviour towards others, and
compulsive lying, despite showing superficial charm. We have seen this factor
in a major political leader since 2016 and in many con artists profiting from
the pandemic.
It should be noted that many conspiracy theorists do not have personality disorders. They have been swept into false beliefs due to misinformation and group allegiances, but are otherwise mentally well, sometimes well-educated and intelligent.
In order to help people who have personality disorders, compassionate understanding is required. There are various therapeutic systems that are useful, including CBT and DBT. Motivational interviewing techniques are also likely valuable. Sometimes medications could be of some modest help, at least to reduce specific symptoms such as anxiety, irritability, or low-grade paranoia. But in order for there to be any possibility of therapeutic help, there would need to be a safe, stable therapeutic frame. If a person is angry, volatile, or behaving dangerously, therapy is impossible unless there can be strict boundaries guaranteeing safety. For some people, these boundaries can be difficult or impossible to negotiate. Furthermore, many people suffering from personality disorders lack insight about their problems, and lack the desire to work on personal change in a therapy setting. The first step, in these cases, is often to impose limits on negative behaviour. This is why antisocial personality, for example, usually needs to be dealt with in the justice system.
Low Education, Innumeracy, & Lack of knowledge about the world
Many conspiracy theorists have lower
levels of education, lower levels of intelligence, and a desire for accuracy or
meaning but a lack of the cognitive tools to find this rationally.
Innumeracy, a lack of scientific
knowledge, a lack of statistical knowledge, and a general lack of knowledge
about the world (for example, about history, culture, geography, or economics)
are significant factors contributing to poor personal and political decisions.
Even relatively intelligent people who are not broadly educated and informed
are more prone to ingroup biases and conspiracy theories.
Ellen Peters' book Innumeracy in the Wild: Misunderstanding and Misusing Numbers is a detailed account
of poor mathematical skill in the population. She shows that only a small
minority of people have the skills needed to accurately interpret data, and to
correctly guide decision-making. As a result, most people either make erroneous
conclusions about data, or are dependent on others to interpret the data for
them. This makes people vulnerable to political influence from people who
misconstrue data. These influencers may have a deliberately manipulative goal,
or may be inadvertently misleading because they are also innumerate.
Without aptitude in
science, critical thinking, and reason, verbal ability alone does not protect
against being drawn into ingroup biases or conspiratorial thinking. Brittany
Shoots-Reinhard, Ellen Peters, and others have done a lot of work over the past
decade looking at the relationship between intellectual ability and
decision-making. They recently published an article showing that people with higher verbal ability are more likely to have polarized responses to COVID-19, and to consume more polarized media. Numerical skill did not predict higher polarization.
This suggests that people often use
verbal intelligence, not to improve their reasoning or judgment, but to more
efficiently gather information that supports their pre-existing views, often
ideological and determined by ingroup biases. This is especially problematic at
a political level, since verbal intelligence is a more important skill than
numerical or scientific intelligence for a politician to be successful, or for
a celebrity to be influential. Therefore, we have people who are more likely to
have polarized beliefs holding positions of influence in society.
If we see studies looking at the
effects of education on various psychological phenomena, beliefs, or
ideologies, we need to look at the type of education, with a particular look at
numeracy, logic, and reasoning skills, as well as the degree to which the
education contains subject matter about global issues, such as history,
geography, environmental science, economics, etc.
It should be noted that some individuals swept into conspiratorial thinking are intelligent and well-educated, and may have good analytical skills. They are influenced in their misinformation journey by other factors such as group allegiances. They may use their intellectual skills to spread misinformation more effectively, or even become seen as experts in their communities.
The internet and news-bubbles
The internet provides a medium in
which people with extreme beliefs can easily form a community, which in
conjunction with traits for group loyalty, leads to them forming a strong
identity, an "us vs. them" mentality, and a resistance to rational
evidence from outside the group.
It is not enough to address this
problem on a one-on-one basis. There are political, economic, and educational
factors that are likely to help, on an individual and societal level. I'll come
back to this later.
Polarized News Sources & Propaganda
Major news networks in various parts
of the world are deliberately propagating conspiratorial thinking and fomenting
polarization, catering to entrenched members of particular ingroups. These
networks have a profit motive, but the owners of the networks are also driven
by ideological beliefs to push this to further extremes. They are popular and
tend to have high ratings, especially when they are denigrating ideological
opponents in a dramatic way. These news networks lack any form of regulation
that prevents or limits harm (particularly in the U.S., after the removal of
the FCC fairness doctrine in 1987).
Unfortunately, this has led to a steep
decline in the quality of news information that many people are consuming. Fans
also form an ingroup loyalty to the news service itself, such that mainstream
news may be deemed "fake" or biased. Many fans normally value
kindness, civility, education, politeness, the rule of law, balanced debate,
and religious beliefs rooted in love and compassion. But due to powerful
ingroup loyalty effects, the fans of these news services can embrace leaders or
pundits who are unstable, mean-spirited, and bullying.
It is important not to underestimate
how powerful and destructive propaganda can be; we have to realize that the
freedoms we have enjoyed in modern democracies can be quickly eroded under the
influence of powerful and well-financed propaganda efforts.
Cognitive Biases
Cognitive biases are
"shortcuts" of thinking which allow us to make decisions more
quickly. These shortcuts can be useful, since we don't always have the time to
analyze every issue in our life in detail. But they can cause massive errors in
judgment, especially when we are not even aware of them. For an introduction to
this area of psychology, I recommend reading Daniel Kahneman: he is the one
psychologist to have won a Nobel Prize. His book Thinking: Fast and Slow is fun to read and a summary of Kahneman's masterful research. I'd like to review some of the more common cognitive biases which perpetuate conspiratorial thinking, political polarization, and ideological extremism:
Reactance
Reactance is the urge to do the opposite of what
someone wants you to do to resist a perceived constraint upon your freedom.
This has been one of the driving factors causing resistance to pandemic-based
public health restrictions and vaccinations, and which drives political
polarization more generally. A component of the reason many people are refusing
vaccination or defying pandemic restrictions is reactance or defiance, because
they don't like being told what to do, especially by people who they may see as
outside their ingroup.
Reactive Devaluation
Reactive devaluation is the tendency to devalue an idea or
a proposal, only because the idea comes from an opponent. So almost any idea
coming from a political opponent is reflexively devalued and opposed,
regardless of whether it is rational, correct, or helpful. If the same idea had
come from an ingroup member, it would be approved enthusiastically. Reactive
devaluation is profoundly self-destructive, not only to individuals, but to
entire nations. Unfortunately we see this daily in U.S. politics. Once again,
this is a reason many people oppose advice about vaccination or public health
measures.
Projection
Projection is attributing to other
people the feelings or problems that you have yourself. For example, you may
feel angry with someone, but in a conversation you may have a strong belief
that it is the other person who is angry at you. While projection is not
typically considered a cognitive bias, it is a common psychological mechanism
among those with personality disorders, and among con artists. In the former
group, projection is often "unconscious"-- that is, people project
without even realizing they're doing it. It would be an issue to be addressed
in psychotherapy. In the latter group, it is used deliberately and consciously
as a manipulative technique. A well-known political leader after 2016 could be
seen to engage in both forms of projection every week--accusing others of bad
qualities or behaviours that were obviously his own.
In a conversation with someone holding
fanatical anti-vax beliefs, you may encourage the person to be more informed of
evidence. But that person will project: they will claim that it is you who are
not aware of the evidence! They will deny being conned themselves, but will
claim that it is you who have been conned! Many anti-vaxxers are calling people
who follow public health guidelines "sheep," while it is the
anti-vaxxers who are often passively swept up in mindless herd behaviour.
The Availability Cascade
The "availability cascade"
and the "illusion of truth effect" refer to the tendency to believe a
statement simply because it has been repeated frequently, or because it is easy
to understand, even though the statement is false. Many beliefs about the
pandemic, including those from conspiracy theorists or those from the
"anti-vax" groups, seem more believable simply due to frequent
repetition. The staggering daily abundance of frank lies emerging from a major
world leader from 2016-2020 were often not perceived to be lies by many people,
due to the frequency of exposure and the cognitive ease involved in processing
such statements. Or sometimes people did not care that they were lies.
Sometimes hateful speech is unfortunately too easy to process cognitively; it
may appeal to some deep, primitive component of our brains that is excited by
rage and deprecating others.
Confirmation Bias
Confirmation bias is the tendency to only look
selectively at evidence which supports a previous position. This is driven
partly by powerful ingroup loyalty. Even when there is overwhelming evidence to
support a contrary position, people suffering from confirmation bias will often
remain stubbornly insistent that their own narrow, outdated, or invalid
research findings are correct.
Anchoring
Anchoring is the tendency to stick with an
initial position or estimate, or to be swayed by it strongly. If you have
started having a particular belief, there is a tendency to maintain it. This is
particularly true if there are personality traits valuing consistency,
commitment, and loyalty more strongly than traits valuing rationality,
compassion, or wisdom. One can become irrationally "loyal" to
initially-held ideas (such as about perceived harms of vaccines. or about
supporting a political leader who proves to be dangerously unstable) even if
these ideas are self-destructively inaccurate and contrary to other values.
The Dunning-Kruger Effect
The Dunning-Kruger effect and the "overconfidence
effect" refer to a tendency for unskilled people to overestimate their
ability. We see this with many people making strong claims about specialized
areas (such as about epidemiology or virology during the pandemic) despite
minimal expertise. Unfortunately, such people can be quite persuasive, not
because of their expertise, but because they may be popular and have a loud or
persistent voice. On the other hand, many experts may have a rather modest
voice, and therefore their accurate messages are under-amplified.
Present Moment Bias
"Hyperbolic discounting" or present moment bias, is the preference for immediate
payoffs relative to later payoffs. On an individual level, this reflects a lack
of self-control when faced by temptations. On a community level, it leads to
neglect of long-term societal needs, such as health, environmental integrity,
and education, in favour of immediate profits, even if such profits cause
severe long-term pollution, economic damage, or health damage. We see this in
the pandemic management as well--many are unwilling to make a short-term
sacrifice (such as maintaining social distancing or mask use) even though such
small sacrifices would lead to much larger longer-term gains in health,
prosperity, and survival for themselves, their families, and their communities.
The Sunk Cost Fallacy
The "irrational escalation" fallacy or sunk cost fallacy is the tendency to continue investment in a
decision that was made previously, despite new evidence that the decision was
wrong. Basically, it can be humiliating or injurious to pride to change one's
mind, so it can feel easier to hold onto one's mistaken views or decisions
rather than change them.
Normalcy Bias
The normalcy bias is the refusal to plan for or react to
a disaster which has never happened before. If you live in an earthquake zone,
but have never seen or experienced an earthquake, you are less likely to
consider how to survive an earthquake or protect your home. It is much less
likely that you would undertake expensive large-scale disaster preparations.
This phenomenon has happened with COVID. Many experts were well-prepared; there
were even organized national preparations for pandemics, but some leaders of
major governments dispensed with all of this. The same problem is likely to
happen on a much worse scale regarding the ongoing degradation of the earth's
environment (disappearance of forests, mass extinctions, degradation of
fisheries, loss of wildlife habitats, and climate change). Once a disaster is
already underway (such as a house fire or earthquake or flood or pandemic or
climate change) it is much, much harder to reverse the situation; it becomes
much, much more expensive if not impossible to find a solution. Prevention is
much, much more affordable and efficient than expensive disaster management.
The Ostrich Effect
The ostrich effect is the tendency to
ignore an obvious negative situation. Once again, we saw this in a major
country upon the outbreak of COVID, and we see this with the environmental
& climate change problems. On a personal level, we see this in the tendency
for people not to seek medical help when they notice a serious problem, just
hoping that it will go away on its own. It is driven by some combination of
fear (in this case fear of the truth and fear of how difficult the treatment
might be), and magical thinking (i.e. somehow believing that if you don't look
at a problem, then it will go away).
What to Do About Cognitive Biases
Daniel Kahneman, the world's leading expert on cognitive biases, is doubtful that we can eliminate cognitive bias. Even people who are very well-educated about this issue are still prone to bias, just like everybody else. The best we can do is educate ourselves about this, be watchful for bias in ourselves and others, collaborate together to make better judgments, and be open to feedback from others on this issue. Whole communities should be open to critique from other communities, and not try to shut down debate or discussion.
------
Similarity to Addictions
Ideological bias and conspiracy
thinking have a lot in common with addictions, since they are harmful to
individuals and communities, but hard to escape. People often dabble with
polarized or conspiracy-based ideas a little bit at first, often influenced by
psychological adversities, family or peer culture, and genetic risk factors,
then become more and more drawn into problematic behaviour over time; in this
way, it is like someone trying cocaine with their friends a few times per year
at parties, then escalating gradually towards weekly, then daily use, all the while justifying the behaviour as harmless, enjoyable, or a cultural norm.
Addictive behaviour can cause deep
satisfaction or relief in the moment; moving away from addictions can be
challenging and painful; people often cannot do it without external help.
Furthermore, many people with moderate to severe addictions deny they have a
problem, and do not see any reason to change. They may see their addictive
behaviour as simply a lifestyle choice, enjoyed by many friends, with any
problems lying with other people who criticize them.
Addictions are strongly entrenched by
a peer group of fellow addicts. Moving away from addictions often requires
that people let go of their current social network, leading to feelings of
loss, loneliness, boredom, and a lack of meaning. This is one of the reasons
that we have to offer social and community support to people if we would like
to help them move away from entrenched polarization or ideological biases.
Biases & Educational factors beginning in childhood
Many biases and educational factors
causing people to be trapped in a narrow or hostile ingroup begin during
childhood, with parents, family, and community members teaching and influencing
the children. Many people believe things only because their parents, teachers,
and peers believed them. After childhood, people will be more likely to
associate with, befriend, marry, or have children with others who are similar; this further entrenches previous beliefs
and makes differing belief seem strange or wrong.
Heredity
There is a hereditary influence on the
tendency to be dogmatic or stubbornly adherent to ideologies, and on general
intelligence. But hereditary predispositions are never absolute, and are never
purely good or bad. Hereditary factors, if channelled through a healthy
environment, can lead to good individual lives and a healthy community.
Refusal to admit mistakes
One last huge psychological factor is
refusal to admit mistakes. Many people would rather carry on with a previous
decision even if it is leading to disastrous results. They would be
embarrassed, ashamed, or would not "save face" if they had to admit
they made a terrible mistake, or if they had to reverse their position on an
important issue.
This stubbornness can be an extremely
powerful factor; it could be a psychological defense, a way of protecting a
person against the need to feel intense shame and regret for past decisions
which caused terrible harm. This phenomenon is fed by some of the biases listed
above, such as the sunk cost fallacy, anchoring, and ingroup biases. Instead of
owning up to a bad decision, people will go through a remarkable feat of
denial, to persuade themselves that they didn't make any mistake at all. Many
people hold onto strong anti-vax beliefs or conspiracy theories for this
reason. They might be willing to change their mind, but the cost of admitting a
big mistake is too high.
Well-funded corporate groups & "think tanks"
Wealthy corporate donors with strong
ideological positions are funding marketing campaigns and employing the small
cohort of contrarian scientists to push policies opposing vaccination, public
health measures, environmental protections, and other public policy ideas they
see as relevant to their ideologies or profits. These corporate groups or
"think tanks" have members who are part of the political or religious
ingroups described above. Their biases are not just individual, but organized,
powerful, and very well-funded, often with billions of dollars of financial
support.
Oxford-trained Duke University public health scholar Gavin Yamey has warned us about the influence of such groups, and has compared their tactics to those used in past decades by the tobacco industry: denying or twisting health risk data, to plant seeds of doubt in the population, in order to maintain profits of a multi-billion dollar industry despite the terrible harms it caused.
External Political Interference
Other nations with antagonistic
relationships with our own are attempting to propagate conspiracy theories and
extremist groups, mainly using social media, in an attempt to disrupt or weaken
our nations. This is a national defense issue.
What to do about it
There is a lot that can be done about this problem:
1) How Minds Change
David McRaney, in his 2022 book How Minds Change: The Surprising Science of Belief, Opinion, and Persuasion, describes techniques to communicate with people who have strongly entrenched beliefs. I strongly recommend this book. The ideas are based on showing respect, empathy, establishing rapport, sharing personal stories, searching for higher principles in common, gentle inquiry about the person's sources of evidence for their belief, and honest exchange of your own view, but without arguing or showing anger or contempt.
2) Inoculation against Misinformation
Van der Linden et al. have shown that so-called "inoculation" techniques can protect people from misinformation.
(see link). This involves exposure to videos explaining how misinformation techniques work. They have even devised an online game called
"Bad News" (see link) which helps people see how powerful misinformation techniques can be, in a gamified form. This technique is similar to the best ideas from CBT (cognitive-behavioural therapy): the therapy requires exposure, following informed consent, to the heart of the problem, in order to overcome it and be protected from future adversity.
3) Massive campaign to provide information & counter misinformation
According to MIT post-doc Ben Tappin, people differing due to an ideological
divide are still persuadable using reasoned arguments. A major reason
ideological divides lead to such tenacious resistance to change is the lack of
exposure of ingroups to such reasoned argument, and the extreme prevalence of
ingroup exposure to false arguments. Therefore, it makes sense to keep up our
efforts to provide accurate information, and not give up or become resigned,
believing that anti-vaxxers or other ingroup members are not persuadable.
We cannot only have health experts,
such as government health officers, speaking to the public. Many anti-vax
people will not be persuaded at all by a public leader. We need to have
spokespeople in the information campaign representing ingroups associated with
the anti-vax movement. We need to have right-wing political leaders, religious
leaders, celebrities, sports stars, people with different levels of education,
and people from different employment groups, all involved in this marketing and
information campaign.
We specifically need to hear from
people who were formerly part of the anti-vax movement, who have changed their
mind. We need to hear directly from people who are severely ill in hospital,
preferably with video.
Consideration should be given to
prosecution of those spreading misinformation.
4) Addressing Myside Bias by emphasizing unity
Polarized differences and myside bias can be reduced by emphasizing that we are all on the "same team." Such a principle of unity, of treating all fellow humans with equal respect, regardless of nationality or culture, is consistent with the laws or constitutions of most countries, and with the principles of most religions. We all want to prosper, to live a healthy, happy life, to live in peace, and to have freedom while also acknowledging the need for cooperation. We have a shared humanity despite the existence of national or cultural borders. Unity between divided groups often improves if the groups need to work together to solve a larger external threat. In this sense, it is unfortunate that the pandemic (a worldwide threat) has not led to a greater degree of unity between nations. Environmental degradation is another massive looming threat of this kind. Disagreement about the nature of such threats, and about the type of action needed, is part of the problem.
5) Friendship, Diplomacy, and Trade between opponents
We should strive to develop friendships and trade relationships between members of opposing groups. Steven Pinker emphasizes this point in his book on the history of violence in society, Better Angels of Our Nature: Why Violence has Declined.
This principle could be objected to,
using extreme examples: most of us would not consider it appropriate or helpful
to have cultivated friendships with Nazis during World War II. But most members
of opposing groups are not extremists; they are moderates. It is necessary to
denounce extremism, but this does not mean denouncing almost half of the entire
population on the other side of an ideological divide. If there is to be anyone
influencing or learning from each other, there has to be ongoing friendship.
6) Experiential Education
Direct experiential education is
extremely important. People need to take tours through overflowing intensive
care units, meet the burned out but highly compassionate and expert staff, and
be aware of the suffering patients there. There will be many patients who are
members of their very own ingroup. I think this will be very persuasive, but
this has barely been done at all during the pandemic. Of course, there are
technical, ethical, and privacy-related barriers to having such tours, but
these barriers could be overcome with good planning. At the very least, there
should be embedded journalists in these environments, just as embedded
journalists have been allowed access to war zones.
7) Vaccine Education
Specific education about
vaccine-preventable diseases (such as polio, measles, or smallpox) is important
and helpful. Many people don't understand how severe these diseases were, and
how remarkably effective vaccines have been to spare hundreds of millions of
people (mostly children) terrible suffering and death.
Specific education about how vaccines
work is important. Many people simply do not know these things.
8) Ingroup leaders as educators and influencers
Members of ingroups (most likely
moderates) will be much more influential as sources of education and
information, than members of outgroups, who will most likely be dismissed if
they are even heard at all.
In the case of the pandemic,
encouragement of vaccination from religious leaders and right-wing moderates
will be useful to persuade others in this community to be vaccinated. Leaders
of these ingroups must denounce extremism and violent behaviour.
9) Emphasis on underlying values
The importance of emphasizing
underlying values is a point made by Haidt. People on the right-wing of the
political spectrum tend to value loyalty, family, and purity. Issues such as
environmental protection and vaccination are consistent with values of loyalty
and purity. It is loyalty to country, loyalty to one's own children (looking
after their present and future well-being, enjoyment, and prosperity), and
loyalty to God (who would want to care for all people, to encourage peace on
earth, to care for the place we live, and to help people help one another). The
idea of purity is well-served by plans to protect the environment and to
protect the body from a devastating infection.
10) Stop funding propaganda outlets
Steps should be taken by individuals
and corporations to stop financial support for propaganda outlets, and to
support independent, unbiased journalism. In general, we would not want our
news sources to be influenced by wealthy donors or political parties.
11) Beware of partisan "think tanks"
Good investigative journalism is
needed to show financial and political influences coming from partisan think
tanks and corporate lobby groups. I hope that if people can become more aware
of these issues, there could be organized efforts to oppose such groups, and
legislation to limit their power.
National security efforts are
critical, to prevent other nations from contributing to propaganda and
extremist groups in our countries. Investigative journalism is essential, as is
monitoring of "bots" and fake social media accounts, etc. Government
action is likely to be necessary.
12) Reduce social media polarization
Steps should be taken, on a personal
and political level, to reduce the tendency for social media platforms such as
Twitter and Facebook to produce "news bubbles" and to foment division
or extremism. This could involve persuading social media companies (through
individual and government intervention) to adjust the algorithms on their
sites, to help reduce exposure to extremist positions or false information, and
to help "fact check." On a personal level, one of the options is
simply to reduce or stop using social media.
13) Psychiatric techniques
As a psychiatrist, it is often
impossible to challenge entrenched biases with a patient unless there is a very
strong therapeutic alliance, rapport, and trust. Even then, the amount of
change to expect is very limited and slow, especially in the short term.
It is possible to encourage education,
to help patients expand their horizons a little bit.
If there are low-grade psychotic
symptoms underlying belief in conspiracy theories, an antipsychotic medication
could be useful, but most people with this issue would not be willing to try
this.
If past trauma or adversity is driving
involvement with conspiracy theories or destructive ingroup behaviour, then
compassionate, empathic trauma-informed treatment could be helpful.
Cognitive-behavioural therapy (CBT),
in principle, could help people to recognize and change cognitive distortions
or biases, but the nature of longstanding ideological bias is less amenable to
change, in part due to a lack of insight on the part of those having these
problems, and in part due to powerful resistances to change that people have
developed over a lifetime, maintained or magnified by like-minded family and
peers.
Motivational interviewing is another
set of techniques that would be useful to engage with someone having problems
due to polarization, conspiracy theories, ideological propaganda, or anti-vax
ideas. This is a style of therapy used to help people with addictions. Its
foundation has to do with acknowledging a spectrum of insight and
willingness/readiness to change for people with addictive problems, and to
match the treatment with the level of readiness. While motivational
interviewing is a suitable therapeutic style, it is also to some degree a
pretty obvious, common-sensical approach. In any case, I encourage checking out
a workbook about motivational interviewing, or some YouTube videos teaching the
basics.
14) Empathy with honesty
In a conversation or debate with a
person espousing a conspiracy theory or following some type of propaganda,
empathy is needed for the conversation to continue. In conversing with someone
who has a delusional belief, it is important that the person you're talking to
knows your honest position on the issue, and knows that you are prepared to
back up your position with good evidence, but it is essential that you show
understanding of their feelings about the matter, and that the discussion does
not deteriorate into a shouting match or into personal attacks.
As stated in number 8) above, it could
be useful in a debate or conversation with conspiracy theorists, anti-vaxxers,
etc. to find examples of prominent people within their ingroups who have
changed their mind and moderated their position, while still endorsing and
supporting the ingroup. This could include examples of politicians, religious
leaders, and celebrities your debate partner might support or admire, who are
now endorsing vaccination or other relevant policy issues. As of December 2021, I am aware of one notorious
member of this ingroup, a major U.S. political leader, who is now supporting
vaccinations.
15) Possible need to end the conversation or relationship
Open dialogue requires safety and
fairness. It is impossible to have a productive discussion with someone
shouting at you, threatening you, or monopolizing the conversation. If the
person you are talking to cannot behave in a physically safe and respectful
manner, then it is necessary to end the discussion.
It may be necessary to end some
relationships altogether, because continued contact may prove to be too
aggravating and stressful over time, distracting us from more positive and
helpful engagements or relationships. But if the conversation or relationship
does end, I encourage people to remain polite, gentle, and civil, maybe with the possibility of re-establishing the relationship in the future, if the situation improves.
16) Social Pressure & celebrity influence
It can be helpful to make use of media
to show that public health measures such as vaccination & mask usage--and
environmental measures such as recycling, reducing carbon emissions, and
ecological protection--are attractive, fashionable, and cool. Conversely, media
can help show that being an anti-vaxxer or a polluter is very unattractive. This
type of work could involve the help of celebrities, sports stars,
"influencers" and models.
17) Justice
In order to deal with con artists or
fraud, we usually need to involve the criminal justice system. For a person who
willfully neglects safety behaviour, and causes harm to others, there would be legal consequences. For example, almost everyone, regardless of political
orientation, would agree that we should prosecute drunk drivers, especially if they harm someone on the road. Rehabilitative treatment should
be offered as well, for example to treat alcoholism. There is a continuum in our society between debatable contrarian opinion on one side, fraud and frank propaganda on the other. Obviously we shouldn't suppress contrarian opinion using the power of the legal system, but there should be consideration of prosecution in cases where frank lies are causing substantial harm to individual and public health, especially when the perpetrators are profiting financially.
For con artists who are successfully
prosecuted, it can often be the case that the victims who were conned,
sometimes leading to severe financial or physical harm, will still insist that
they were not victims at all. They may continue to support the con artist even
after prosecution and conviction. Such is the tenacious power of people's need
to "save face" -- admitting they were conned by someone they and their family and friends have admired for years as a hero can be embarrassing and
humiliating. In order to make this process easier, it is necessary for fellow
con victims to come forward and admit the truth. We see a few examples (though not enough) of this
happening with previous supporters of a well-known political leader since 2016,
which hopefully will lead the way to broader positive change.
18) Be politically involved! Vote!
Some extremist or fanatical groups
have been organizing protests, frightening and obstructing health care workers
and patients at hospitals in recent days. Others are threatening the foundations of democracy itself.
It is necessary to become more
politically aware and involved. In an age where democracy itself is under
threat, it is essential to use your right to vote, and to help & encourage
others to vote as well. If people become so discouraged or cynical about the
present state of affairs that they don't even bother to vote, then our nation's
and our world's problems will be dealt with by people who are very ill-equipped
to solve them.
Selected Readings & References
Armstrong, Karen. The Battle for God: A history of
fundamentalism. (2001)
Brashier, N. M., Pennycook, G., Berinsky, A. J., & Rand,
D. G. (2021). Timing matters when correcting fake news. Proceedings of the
National Academy of Sciences, 118(5).
Bergstom, C. and West, J. Calling Bullshit: The Art of Skepticism in a Data-Driven World (2020).
Briant, Emma Louise (2015). Propaganda and
Counter-terrorism. Manchester: Manchester University Press. p. 9.
Christakis, Chris. Apollo's Arrow: The profound and enduring
impact of coronavirus on the way we live. (2020)
Dawkins, R. The God Delusion (2006).
Dawkins, R. Outgrowing God (2019).
(note: Dawkins is a critic of religion, but I think it is good for any religious person to
understand the reasons for this; I mention these books
here because they address the subject of how people come to form extremely
strong and often irrational ideological positions, and how people can move away from this, while gaining some education about basic science)
Douglas, K. M. (2021). COVID-19 conspiracy theories. Group
Processes & Intergroup Relations, 24(2), 270-275.
Douglas, Karen et al, Understanding Conspiracy Theories.
Political Psychology 40, Suppl. 1, 2019
Epstein, Z., Berinsky, A. J., Cole, R., Gully, A.,
Pennycook, G., & Rand, D. G. (2021). Developing an accuracy-prompt toolkit
to reduce COVID-19 misinformation online. Harvard Kennedy School Misinformation
Review.
Haidt, J. The Righteous Mind: Why Good People are Divided by
Politics and Religion (2012).
Harrington, B. (2012). The sociology of financial fraud. In
The Oxford handbook of the sociology of finance.
https://theconversation.com/why-people-believe-in-conspiracy-theories-and-how-to-change-their-minds-82514
Johnson DK et al. "Combating Vaccine Hesitancy with
Vaccine-Preventable Disease Familiarization" Vaccines 2019, 7. 39
Kahneman, D. Thinking: Fast and Slow. (2013).
Kelly, J. (2006). The Great Mortality: an intimate history
of the Black Death.
Konnikova, M. (2016). The confidence game: Why we fall for
it. Every Time. New York.
Lewandowsky, S., & Van Der Linden, S. (2021). Countering
misinformation and fake news through inoculation and prebunking. European
Review of Social Psychology, 1-38.
Marchlewska, M., Green, R., Cichocka, A., Molenda, Z., &
Douglas, K. M. (2021). From bad to worse: Avoidance coping with stress
increases conspiracy beliefs. British Journal of Social Psychology.
McRaney, David. How Minds Change: The Surprising Science of Belief, Opinion, and Persuasion. 2022.
Pennycook, G., & Rand, D. G. (2021). The psychology of
fake news. Trends in cognitive sciences.
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., &
Rand, D. G. (2020). Fighting COVID-19 misinformation on social media:
Experimental evidence for a scalable accuracy-nudge intervention. Psychological
science, 31(7), 770-780.
Pennycook, G., McPhetres, J., Bago, B., & Rand, D. G.
(2020). Predictors of attitudes and misperceptions about COVID-19 in Canada,
the UK, and the USA. PsyArXiv, 10, 1-25.
Peters, Maertens, R., Roozenbeek, J., Basol, M., & van
der Linden, S. (2021). Long-term effectiveness of inoculation against
misinformation: Three longitudinal experiments. Journal of Experimental
Psychology: Applied, 27(1), 1.
Peters, Ellen. Innumeracy in the Wild: Misunderstanding
and Misusing Numbers. Oxford (2020).
Pinker, S. Better Angels of Our Nature: Why Violence has
declined (2012).
Pinker, S. Rationality: What it is, why it seems scarce, why it matters (2021)
Pretus, C., Hamid, N., Sheikh, H., Ginges, J., Tobeña, A.,
Davis, R., ... & Atran, S. (2018). Neural and behavioral correlates of
sacred values and vulnerability to violent extremism. Frontiers in psychology,
9, 2462.
Prum, Richard. The Evolution of Beauty: How Darwin's
Forgotten Theory of Mate Choice Shapes the Animal World - and Us. Anchor
(2018).
Rathje, S., Van Bavel, J. J., & van der Linden, S.
(2021). Out-group animosity drives engagement on social media. Proceedings of
the National Academy of Sciences, 118(26).
Rutjens et al, "Science skepticism across 24
countries." Social Psychological
and Personality Science 2021.
Shoots-Reinhard et al. "Ability-related political
polarization in the COVID-19 pandemic" Intelligence 88, 2021, 101580
Swire‐Thompson, B., Ecker, U. K., Lewandowsky, S., &
Berinsky, A. J. (2020). They might be a liar but they’re my liar: Source
evaluation and the prevalence of misinformation. Political Psychology, 41(1),
21-34.
Tappin, B. M. (2021, October 4). Exposure to Arguments and
Evidence Changes Partisan Attitudes Even in the Face of Countervailing Leader
Cues. https://doi.org/10.31234/osf.io/247bs
Taylor, S. (2019). The psychology of pandemics: Preparing
for the next global outbreak of infectious disease. Cambridge Scholars
Publishing
technologyreview.com/2020/07/15/1004950/how-to-talk-to-conspiracy-theorists-and-still-be-kind/
Van Bavel, J. J., Baicker, K., Boggio, P. S., Capraro, V.,
Cichocka, A., Cikara, M., ... & Willer, R. (2020). Using social and
behavioural science to support COVID-19 pandemic response. Nature human
behaviour, 4(5), 460-471.
Van der Linden, S., Panagopoulos, C., Azevedo, F., &
Jost, J. T. (2021). The paranoid style in American politics revisited: An
ideological asymmetry in conspiratorial thinking. Political Psychology, 42(1),
23-51.n
Van Prooijen & Kuijper, "A comparison of extreme religious and political ideologies: Similar worldviews but different grievances", Personality and Individual Differences 159 (2020)
Van Prooijen & Krouwel, "Psychological features of
extreme political ideologies."
Current Directions in Psychological Science 2019 28(2) 159-163.
van Prooijen et al, "connecting the dots: Illusory
pattern perception predicts belief in conspiracies and the
supernatural." Aug 21, 2017/
van Prooijen and Song, "The cultural dimension of
intergroup conspiracy theories."
August 13, 2020.
Zimbardo, P. (2011). The Lucifer effect: How good people
turn evil. Random House.
