Evaluating Evidence

—Seeking Reality

Introduction

The fictional character Sherlock Holmes became famous for his skill in evaluating evidence.

As we go about our lives we inevitably encounter clues that tell us something about the world we live in. Each of these clues is a piece of evidence that provides some glimpse of reality, however it is up to us to assess the quality of that evidence, to interpret that evidence, and to constantly assemble a lifetime of evidence gathering into a coherent description of our world. That model of our world continues to expand and evolve as we become aware of new evidence and gain new insights. We are constantly striving to better understand the full extent of reality.

Evidence can be broadly defined as anything presented to support an assertion. With such a broad range of evidence available it becomes difficult to determine the meaning of any particular piece of evidence. When evaluating evidence it is natural to ask: What forms of evidence are more reliable than others? How can we best draw reliable conclusions from evidence? How can evidence be interpreted reliably? How does new evidence fit into, or change, my existing coherent concepts of the real world?

This course advocates reason and is provided as a refuge and antidote to post-truth trends.

Objectives

Completion status: this resource is considered to be complete.

Attribution: User lbeaumont created this resource and is actively using it. Please coordinate future development with this user if possible.

The objectives of this course are to:

• Learn to assess the quality of evidence,
• Learn to evaluate various forms of evidence,
• Weigh the value of one particular piece of evidence against another,
• Assess what each piece of evidence reveals about the observer, and what is being observed,
• Gain a stronger, deeper, more accurate, and more coherent understanding of reality,
• Better understand the interplay among evidence, your current beliefs, and your current worldview,
• Identify and avoid manufactured controversies,
• Identify nonsense or untrue claims and dismiss it along with other nonsense.

The course contains many hyperlinks to further information. Use your judgment and these link following guidelines to decide when to follow a link, and when to skip over it.

This course is part of the Applied Wisdom curriculum and of the Clear Thinking curriculum.

If you wish to contact the instructor, please click here to send me an email or leave a comment or question on the discussion page.

OK, let’s evaluate the evidence and move toward reality!

Evaluating Evidence Audio Dialogue

The Nature of Evidence

The concept of evidence has often been called upon to fill a number of distinct roles, and as a result there are several concepts and definitions of evidence, including:

1. Evidence is that which justifies belief,
2. Evidence is the basis of rational thinking,
3. Evidence guides us toward truth, and
4. Evidence serves as a neutral arbiter among rival theories and their adherents.

Evidence provides justification for beliefs. Evidence can be cited as a reason for believing, or not believing some claim. Each person’s beliefs reflect their total assimilation of evidence over time. This might include consideration of evidence that supports a particular belief, along with further evidence that can contradict the belief, while considering certain evidence that can undercut, or rebut contradictory evidence, and so on.

“In order to be justified in believing some proposition then, it is not enough that that proposition be well-supported by some proper subset of one's total evidence; rather, what is relevant is how well-supported the proposition is by one's total evidence.”

Are crop circles more likely created by alien visitors or by pranksters?

A person’s beliefs will depend on the variety of alternative explanations, causes, or hypothesis that come to mind as plausible explanations for the evidence. Defense attorneys often present an “alternative theory of the case” to provide some alternative narrative that also fits the evidence. Before the process of biological evolution was discovered and described by Charles Darwin and others, evidence for the diversity of life favored divine interpretations. Do you think it is more likely that crop circles are created by pranksters, or by alien visitors? Conspiracy theories, fringe theories, and pseudo-scientific explanations are sometimes proposed as alternative explanations of certain bodies of evidence. Therefore the set of alternative hypotheses considered serve the same role in justifying belief as do other forms of evidence.

Philosopher David Hume notes that “A wise man proportions his belief to the evidence”.

The hallmark of a rational thinker is respect for evidence. The definition of rationality focuses on factual conformity of one's beliefs with one's reasons to believe, or of one's actions with one's reasons for action. “It's plausible to suppose that much if not all of the value that we place on believing rationally depends on a connection between believing rationally and believing what is true.” Correct reasoning based on incorrect evidence is likely to result in incorrect conclusions.

Evidence may be direct or indirect. For example, a meteorologist may consider various weather data as evidence and conclude that it will rain. You may consider the meteorologist’s forecast, along with her skill in making such forecasts as evidence and conclude it will rain. In this context the meteorologist is relying on first-order evidence, and your concluding that it will rain relies on the higher-order evidence in the form of her forecast.

Evidence plays a mediating role in our efforts to arrive at an accurate picture of the world. Evidence guides us toward what is true and away from what is false. We seek to believe what is true by holding beliefs that are well-supported by the evidence, and we seek to avoid believing what is false by not holding beliefs that are not well-supported by the evidence.

The capacity of evidence to justify belief and the fact that rational thinkers respect their evidence reasonably depends upon the connection between evidence and truth.

In 1896 medical science learned that distinctive clustered white lesions called Kopliks spots are a reliable early symptom of the measles. With this knowledge, doctors who see Kopliks spots have reliable evidence of the measles. However, a person seeing Kopliks spots who has no knowledge of their medical importance does not have evidence of the measles, even though this lay person and the doctor are observing the same evidence. As this example illustrates, the extent to which someone is in a position to gain new information on the basis of particular pieces of evidence typically depends upon that person’s background knowledge.

The previous paragraph can be summarized as: Kopliks spots are evidence for measles, given the theory that Kopliks spots are a symptom of measles. Generalizing from this case, we interpret evidence using the rule:

E is evidence for hypothesis H relative to background theory T.

We can now see that the theory itself becomes an essential part of the evidence. This makes it important to be able to assess the likelihood that a theory is correct.

For the mathematically inclined, the role of evidence in supporting a theory (represented here as a tentative theory or hypothesis) can be compactly expressed as:

E is evidence for H if and only if Prob(H|E) > Prob(H).

This is read as: E is evidence for hypothesis H exactly when the conditional probability of H given the existence of E is greater than the probability of H in the absence of evidence E. As a simple, gruesome example, the evidence of blood on the knife is evidence that the suspect committed the murder if and only if the probability the suspect committed the murder is greater given that the blood is on the knife.

Similarly:

E is evidence against H if and only if Prob(H|E) < Prob(H).

For example, the fact that the suspect's fingerprints are not on the knife is evidence against the hypothesis that the suspect committed the murder if and only if the probability that the suspect committed the murder is lower given the absence of his fingerprints on the knife than it would be otherwise.

A central function of evidence is to serve as a neutral arbiter among rival theories and their adherents. Whatever disagreements might exist at the level of theory, if those who disagree are objective, then when evidence which decisively resolves the dispute in one direction or the other emerges the dispute will be settled. Because objective inquiry is evidence-driven inquiry, and because we gather evidence from the same world, views based on objective-inquiry converge toward consensus agreement as shared evidence accumulates.

To act reliably as a neutral arbiter, evidence needs to be available to each of the multiple parties to the dispute. However, not all evidence is public or available to multiple observers. Consider the case of believing you have a headache, because your head does ache. Your conclusion is justified, but the evidence for your conclusion is only accessible to you. Others would have to rely on your testimony rather than the actual pain you are feeling as evidence to conclude that you do actually have a headache.

An Evidence Model

Relationships between reality, evidence and belief

The diagram on the right is intended to illustrate the relationships between reality, evidence and belief. Reality is vast, complex, and dynamic and despite extensive and on-going exploration, discovery, and research efforts, the total collection of all existing evidence represents only a small fraction of our universe. Most events from the past were never recorded, and many records are lost, destroyed, or discarded over time. After collection, evidence needs to be published, distributed, or displayed for it to be potentially available to others.

For evidence to be readily available, it has to appear in some form in some place that is reasonably accessible. Accessible repositories of evidence are finite and are necessary curated—someone or something selects what is contained in each collection and what is omitted. The local library may be well stocked, but it contains only a small fraction of all books ever written. Bookstores stock only a fraction of books in print. University libraries may subscribe to several scholarly journals, but their collections are necessarily limited and may be narrowly focused. Museum collections are limited; news programs broadcast only a small fraction of the information that is available, and we can only visit a few places and speak with a few people, relative to the entire world. If we want to know if it is likely to rain today, we typically consult available reports prepared by meteorologists, rather than access raw weather data to create our own forecast.

Censorship in many forms prevents access to various evidence. Private conversations, privileged conversations, classified materials, state secrets, taboo subjects, data omitted from reports, unfavorable studies, inconvenient truths, and stories never told are inaccessible to most of us.

When we are young, much of what we learn is based on evidence selected for us by others. Parents often decide where we live as children, what we see, where we can go, who we talk to, the stories we are told, who we can play with, and what we can do. In school: curriculum guides, lesson plans, approved text books, and teacher discretion determine what evidence we are presented with and what we are told is true and untrue. We are told who to trust and who to be wary of. Cultural influences determine the languages we learn, the customs we are taught, the traditions we uphold, the behaviors we observe, what is taught in schools, the ethics that are espoused and practiced, and the ideologies we value. Advertisements and other forms of indoctrination find us wherever we go throughout our lives.

As we get older we have more opportunities to select the evidence we study. A personality trait called openness to experience becomes important. People high in openness are motivated to seek new experiences and to engage in self-examination. Openness involves six facets, or dimensions, including active imagination (fantasy), aesthetic sensitivity, attentiveness to inner feelings, preference for variety, and intellectual curiosity. As a result, people who are high in openness select from a broader range of evidence than people who are low in openness. People who are high in openness are more likely to examine evidence and engage in activities that challenge their existing beliefs than are people who are low in openness.

Each of us has some approach to discovery, exploration, study, learning, inquiry, assessment, and verification. Together these can be called our individual research methods. Our approaches will determine what questions we choose to explore, what evidence we seek out, how gullible or skeptical we are regarding various forms of evidence, what we decide is true or false, and what we choose to believe or reject. We may be passive, learning from whatever we happen to come across and believing what we hear. Alternatively we can become more active in seeking out a broader range of evidence and experiences, investigating controversies, exploring apparent contradictions, comprehending opposing viewpoints, challenging assumptions, and exercising critical thinking.

Our curiosity determines what we explore, the questions we ask, and what we seek to learn. Our attention determines what evidence we will consider most important and what is most influential.

Our world is so large and complex and the amount of information available is so vast, that despite our best efforts, we are all at least 99.9% ignorant. We only ever learn about a tiny fraction of our world.

Confirmation bias is the human tendency to search for, interpret, favor, and recall information in a way that confirms one's preexisting beliefs or hypotheses, while giving disproportionately less consideration to alternative possibilities. People display confirmation bias when they gather or remember information selectively, or when they interpret it in a biased way. The effect is stronger for emotionally charged issues and for deeply entrenched beliefs. People also tend to interpret ambiguous evidence as supporting their existing position. Confirmation biases contribute to overconfidence in personal beliefs and can maintain or strengthen beliefs in the face of contrary evidence.

Each person holds some worldview as their fundamental cognitive orientation toward their entirety of knowledge and point of view. Each person’s worldview is a comprehensive set of opinions, seen as a coherent whole, about what the world is, how the world works, and the nature of human existence. Our worldview serves as a framework for generating various dimensions of human perception and experience like knowledge, politics, economics, religion, culture, science, and ethics. Our worldview may consist of assumptions so basic and unexamined that it may go unnoticed, even though it is frequently used as the basis for forming beliefs and making decisions. Our worldview may include assumptions regarding basic human nature, the creation of the earth, the origins of biodiversity, the relationships of cause and effect, the nature of good and evil, trustworthiness of government, the reliability of science, the role of religion, basic moral principles, basic scientific principles, and other foundational assumptions.

A recursive or positive feedback loop exists between our current beliefs and our confirmation biases that creates and sustains our worldview. In the absence of a worldview and confirmation bias, our current beliefs would be based heavily, if not entirely, on the most recent evidence we have been exposed to. However our beliefs accumulate over time to form our worldview. At the same time our worldview is influencing how we interpret each new piece of evidence. Our beliefs and worldviews inform our confirmation bias, which selects, ignores, emphasizes, discounts, and interprets each new piece of evidence in ways that tend to confirm our existing beliefs.

The ways in which children typically come to believe, defend, question, doubt and eventually reject their belief in Santa Claus provides an example. At a young age children may be told the story of Santa Claus as a cheerful, benevolent, and generous man who arrives Christmas Eve bringing presents. Because they consider parents as authority figures, they enjoy receiving presents, and the culture often reinforces the story, the children believe the story without question. Their worldview now includes features that allows for, sustains, justifies, and explains their belief in Santa Claus. Evidence available to the young child tends to reinforce the story and children are often isolated from information that would contradict the belief. Their belief in Santa Claus strengthens to a point where they will dismiss, and even argue against evidence that challenges their cherished belief. As the child gets older, they gain a broader perspective of the world and may begin to think critically and question the story. They may begin to reflect on and ask questions such as: Where does he live? How old is he? How does he learn about all the children? How can he make so many toys? How can he travel so far and visit so many children? Parents may respond to these questions in ways that are more creative than truthful, and are intended to allow the child to enjoy the story a bit longer. Eventually, however, the weight of evidence against the story becomes inescapable; the child rejects the story, revises his beliefs, and updates his worldview. At this time the parents may have to reconcile with the child, explaining that the fantasy was promoted with the best interests of the child in mind.

A similar awakening may occur as an adolescent learns about friendships, betrayal, human sexuality, romantic relationships, family secrets, the value of education, the job market, office politics, religious dogma, moral virtue, factory farming, death, and many other coming-of-age epiphanies and insights.

In each such case new evidence bears the burden of overcoming presently held beliefs. Presently held beliefs are formed from the evidence accumulated throughout our lives and often form a coherent worldview. New evidence is persuasive depending on the strength of the present worldview—the a priori beliefs—at least as much as on the strength of the new evidence.

In the face of contrary evidence, motivation to protect the current worldview is often strong, and contrary evidence may actually strengthen the person’s commitment their current beliefs. There may reach a point, however, when the accumulation of contrary evidence overwhelms the long-held beliefs, and a sudden shift in beliefs happens. The old beliefs are quickly rejected in favor of some new coherent model that accommodates the new evidence. Santa Claus, the tooth fairy, the baby delivery stork, naïve ideals, and other comfortable fantasies are quickly left behind.

Each piece of evidence is one piece of some larger puzzle. It is our responsibility to determine what puzzle it belongs to, how it fits into that puzzle, what portion of the reality puzzle it represents, and what insight it reveals. As we encounter evidence we need to ask: What does this evidence convey about reality? Is the information being conveyed true? Does it tell me more about the observer or what is being observed? How representative is this? To what extent can I draw a more general conclusion from this data point?

Assignment

The purpose of this assignment is to examine how you respond to evidence.

1. Read the essay Being 99.9% Ignorant.
2. Using the approach described in that essay, calculate how ignorant you are.
3. How open to experience are you? What is the evidence that you are open to experience? What it the evidence you are not open to experience? How open to experience are you?
4. If you believed in the Santa Clause myth, or some analogous childhood myth, recall the various phases you went through in forming the belief, defending the belief, questioning the belief, and finally rejecting the belief. What evidence caused your beliefs to snap?
5. Recall some belief you held strongly until your adolescence when you questioned, doubted, challenged, and then rejected that belief. Did you defend that belief in the face of contrary evidence? What body of evidence was finally successful in shifting your beliefs?
6. Write down your current worldview. How has it changed over the years? Do you foresee it changing again?
7. What beliefs do you hold to strongly, although they may be controversial? What evidence are you aware of that is contrary to that belief? How do you reconcile your beliefs with that contrary evidence?
8. What steps do you take, if any, to continually to align your beliefs with reality?
9. Work to overcome the illusion that what you see is all there is.

Forms of Evidence

Evidence appears in many forms. These include: physical evidence, documentary evidence, and testimony, along with experimental results and reports, statistical evidence, historical research, experts, authorities, hearsay, rumors, and gossip. It is difficult to assess the reliability of various forms of evidence, and to draw conclusions when faced with conflicting evidence. The next sections of this course establish various criteria useful in assessing the relative reliability of various forms of evidence.

Evaluation Criteria

Each piece of evidence is a glimpse of reality as seen through a fog.

We examine evidence searching for clues that will tell us more about the world we live in, and to help us investigate and answer specific questions we may have. Certain forms of evidence will provide more relevant, more revealing, and more reliable information than others. Evidence is gathered when an observer examines an object or event and reports on the findings. When interpreting evidence it is important to consider what is being learned about the object, the observer, and our own biases in making those interpretations.

Separating the signal from the noise is important when interpreting evidence. The signal is that portion of the information that is relevant to the problem at hand. The noise is information that is irrelevant, unhelpful, and often a distraction. Signal-to-noise ratio is a precise term in science and engineering that is sometimes used informally to refer to the ratio of useful information to false or irrelevant data in a conversation or exchange.

Relevance

The courts have deliberated at length on the relevance of various forms of evidence. These principles can be usefully extended to more general cases where the relevance of evidence has to be evaluated during any exploration, discovery, investigation, decision, or dialogue.

In a U.S. Federal court of law, evidence is relevant if:

1. it has any tendency to make a fact more or less probable than it would be without the evidence; and
2. the fact is of consequence in determining the action.

Relevancy is not an inherent characteristic of any item of evidence but exists only as a relation between an item of evidence and a matter properly provable in the case, or more generally, relevant to the determination that is being made, or the question that is being explored. The essential question in determining relevance is: does the item of evidence tend to prove the matter sought to be proved?

Relevant evidence represents signal—the helpful information, irrelevant information represents noise—unhelpful distractions. Focusing on relevant evidence while dismissing irrelevant information increases the clarity of the investigation. Irrelevant information is a distraction to be avoided; however, sometimes a detail that seems irrelevant at first can later provide an insight that becomes a key to the solution.

In U.S. Federal Court cases even relevant evidence may be excluded if its probative value—importance in seeking truth or deciding facts—is substantially outweighed by a danger of one or more of the following: unfair prejudice, confusing the issues, misleading the jury, undue delay, wasting time, or needlessly presenting cumulative evidence. In the general case, efforts to prejudice, confuse, mislead, delay, or waste time are unhelpful because they distract and mislead decision makers from arriving at thoughtful conclusions.

Many informal logical fallacies are classified as fallacies of relevance because the argument relies on premises that are not logically relevant to the conclusion. These fallacies of relevance include:

• The Argument from Ignorance,
• The Appeal to Inappropriate Authority,
• Argument Ad Hominem,
• The Appeal to Emotion,
• The Appeal to Pity,
• The Appeal to Force, and
• Irrelevant conclusions.

Learn to recognize and avoid these fallacies.

Assignment

Complete the Wikiversity course module on Fallacies of Relevance.

Origin

An original item is the primary form or type from which varieties are derived. An original item is not a substitute. It is actually and exactly what is claimed, and not counterfeit.

Knowing where any particular piece of evidence was originally found, obtained, or created, is important in interpreting that evidence. To determine the origins of a piece of evidence questions such as these might be asked: Was the gun found at the crime scene, or located somewhere else? Is this the original document, or some copy, translation, transcription, reconstruction or recollection of the original? In what geological layer and from what geographic location were these fossils found? Was the skull fossil found near fossils of other body parts, or were they found in separate locations? Is this the original or a copy?

In legal proceedings a foundation has to be established for each piece of evidence before it can be admitted. A “foundation” is the minimum amount of facts needed to demonstrate that a particular piece of evidence is authentic enough and relevant enough that it can be admitted at trial.

In legal contexts a chain of custody documents the chronological sequence of the seizure, custody, control, transfer, analysis, and disposition of physical or electronic evidence. In other contexts, the provenance is the chronology of the ownership, custody or location of a historical object. The term was originally mostly used in relation to works of art, but is now used in similar senses in a wide range of fields, including archaeology, paleontology, archives, manuscripts, printed books, and science and computing. The primary purpose of tracing the provenance of an object or entity is normally to provide contextual and circumstantial evidence for its original production or discovery, by establishing, as far as practicable, its later history, especially the sequences of its formal ownership, custody, and places of storage. The practice has a particular value in helping authenticate objects.

Originals provide more reliable evidence than copies, or items of dubious or unknown origin.

Authenticity

In U.S. Federal court a witness may testify to a matter only if…the witness has personal knowledge of the matter. Generalizing on this rule, first person accounts are more reliable than hearsay, rumors, or gossip. Unfortunately we can rarely rely on first person accounts for our everyday decision making, so the primacy of each source—the degrees of separation from those that have direct knowledge of the matter—have to be determined and evaluated. When hearing a rumor, identify the original source, research the facts, and evaluate the reliability of the information. Dispel untrue, misleading, and unreliable rumors. Don’t promote falsehoods, or misleading information. Use a reference chosen from these resources for fact checking to assess the reliability of information obtained casually.

Analogous to the primacy discussion above, the U.S. Federal courts specify “an original writing, recording, or photograph is required in order to prove its content…” Generalizing this rule, original documents, photographs, objects, and other artifacts are more reliable and therefore preferable to copies, transcriptions, translations, models, illustrations, or other derived materials when they are available. When copies are relied on, it is helpful to determine how the copies were made, and what differences can be expected between the copy and the original.

Because truth is correspondence with reality, attaining true beliefs requires examining reality as closely and as carefully as possible. Closer is better. The more closely the evidence is connected to the reality being reported on the more authentic that evidence is. The more distant the connection between the evidence and the reality, the less authentic that evidence is.

Each degree of separation imposes filters that can occlude and distort, include or exclude, amplify or diminish, emphasize or attenuate each glimpse we get of the real world. The distortion created by these filters need to be removed or compensated for.

Evidence that brings the observer closer to a larger fragment of reality for an unobstructed examination is more valuable than smaller fragments, or more distant fragments, or more obstruction.

Assignment

Each time you hear a rumor or come across the next Internet or social media trend, take time to check the facts from a reliable source before spreading disinformation or forming unfounded beliefs.

Objectivity

“Individuals or groups with a vested interest in convincing others of their point of view can be skillful at spinning information.” Therefore to accurately evaluate evidence it is important to identify the affiliations, incentives, motivations, or biases each person has who is presenting or interpreting evidence.

Even without any apparent conflict of interest, every human has a tendency to search for, interpret, favor, and recall information in a way that confirms one's preexisting beliefs or hypotheses, while giving disproportionately less consideration to alternative possibilities. This human tendency, called confirmation bias, affects all of us, often unconsciously. Confirmation biases contribute to overconfidence in personal beliefs and can maintain or strengthen beliefs in the face of contrary evidence.

No one can be entirely objective and no one is entirely neutral on a particular issue. To better understand the biases that are involved, it is helpful to identify the interests the various stakeholders have on any particular issue or controversy. Consider

1. Who is buying or selling a product or service involved in the issue?
2. Who are the competitors of those buying or selling?
3. Who makes the rules? (e.g. regulators, politicians, governments)
4. Who else is affected by this issue? Who stands to gain or lose (e.g. money, power, prestige, pride, status quo) as a result of these findings?

Disclosing potential conflicts of interest of each party allows a more accurate assessment of their viewpoint, the conclusions they draw, and the recommendations they make. Discount evidence provided by anyone with a conflict of interest when the evidence presented advances their interests.

When considering evidence, consider the possibility that a manufactured controversy has deliberately contrived disagreement, typically motivated by profit or ideology, with the intent of creating public confusion concerning an issue about which there is no substantial academic dispute. In a manufactured controversy:

“The formula is to amplify uncertainties, cherry-pick experts, attack individual scientists, marginalize the traditional role of distinguished scientific bodies and get the media to report ‘both sides’ of a manufactured controversy.”

As an example of how lack of objectivity can influence results, consider research on the health effect of Bisphenol A, often called BPA, which remain controversial. “Many studies have been performed. Studies’ outcomes depend on the source of the funds used to complete them. A review of BPA studies published in the August 2005 issue for the journal Environmental Health Perspectives found that 90% of government funded studies reported significant health effect of low doses of BPA, but none of the industry-funded did.”

Evidence provided by objective sources is more reliable than evidence provided by biased sources.

Scope

The scope of the evidence refers to the range of conditions represented.

“No matter how well a study is designed, its results cannot automatically be applied beyond the population studied.” People are diverse. We vary by age, gender, socioeconomic status, ethnic origin, education, health status, psychological status, relationship status, fitness levels, stress levels, dietary habits, body type, lifestyle, experience, work habits, and other factors. Cultural, geographic, workplace, and other environmental conditions vary among individuals and groups. Results from studies of one group cannot be reliably generalized to include groups not studied.

Species are diverse. Laboratory studies conducted on mice, rats, or other animals cannot be generalized to predict results in humans. Studies that show promising results in mice must still undergo clinical trial to assess the results on humans.

The environment is diverse. Conditions vary from one location to the next, and studies performed in one location cannot be generalized to other locations. Similarly, the biosphere is diverse. Conditions vary from one species to the next. Studies performed in one species cannot be generalized to other species or other locations. Promising results studied over a short time span cannot be generalized to predict long-range effects.

Studies that rely on large representative samples typically provide more reliable estimates of results in the whole population than those that use small samples or that rely on biased samples.

It is important to distinguish statistically significant results from short term trends or chance. In any experiment or observation that involves drawing a sample from a population, there is always the possibility that an observed effect would have occurred due to sampling error alone. A confidence level is chosen before data collection, and typically set to 95% or much higher, depending on the field of study, to ensure there is less than a 5% probability the results are due to chance or sampling error.

Anecdotal evidence is evidence collected in a casual or informal manner relying heavily or entirely on personal testimony. Although anecdotal evidence is often emotionally appealing, it is very limited in its scope. When compared to other types of evidence, anecdotal evidence is generally regarded as limited in value due to a number of potential weaknesses.

When only one or a few anecdotes are presented, there is a larger chance that they may be unreliable due to cherry-picked or otherwise non-representative samples of typical cases. Similarly, psychologists have found that due to cognitive bias people are more likely to remember notable or unusual examples rather than typical examples. Thus, even when accurate, anecdotal evidence is not necessarily representative of a typical experience.

In summary: evidence with a broader scope is stronger that evidence with a narrow scope, representative evidence drawn from a representative sample is better than a biased sample, and anecdotal evidence is not representative. Evidence drawn from an unrepresentative sample is meaningless.

Assignment

Part 1:

How representative is the story of the welfare queen?

Part 2:

Quincy Bioscience markets an over-the-counter pill called Previgen. They make several claims that Previgen improves memory, is clinically shown to improve memory, and reduces memory problems associated with aging. What is the evidence for these effectiveness claims?

1. Begin by reviewing the claims made on the Manufacture’s website. See: https://www.prevagen.com/
2. Read this USA Today news article about the product.
3. Read this FTC News Release about the product.
4. Read pages 24-26 of FTC Case CV-00124, Filed 1/9/2017.
5. Based on the scope of the evidence, do you conclude the marketing claims are sufficiently supported?

Part 3:

Read the essay perceptions are personal. What is the scope of a personal perception?

Direct

Direct evidence supports the truth of an assertion directly, i.e., without an intervening inference. Circumstantial evidence, by contrast, consists of a fact or set of facts which, if proven, will support the inference that the matter asserted is true.

For a criminal law example: a witness who testifies that he saw the defendant shoot the victim gives direct evidence. A witness who testifies that he saw the defendant fleeing the scene of the crime, or a forensics expert who says that ballistics proves that the defendant’s gun shot the bullet that killed the victim both give circumstantial evidence from which the defendant’s guilt may be inferred.

In direct evidence multiple senses cross check and provide consistent information regarding the observation. Evidence is more complete and convincing when the way something feels is consistent with the way it looks. For example, after seeing a rock, you might pick it up to examine more closely. If the texture feels like a rock and the weight is what you expect, then you have additional evidence that what you are observing is a rock. However if the surface texture is unusual, or the weight is unexpected, then perhaps you are examining an artificial stone, or a hollow stone, or something quite different made of unexpected materials.

Careful records such as notes, photographs, audio and video recordings, and diagrams are more reliable and should be used instead of unaided recall when information is required later, after the evidence has been examined first hand.

It is important the observation is separated from interpretation. We observe the sunlight appearing in the morning, traversing a path across the sky, and disappearing in the evening. One interpretation is that the earth is the center of the universe and the sun travels across the sky. An alternative interpretation is that the earth rotates on its axis as it revolves around the sun. Similarly we need to ask: Is that clear liquid water or vinegar? Is that attractive woman he is with his wife, daughter, co-worker, assistant, boss, or mistress? Have the people stopped asking questions because everything is OK or because inquiry is being punished or otherwise suppressed? Does the sun move across the sky, or does the earth move past the sun? What are alternative explanations for what we are seeing?

In direct evidence, a witness relates what he or she directly experienced.

Assignment

The purpose of this assignment is to emphasize the importance of separating observation from interpretation of evidence.

• Read these two very different interpretations of identical evidence. Does the evidence favor one interpretation over the other?
• Read this essay on the Tyranny of Evidence.
• Be careful to separate interpretation from observation.

Circumstantial

Circumstantial evidence is evidence that relies on an inference to connect it to a conclusion of fact—for example a fingerprint at the scene of a crime. By contrast, direct evidence supports the truth of an assertion directly—i.e., without need for any additional evidence or inference.

On its own, circumstantial evidence allows for more than one explanation. Different pieces of circumstantial evidence may be required, so that each corroborates the conclusions drawn from the others. Together, they may more strongly support one particular inference over another. An explanation involving circumstantial evidence becomes more likely once alternative explanations have been ruled out.

Direct evidence is stronger than circumstantial evidence.

Context

Because each piece of evidence provides only a glimpse of the real world, it is helpful to place each piece of evidence into a broader context. When numbers are not given, ask about the numbers. When numbers are given, ask about the rate of occurrence. Evaluate numbers in the context of the larger system they are part of. When hearing about deaths attributed to cause X, ask how many deaths occur each year due to other causes. When rates are given, ask about the numbers. When percentages are given, ask about the natural frequencies. Draw natural frequency diagrams to understand what is being measured by the percentages cited.

Shark attacks make gripping headlines, and many people fear shark attacks. Putting the numbers in context, however, tells a different story. For example, there were a total of 1,104 shark attacks in the United States (excluding Hawaii) in the 56 years from 1958 – 2014, of which 35 attacks were fatal. That is an average of less than one fatality per year. By comparison, the United States suffered 42,815 fatalities from motor vehicles in the year 2002. Based on these statistics, automobile accidents are approximately 68,000 times more deadly than shark attacks. Using numbers from equivalent time intervals would give a more accurate estimate.

In another example, toddlers killed more Americans than terrorists did in 2015 depending on the exact definitions used.

Establish a relevant context for interpreting each piece of evidence.

Expertise

An expert is someone with a great deal of knowledge about, or skill, training, or experience in, a particular field or activity.

Most of what we learn and come to believe is the result of various experts’ work. Historians have researched original documents, visited sites around the world, and may have interviewed eyewitnesses or examined physical evidence to be able to accurately document historical events. Biologists have examined individuals of various species in depth and have described the characteristics of each species in detail. We rely on physicians and other healthcare professionals to diagnose and treat our illnesses.

Conclusions or opinions offered by experts within their field of study are more reliable than conclusions arrived at by laypeople, or by experts operating outside their particular field. Scientific consensus is the collective judgment, position, and opinion of the community of scientists in a particular field of study. Consensus implies general agreement, though not necessarily unanimity. An area where there is a scientific consensus typically represents the best available evidence on that topic. While there are large areas of expert consensus, research is on-going and experts are always eager to ask the next question to satisfy their curiosity. This results in areas of active research and current controversy.

Beware of the psuedosymmetry of scientific authority when evaluating claims of disagreement or controversy. The psuedosymmetry of scientific authority is a form of false balance that occurs when the mass media incorrectly portrays scientists as equally divided on an issue. In its attempt at journalistic balance or in an effort to make a story more compelling, the mass media can give the public the impression that a single individual or a small band of contrarians represents a large group of scientists which disagrees with another large group of scientists. Such “balance” can give the impression that a scientific consensus has not been reached and that an issue is considered controversial within the scientific community when, in fact, it is not.

Recognized levels of evidence systems generally rank the opinion of a single expert below the results of carefully conducted case studies.

It is important to contrast the conclusion of experts with an argument from authority. It is a fallacy to cite an authority on a topic outside their area of expertise or when the authority cited is not a true expert. This often occurs when famous people such as actors, musicians, or athletes, are asked their options on policy questions outside their professional field. People who have obtained positional power may offer opinions on topics outside their field of expertise.

The English language use of the word “authority” has two very different meanings. One meaning describes positional power—such as the right to control, command, or determine—and the other describes expertise—an accepted source of information. Evidence obtained from an authority has to be carefully evaluated based on the expertise of the authority, while respectfully disregarding the power, influence, fame, charisma, attachment, or appeal of the authority. Trust and verify. Exercise critical thinking. A common and seductive fallacy is an appeal to authority. We are often mislead because of a natural tendency to trust some people and distrust others.

An authority often presents only a single point-of-view, and too often this point-of-view advances a vested interest. One example of this is an Internet site claiming to provide expert information on sleep problems as a public service. However, the web site is created, paid for, and edited entirely by the manufacturer of a particular prescription drug sleep aid. This is a manipulative marketing tool, disguised as a source of objective information. It uses factual statements to present a false message. Examine a variety of viewpoints and apply critical thinking to help evaluate information provided by an authority, or even by an aligned group of authorities.

Take particular care to evaluate the reliability of claims of divine or religious experience, pronouncements by authorities, appeals to common sense, the obvious, and other situations where information or conclusions are claimed to be self-evident, beyond question, or beyond our comprehension. When a person in power responds with a preemptive dismissal—refusing to seriously consider an inquiry, or replying without responding by using power, humiliation, ridicule, insult, intimidation, distraction, obfuscation, condescension, or humor—it is often because the evidence is absent or unsubstantiated. Arrogance, belligerence, shouting, sneering, and repetition do not validate evidence; instead these distractions should raise suspicions. Confident experts typically welcome critical examination and discussion of their findings. Charlatans do not. Retain a healthy skepticism. Challenge authority as needed to understand and evaluate their claims and assess evidence. Challenge claims with a respectful and tactful request to “show me” and “help me understand”.

In United States legal proceedings, the Daubert standard is used to determine when expert witness testimony is admissible. Under this standard, use of scientific methods is the primary consideration for establishing expertise.

Do not confuse expert opinion with layperson opinion, luminary opinion, or an appeal to authority. Expert opinion is stronger evidence that non-expert opinion and it is a fallacy to equate the two.

Accessible

Evidence can be evaluated more reliably when several investigators have unfettered access to the evidence. If the evidence is hidden, fleeting, inaccessible, or access to it is restricted, investigators cannot examine the evidence in depth to fully extract all the information it can provide. Stage magicians restrict access to the illusions they are performing to preserve the mystery they are creating. This is lots of fun, but is no way to conduct a serious investigation.

Repeatable

Reproducibility is the ability of an entire experiment or study to be duplicated, either by the same researcher or by someone else working independently. Reproducing an experiment is called replicating it. Reproducibility is one of the main principles of the scientific method.

Evidence resulting from experiments that have been independently replicated is more reliable than experimental results that have not been reproduced.

Survey Data

A survey is a list of questions aimed at extracting specific data from a particular group of people. Surveys may be conducted by phone, mail, via the internet, and sometimes face-to-face on busy street corners or in malls. Since survey research is almost always based on a sample of the population, the success of the research depends on how well the survey sample represents the target population of interest to the researcher.

Opinion polls are human research surveys of public opinion drawn from a particular sample. Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence intervals. Polls based on samples of populations are subject to sampling error which reflects the effects of chance and uncertainty in the sampling process. It is difficult to draw reliable conclusions form opinion polls because there are several sources of error in collecting and sampling the data. Sources of error include:

• Nonresponse bias results when people do not answer calls, or refuse to answer the poll.
• Response bias occurs when answers given by respondents do not reflect their true beliefs.
• Wording of questions affect results. It is well established that the wording of the questions, the order in which they are asked and the number and form of alternative answers offered can influence results of polls.
• Coverage bias occurs when the samples used are not representative of the population as a consequence of the methodology used.

A push poll is an interactive marketing technique, most commonly employed during political campaigning, in which an individual or organization attempts to manipulate or alter prospective voters' views/beliefs under the guise of conducting an opinion poll. A push poll is not an attempt to gather reliable survey data. Beware that the push poll is a form of telemarketing-based propaganda and rumor mongering, masquerading as an opinion poll.

Because of the complexities described above, it is difficult to estimate the reliability of survey data.

Representative

A statement can be entirely true without being representative of a broader reality. Consider these alarming yet true statements appearing on the dhmo.org website:

“Among the many commonly-sited DHMO-related environmental impacts are:

• DHMO contributes to global warming and the ‘Greenhouse Effect’, and is one of the so-called ‘greenhouse gasses.’
• DHMO is an ‘enabling component’ of acid rain — in the absence of sufficient quantities of DHMO, acid rain is not a problem.
• DHMO is a causative agent in most instances of soil erosion — sufficiently high levels of DHMO exacerbate the negative effects of soil erosion.
• DHMO is present in high levels nearly every creek, stream, pond, river, lake and reservoir in the U.S. and around the world.
• Measurable levels of DHMO have been verified in ice samples taken from both the Arctic and Antarctic ice caps.
• Recent massive DHMO exposures have led to the loss of life and destruction of property in California, the Mid-West, the Philippines, and a number of islands in the Caribbean, to name just a few.”

These completely true statements are quite alarming, until you realize that DHMO, Dihydrogen Monoxide, is actually a chemical name for water! This example demonstrates how true statements can be used to send a false message.

It is not enough for a piece of evidence, and especially evidence in the form of a statement, to be true; it must be representative of the conclusions drawn from it. For evidence to be representative:

• It must be relevant to the circumstances being investigated,
• The origin must be known, as stated, and relevant to the circumstances being investigated,
• The evidence must authentically represent what has happened. Hearsay, rumors, and gossip are unreliable and unrepresentative,
• The scope of the evidence must support any generalizations made from it. The scope of small samples, biased samples, and anecdotal evidence are too small to represent generalizations,
• Direct evidence is more reliably interpreted than circumstantial evidence. Many alternative explanations can fit the facts of circumstantial evidence.
• Evidence must be understood in context before reliable conclusions can be drawn.
• Claims made by experts are only relevant within a particular field of expertise.
• Evidence that is fleeting, hidden, unavailable, or partially accessible is unreliable and unrepresentative.
• To be reliable, survey data must be from a survey carefully designed to be objective and used with a representative sample of the population it is used to represent.
• Experimental results that have not been replicated support only tentative conclusions.

Instrument Accuracy

Often observations rely on instruments such as microscopes, telescopes, and various measurement devices. An instrument as common as a thermometer can illustrate the difficulties in obtaining an accurate result. As an example, try determining the temperature where you are right now. You might look at readings from several thermometers placed around the area, and you can also obtain a current weather report. Because these readings are likely to differ, you will need some reliable basis for interpreting the range of results. Consider the accuracy of each instrument. Is the measurement accuracy of each device specified? Has the device been properly calibrated? Is the measurement being taken within the operating range of the instrument? Consider proper use of the instrument. Is the thermometer in the sun or in the shade? Is it near some source of heat such as a fire, stove, equipment, or an engine?

Results from properly used accurate instruments are more reliable than results from inaccurate instruments. Reported results are incomplete unless they include a discussion of the instruments used, the measurement accuracy of each, calibration procedures, use procedures, multiple independent measurements, discrepant results, and various sources of individual and systematic measurement error.

Digging Deeper

Reality is very complex, and evidence often reveals that complexity as exploration of one layer begins to reveal the structure of deeper layers. For example, objects consist of matter, matter consists of molecules, molecules are made up of atoms, atoms consist of electrons, protons, and neutrons, protons and neutrons are made up of quarks, and research is currently underway to examine the substructure of quarks. A set of Russian nesting matryoshka dolls provide a useful analogy. The existence of the inner dolls is only revealed by carefully examining the outer doll, along with the curiosity to look inside.

When this set of Matryoshka dolls is assembled, the existence of the inner dolls is only revealed by carefully examining the outer doll

To gain a complete and reliable understanding of evidence, it is important to expect that the present evidence is revealing only the outer surface of what is often a very deep and complex structure. It is important to dig deeper to get a more detained, nuanced, and reliable understanding.

Studying various examples of science by press conference illustrates the importance of digging deeper, going beyond the headlines, going beyond the press release, and working to understand the peer reviewed studies that report the actual research. A particularly damaging case began in 1998 when Andrew Wakefield held a press conference to claim that the MMR vaccine caused autism. In January 2011, an article by Brian Deer and its accompanying editorial in The BJM identified Wakefield's work as an "elaborate fraud." During this time many parents elected not to have children vaccinated, angry parents of children on the autism spectrum were misled, and vaccine controversies continue today.

In 1912 bone fragments were presented as the fossilized remains of a previously unknown early human. This became known as Piltdown Man, and claims that it was the “missing link” between ape and man drew great attention. In-depth investigation of over a period of more than 40 years uncovered an elaborate hoax.

Headlines highlighting contradictory health or fitness advice are common. Investigate beyond the headlines to read the press release, understand the original research reports, and place the evidence in context to better understand the scope of the research results and determine … what is settled, what is disputed, what is new here?

Cross examination—the interrogation of a witness by the opposing counsel—is routinely practiced in law. The main purposes of cross-examination are to elicit favorable facts from the witness, or to impeach the credibility of the testifying witness to lessen the weight of unfavorable testimony. Cross-examination frequently produces critical evidence in trials, especially if a witness contradicts previous testimony. Cross examination allows the court to dig more deeply into testimony.

Reliable Sources

“Unpublished findings are not a good basis for making important decisions.” Statements drawn from unreliable sources may be no better.

Evidence is no more reliable than the sources from which it is obtained. Therefore it is important to assess the reliability of the various sources being consulted. The Wikipedia policy on identifying reliable sources provides well-thought-out guidelines that can be used to evaluate source reliability. Brief excerpts of that policy are provided here:

Scholarship material such as an article, book, monograph, or research paper that has been vetted by the scholarly community is regarded as reliable, where the material has been published in reputable peer-reviewed sources or by well-regarded academic presses.

News sources often contain both factual content and opinion content. "News reporting" from well-established news outlets is generally considered to be reliable for statements of fact (though even the most reputable reporting sometimes contains errors). News reporting from less-established outlets is generally considered less reliable for statements of fact. When consuming information provided by a news source, it is helpful to understand their stance toward journalistic objectivity so that you can better evaluate the material they are providing.

Reliable sources are not required to be neutral, unbiased, or objective, but their bias should be know and disclosed to readers. Common sources of bias include political, financial, religious, philosophical, or other beliefs.

Questionable sources are those with a poor reputation for checking the facts, or with no editorial oversight. Such sources include websites and publications expressing views that are widely acknowledged as extremist, that are promotional in nature, or that rely heavily on rumors and personal opinions.

Understanding the editorial policy of the publication can help to assess the reliability of the publication.

Scientific Studies

Scientific studies can be designed following one of several forms, including Randomized Controlled Trials, observational studies that are either retrospective or prospective, and case-control studies.

"In considering the veracity of scientific findings, studies published in a scientific journal should be given infinitely more weight than those that are not…” “A finding should be given more weight if there are multiple confirming instances...” These and other considerations can be combined to form an evidence hierarchy.

An Evidence Hierarchy.

Evidence hierarchies reflect the relative authority of various types of biomedical research. Although there is no single, universally-accepted hierarchy of evidence, there is broad agreement on the relative strength of the principal types of research, or epidemiological studies. Randomized controlled trials (RCTs) rank above observational studies, while expert opinion and anecdotal experience are ranked at the bottom. Some evidence hierarchies place systematic review and meta-analysis above RCTs, since these often combine data from multiple RCTs, and possibly from other study types as well. Evidence hierarchies are integral to evidence-based medicine.

When evaluating evidence from scientific studies it is helpful to identify the type of study, and determine where that type of study falls in the relevant evidence hierarchy. A levels of evidence pyramid provides a way to visualize both the quality of evidence and the amount of evidence available.

Similar levels of evidence are used throughout evidence-based practices to describe the strength of the results measured in a clinical trial or research study.

Optional Assignment

Read this paper on How to read and understand a scientific paper: a guide for non-scientists.

When a news report or headline makes a surprising claim based on a recently released scientific study:

• Obtain a copy of the scientific paper on which the news article is based.
• Read and understand that scientific paper, using the skills in the guide.
• Is the news report an accurate synopsis of the scientific findings?

Reliable Testimony

Evidence is often in the form of testimony. The reliability of the testimony depends on many factors.

It is important to understand the objectivity of the person giving the testimony, as discussed in the section on objectivity.

Eyewitness reports are more reliable than hearsay, rumors, or gossip, as discussed in the section on authenticity. Because eyewitness testimony relies on human memory, it is often inaccurate.

Eyewitness testimony may be inaccurate due to an eyewitness's memory being influenced by things that they might hear or see after the event was originally witnessed.

Testimony is anecdotal evidence, and does not provide a reliable basis for generalization, as discussed in the sect in on scope.

Various factors may diminish the reliably of testimony, including: age, mental status, stress, distraction, impairments, intoxicating agents, coercion, viewpoint, and others.

Reliable Methods

Crystal balls, tarot cards, astrology, clairvoyance, divine faith, and revelation are unreliable methods for obtaining evidence, forecasting the future, justifying belief, or making predictions. Reject these unreliable methods in favor of the most reliable methods known. Scholarly methods, including historical methods and scientific methods are constantly improved to ensure the resulting claims about the world are as valid and trustworthy as possible. E.O. Wilson describes the scientific method as “demonstrably the most powerful instrument hitherto created by the human mind.” Criminal investigations and forensic science apply scientific methods to reliably gather evidence related to criminal activities. Clinical trials are experiments or observations done in clinical research to gather reliable evidence. These are often designed and performed as randomized controlled trials, generally accepted as the most reliable method to assess proposed interventions. Randomized controlled trials are gaining acceptance throughout the social sciences. Statistical hypothesis testing can assess the degree of confidence in choosing one hypothesis over another.

Assignment

Study the module on Examining Ideologies, and complete the assignment in that module.

Convergence and Coherence

Reliable methods converge toward a coherent description of the real world. This is the principle of consilience—the unity of knowledge. When evidence obtained from various sources and methods fails to converge, the discrepancies must be vigorously investigated and resolved. This usually uncovers some error in some of the evidence collection as in the example of the faster-than-light neutrino anomaly.

In very rare instances resolving the discrepancy leads to revision of currently held theories or to major breakthroughs. Examples of such breakthroughs include the heliocentric model of the solar system, falsification of the Luminiferous aether hypothesis, continental drift, the germ theory of disease, and the existence of transposable elements in DNA sequences.

Correlation

Two variables are correlated if the one variable tends to increases (or decrease) as the other variable increases. For example, a person’s weight tends to increase as their age increases. Discovering a correlation between two variables begins to provide evidence that one variable depends on the other, but the evidence is not definitive.

Consider this fallacious argument:

As ice cream sales increase, the rate of drowning deaths increases sharply.
Therefore, ice cream consumption causes drowning.

It is more likely that children both swim more often and eat more ice cream in the summer heat, and that eating ice cream is not the cause of drowning.

Causation

It is well known, but often forgotten that correlation does not imply causation. Recognizing a plausible mechanism linking the effect to the cause is important, but not essential, for establishing a cause and effect relationship.

Astronomers can confidently state that sunrise and sunset are caused by the rotation of the earth as it orbits the sun. The movements of the earth around the sun are fully described. These movements describe a mechanism that explains sunrise and sunsets precisely as they are observed. Furthermore, this understanding of the earth’s motion is sufficiently detailed and accurate that the times of sunrise and sunset can be accurately predicted for any future date for any location on earth. Similarity, tides are caused by and explained by the combined effects of the gravitational forces exerted by the Moon and the Sun and the rotation of the Earth, and modified by local conditions including winds, currents, and flooding.

Corroboration and Falsification

Every white swan observed corroborates the claim that all swans are white, but a single black swan is sufficient to falsify the claim.

Biologists, ornithologists, and everyday Europeans noticed that swans are white. Thousands of swans were observed throughout Europe, and each one was white.

This naturally invites the conclusion that “All swans are white”, and every white swan observed provides additional corroborating evidence supporting the claim that “All swans are white”.

While exploring the Swan River in Australia in 1697, Willem de Vlamingh became the first European to record seeing a black swan. This single observation was sufficient to falsify the claim that “All swans are white.”

While every observation of a white swan provides corroborating evidence, a single black swan is sufficient to falsify the claim that all swans are white. This asymmetry illustrates the problem of induction. No number of confirming observations can verify a universal generalization, such as All swans are white, since it is logically possible to falsify it by observing a single black swan. Evidence that is consistent with a universal generalization corroborates, but can never prove, that generalization.

Explanation

An explanation attributes an effect to some set of more basic causes. Explanations attempt to answer the questions Why does this happen? or How does this work? An explanation often proposes some mechanism to account for a particular set of evidence.

Throughout the ages people have observed the sun rise each morning and set each evening. Curious people have wondered what causes the sun to rise and have sought explanations for their observations. Over the millennia, various explanations have been proposed within various cultures.

In ancient Greece the god Helios was described as a handsome titan crowned with the shining aureole of the Sun, who drove the chariot of the sun across the sky each day to earth-circling Oceanus and through the world-ocean returned to the East at night. This explanation failed to account for the motions of the moon, the stars, and the various planets that were observed. Chariot mechanics also remained a mystery.

In a more comprehensive explanation, sunrise, along with the movement of stars and planets were explained by celestial spheres. In these celestial models the apparent motions of the fixed stars and the planets are accounted for by treating them as embedded in rotating spheres made of an aetherial, transparent fifth element (quintessence), like jewels set in orbs. This model accounted for more available evidence than the chariot model; however it failed to accurately account for various astronomical observations being made by Kepler, Galileo, Tycho Brahe, and others.

Today’s heliocentric model places the sun at the center of our solar system, with the various planets revolving around the sun in elliptical orbits. The insights of Isaac Newton accurately explained the motions of the planets using three simple laws of motion and the law of universal gravitation.

Recent more detailed observations challenged details of Newton’s analysis and have brought us the theories of general relativity and speculation on the existence of Planet Nine.

Explanations are incomplete and unsatisfactory unless they adequately account for all of the available evidence. In the sunrise example above we see how more comprehensive and coherent explanations supersede less comprehensive explanations. Explanations continue to evolve until they can coherently account for all available evidence from a global perspective.

When encountering crop circles it is natural to ask, how can their formation be explained? Two possible explanations are alien visitors or pranksters. A narrow examination of any particular crop circle is unlikely to uncover evidence favoring one of those explanations over the other. Broadening the investigation opens questions like: When were the crop circles formed? How did the aliens or pranksters arrive, do their work, and leave? Is there any evidence of their arrival, visit, or departure? Have any aliens or pranksters been seen in town? Why would an alien or a prankster to this? Do we have any evidence of alien visitors or pranksters elsewhere? As more evidence is gathered, it may begin to favor one explanation over the other.

Occam’s razer can be helpful in choosing among several equally possible explanations. The principle advises: among competing hypotheses, the one with the fewest assumptions should be selected. It is often best to choose the simplest explanation that fits all the evidence.

Noncontradiction

Aristotle’s law of noncontradiction can be stated as:

“The most certain of all basic principles is that contradictory propositions are not true simultaneously.”

Aristotle says that without the principle of non-contradiction we could not know anything that we do know.

It is important to keep in mind the distinction between the law of non-contradiction, which is true, and the fallacy of false dichotomy. The difference is the distinction between a color that is either black or non-black (e.g. all other colors) and black or white (e.g. excluding all other colors).

Because reality is coherent, various pieces of evidence that appear contradictory are not yet fully understood. The apparent contradiction must be resolved before the evidence can be fully and accurately interpreted. More evidence, a new explanatory hypothesis, or a new interpretation of the existing evidence is needed to resolve the contradiction.

Rhetoric

Rhetoric is the skillful use of language for the purpose of influencing others. Rhetoric makes use of three primary strategies:

• Logos—an appeal to evidence, reason, and logic,
• Ethos—an appeal to character and similar values, and
• Pathos—an appeal to emotions.

Skillful orators adjust their use of these three strategies to influence their audience as powerfully as possible. It is helpful, although often difficult, to distill reliable evidence from rhetoric. The approach is to strip the ethos and pathos away from the rhetoric and reveal the remaining logos. With the logos exposed, then identify the conclusions presented, the logic supporting those conclusions, the premises used to support those conclusions, and the evidence provided for those premises. In short, has a sound argument been presented?

Shouting is not evidence. Displays of passion are not evidence. Repeating unjustified beliefs is not evidence. Righteous indignation is not evidence. Pity is not evidence. Hope is not evidence. Divine faith is not evidence. The extent to which anyone resorts to such drama is direct evidence of their lack of evidence.

Rhetoric cries out for fact checking. The resources provided for fact checking section of this course can provide a starting point.

Other courses in this Clear Thinking curriculum can help you to analyze arguments and determine if they are sound.

Assignment

Provide samples of rhetoric to analyze.

Mumbo jumbo

Mumbo jumbo is confusing or meaningless language. It may refer to practices based on superstition, rituals intended to cause confusion, or languages that the speaker does not understand.

Consider this excerpt from a speech Sara Palin gave supporting the candidacy of Donald Trump:

“Trump’s candidacy, it has exposed not just that tragic ramifications of that betrayal of the transformation of our country, but too, he has exposed the complicity on both sides of the aisle that has enabled it, okay? Well, Trump, what he’s been able to do, which is really ticking people off, which I’m glad about, he’s going rogue left and right, man, that’s why he’s doing so well. He’s been able to tear the veil off this idea of the system. The way that the system really works, and please hear me on this, I want you guys to understand more and more how the system, the establishment, works, and has gotten us into the troubles that we are in in America…”

Deepities

Philosopher Daniel Dennet identifies a particular form of mumbo jumbo he calls “Deepities”. He defines a deepity as:

A proposition that seems both important and true—and profound—but that achieves this effect by being ambiguous.

As an example, he uses the statement: “love is just a word.” This is trivially true because l-o-v-e is indeed a word in the English language. This obvious truth lures the reader into contemplating some deeper meaning, which may seem profound, but is indeed false. Love is a complex concept involving emotions, attachment, caring, and much more. Although you can find the word “love” in the dictionary, you cannot find love in the dictionary.

The RationalWiki article on deepties analyzes many more examples.

Notice deepties as you encounter them, and then quickly dismiss them as nonsense.

Assignment:

The purpose of this assignment is to extract reliable evidence from mumbo jumbo.

1. Select some sample of mumbo jumbo. Use the Sara Palin excerpt quoted above, or some other excerpt.
2. Considering this as rhetoric, extract the logos and discard the ethos and pathos from the passage.
3. Identify the conclusions, the argument logic, the premises provided for each conclusion, and the evidence provided for each premise.
4. Recast the argument in a form that makes the evidence, premise, logic, and conclusions readily apparent.
5. Using the original text as evidence, what conclusions does it support?

Bullshit

Bullshit is a slang profanity term meaning "nonsense", especially as a rebuke in response to communication or actions viewed as deceptive, misleading, disingenuous, unfair or false.

Bullshit provides us direct evidence of bullshitters. Learn to recognize bullshit and dismiss it along with those who spread it.

When U.S. Representative Paul Broun tells us the earth is 9,000 years old, this reveals nothing about the earth, and everything about Paul Broun.

Alternative Explanations

A person’s beliefs will depend on the variety of alternative explanations, causes, or hypothesis that come to mind as plausible explanations for the evidence. Defense attorneys often present an “alternative theory of the case” to provide some alternative narrative that also fits the evidence. This approach applies beyond the courtroom.

When crop circles occur, evidence for their existence is readily available. Investigators can visit the field, examine upright crops, examine flattened corps, take crop samples and investigate them in a laboratory to estimate when they may have been damaged, and look for any marks or other evidence of what may have happened. Aerial surveillance including direct observation, along with video and photographs can establish the extent of the patterns. The existence of crop circles is rarely disputed.

Questions of “how did this happen?” or “who did this?” naturally arise. Two explanatory hypotheses are most often offered:

• H₁ – This is the work of alien visitors.
• H₂ – This is the work of pranksters.

Evidence can then be sought to decide between these two hypotheses. Locals are asked: Did you see anything? Were the crops upright last evening? When were the crop circles first discovered? Who first discovered them? Have you noticed any strangers in town? Did you see any lights in the sky? This evidence is rarely convincing one way or the other. The evidence supporting your choice of H₁ or H₂ after examining crop circles generally comes down to your estimate of the likelihood of Aliens vs Pranksters before you learned of crop circles.

Other controversies demonstrate that conclusions after the event often depend on the beliefs you hold before the event.

Consider the ongoing controversy over the assassination of US President John F. Kennedy. Two explanatory hypotheses are most often suggested:

• H₁– The Single Bullet Theory: Oswald acted alone, and a single bullet killed the President and wounded the Governor, and
• H₂ – The Accomplice theory: More than one bullet was fired, Oswald had an accomplice.

Evidence for and against these two hypotheses continues to be collected and debated today, more than 50 years after the event. An important factor determining your belief in H₁ or H₂ is your prior belief in the veracity of an official government investigation, compared to the likelihood of a larger plot.

At the time Galileo Galilei pointed his telescope toward the sky, the Catholic Church taught that the earth was the center of the universe. Galileo’s observations of the earth’s moon, the moons of Jupiter, and the phases of Venus convinced him that the earth circled the sun. Two explanatory hypotheses for events such as daily sunrise and sunset, phases of the moon, and motions of the stars and planets then clashed:

• H₁—Geocentric: The earth is the center of the universe, or
• H₂—Heliocentric: The earth revolves around the sun

The primary evidence for the geocentric model was divine revelations about the origins of the universe advocated by the Catholic Church. Religious opposition to heliocentrism arose from Biblical references such as Psalm 93:1, 96:10, and 1 Chronicles 16:30 which include text stating that "the world is firmly established, it cannot be moved." In the same manner, Psalm 104:5 says, "the Lord set the earth on its foundations; it can never be moved."

The primary evidence for the heliocentric model included many careful astronomical observations made by Galileo, and available to be replicated by anyone peering through his or any other telescopes. The controversy continued with Pope Paul V ordering Galileo:

... to abandon completely... the opinion that the sun stands still at the center of the world and the earth moves, and henceforth not to hold, teach, or defend it in any way whatever, either orally or in writing

Later when Isaac Newton was able to calculate the elliptical orbits of the planets based on his law of universal gravitation, the heliocentric model gained increased acceptance.

Do you find religious teachings more or less reliable evidence than scientific observations?

Present Worldview

Evidence bears the burden of overcoming presently held beliefs. Presently held beliefs are formed from the evidence accumulated throughout our lives and often form a coherent worldview. New evidence is persuasive depending on the strength of the a priori, at least as much as on the strength of the evidence.

Greek Philosopher Epictetus noted: "It is impossible for anyone to begin to learn that which he thinks he already knows."

If someone is 100% certain of their current beliefs, then no amount of contrary evidence can overcome their certainty. Their current beliefs have become immune to evidence. Because it is widely held that absolute certainty about the real world is a failed historical enterprise, inexorable people who are 100% certain of their beliefs have lost touch with reality.

Assignments

The assignments in this section require the student to draw on many lessons in the course.

Part 1:

The purpose of this assignment is to practice evaluating evidence in actual cases.

1. Scan this list of hoaxes, concentrating on the “Proven Hoaxes” and choose one to study for this assignment. If you are having difficulty choosing, perhaps the Piltdown Man hoax will be interesting.
2. Study accounts describing the hoax, the evidence presented, early suspicions about the hoax, and the investigations that finally demonstrated the claims were part of a hoax.
3. What defects in the evidence were uncovered, if any, that revealed the hoax?
4. Read this list of Questions for Evaluating Evidence.
5. What, if any, questions on that list could have uncovered the hoax more quickly if investigations had focused on answering that particular question?

Part 2:

Research and evaluate the evidence for and against one or more of the following controversies to decide for yourself:

• Is the Shroud of Turin the burial cloth of Jesus?
• Where was Barack Obama born?
• Does the naturally occurring process of evolution explain biodiversity?
• Does homeopathy provide effective disease treatment? For what diseases, if any, is it effective?
• Is Uri Geller able to bend spoons using psychic powers?
• Are crop circles created by alien visitors or by pranksters?
• Does vaccination improve health outcomes?
• Does cupping therapy provide health benefits?
• Can reincarnation allow a living being to start a new life in a different physical body?
• Does the Loch Ness Monster exist?
• How old is the earth?

In each case studied, discuss how you weighed the evidence available to come to your conclusion.

Part 3:

The case studies presented here provide examples and encourage practice in identifying misleading interpretations of evidence. Study cases from this list that interest you.

Part 4: Complete the Wikiveristy course on exploring worldviews.

Resources for Fact Checking

The sources listed here are often more reliable than other sources. Start here as you work to evaluate various forms of evidence, especially rumors, claims made on social media, advertisements, opinion pieces, ordinary conversations, and gossip. Verify statements before accepting them as facts.

• Snopes.com is a well-known resource for validating and debunking such stories in American popular culture. Check rumors here before believing them or passing them along.
• FactCheck.org is a nonprofit website that describes itself as a non-partisan "consumer advocate for voters that aims to reduce the level of deception and confusion in U.S. politics." Most of its content consists of rebuttals to what it considers inaccurate, misleading, or false claims by politicians. FactCheck.org has also targeted misleading claims from various partisan groups.
• PolitiFact.com is a project operated by the Tampa Bay Times, in which reporters and editors from the Times and affiliated media fact check statements by members of Congress, the White House, lobbyists and interest groups.
• Wikipedia:Current science and technology sources—A list of resources that are likely to be reliable for science and technology articles.
• Wikipedia:News sources—a list of links that can be used to research current events and news stories.
• The Wikipedia:Reliable sources checklist can help organize your evaluation of a reference.

Further Reading

Students interested in learning more about evaluating evidence may be interested in the following materials:

• Seethaler, Sherry (January 23, 2009). Lies, Damned Lies, and Science: How to Sort Through the Noise Around Global Warming, the Latest Health Claims, and Other Scientific Controversies. FT Press. pp. 224. ISBN 978-0132849449. 
• Pinker, Steven (September 28, 2021). Rationality: What It Is, Why It Seems Scarce, Why It Matters. Viking. pp. 432. ISBN 978-0525561996. 
• Jarrard, Richard D.. Scientific Methods. https://archive.org/details/sm_all_cc.  This book is slowly moving through the Wikisource validation process and is available at: Scientific Methods
• Burton, Robert A. (March 17, 2009). On Being Certain: Believing You Are Right Even When You're Not. St. Martin's Griffin. pp. 272. ISBN 978-0312541521. 
• Wilson, Edward Osborne (March 30, 1999). Consilience: The Unity of Knowledge. Vintage. pp. 384. ISBN 978-0679768678. 
• Frankfurt, Harry G. (January 30, 2005). On Bullshit. Princeton University Press. pp. 67. ISBN 978-0691122946. 
• Caulfield, Michael A.. Web Literacy for Student Fact-Checkers. BCcampus OpenEd. pp. 127. https://open.bccampus.ca/find-open-textbooks/?uuid=ccbb4e77-b20f-4dec-8a9f-67ccc9bc774b. 
• Doyle, Sir Arthur Conan (October 1, 1986). The Complete Sherlock Holmes (2 Volumes). Bantam Classics. ISBN 978-0553328257. 
• Andersen, Kurt (September 5, 2017). Fantasyland: How America Went Haywire: A 500-Year History. Random House. pp. 480. ISBN 978-1400067213. 
• Galef, Julia (April 13, 2021). The Scout Mindset: Why Some People See Things Clearly and Others Don't. pp. 288. ISBN 978-0735217553. 
• Ariely, Dan (September 17, 2024). Misbelief: What Makes Rational People Believe Irrational Things. Harper Perennial. pp. 320. ISBN 978-0063280434. 

I have not yet read the following books, but they seem interesting and relevant. They are listed here to invite further research.

• [Evaluate the book How Science Works: Evaluating Evidence in Biology and Medicine, by Stephen H. Jenkins ]
• [Evaluate the book Bell, Swenson-Wright, and Tybjerg, eds., 2008. Evidence, Cambridge: Cambridge University Press.]
• [Evaluate the book Patterns of Discovery: An Inquiry into the Conceptual Foundations of Science, by Norwood Russell Hanson ]
• [Evaluate the book The Book of Evidence, by Peter Achinstein ]
• [Evaluate the book The Death of Expertise: The Campaign Against Established Knowledge and Why it Matters, by Thomas M. Nichols ]