Be Skeptical: Finding and Evaluating Online Health Resources

Adapted from: Peper, E. & Harvey, R. (2023). Be skeptical: Finding and evaluating online health resources. Townsend Letters. The Examiner of Alternative Medicine, October 21, 2023. https://www.townsendletter.com/e-letter-20-evaluate-sources-to-make-informed-choices/

Erik Peper, PhD, BCB and Richard Harvey, PhD

Source: https://live.staticflickr.com/65535/48445803437_726b61e3d1_b.jpg

An unprecedented flood of information is available today at our fingertips in the form of cell phone apps, news stories, blog posts, social media feeds, advertisements, websites, videos, and audio resources. Artificial intelligence (AI) applications such as ChatGPT are also capable of curating health and wellness information all proclaiming to optimize our health or treat our illnesses. This article provides strategies to determine how to trust the information.  It offers strategies for assessing information, reasons to have a skeptical perspective, suggestions for finding credible resources and includes a framework to identify beneficial health information, which may be used for improving activities of daily living. The recommendations are based upon an evolutionary perspective in which anything that was not part of our evolutionary past should be viewed with healthy skepticism.

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines.  I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.” —Dr. Marcia Angell (2009), the first woman editor of the highly respected New England Journal of Medicine.

How to make sense of the flood of health information

An unprecedented flood of health information is available today proclaiming useful information to optimize our health or treat our illnesses, A simple question is, “How do we know which information is accurate?” To what extent do we trust the information in an era of fake news, commercial health apps trying to sell us things, and news stories from publishers and media conglomerates that are dependent on advertising revenues? This article offers strategies for assessing information, reasons to have a skeptical perspective, and suggestions for finding relevant and accurate information.

Skepticism about health information takes many forms including ‘conspiracy theories’ about vaccines such as when people claim a SARS-COVID-19 vaccine will kill you, to when people doubt the efficacy of HIV or similar vaccines. Several authors have suggested political as well as individual personality factors which explain conspiratorial skepticism about health information, most commonly about vaccines (Crescenzi-Lanna, Valente, Cataldi, & Martire, 2023; Koinig, & Kohler, 2021; Putois, & Helms,. 2022). This article takes a broader view of health information skepticism, focusing on perspective building as well as asking relevant, accurate and meaningful questions about health care decisions. 

Take a skeptical perspective and ask, “What is the evidence that the product, procedure, or treatment is going to be effective for me compared to others?” The answer could appear obvious: published peer-reviewed systematic meta-analyses of double blind, randomized, controlled trials describing specific products or procedures. However, the answer is more complex. In numerous cases, finding relevant reports can be challenging. In some cases, it may be unethical or impossible to run double blind, randomized, controlled trials to detect the scope of effectiveness or generalize the finding from animal studies to human beings. For example, surgery cannot be evaluated in a double-blind study. (Would you really want your surgeon not to be aware of what he/she was doing?). Although treatment effectiveness can be studied using a matched comparison or a control group receiving mock surgery, in those cases the surgeon would still be aware of the procedure.

The Challenges of Assessing Clinical Efficacy

It is challenging to know what actually contributes to the beneficial outcomes as well as how to measure the outcome.  Some of the factors that affect the outcomes include:

Placebo interactions: Intrinsic to all procedures are placebo and nocebo components. In some cases the direct benefit effects of a drug or procedure demonstrated in a randomized controlled trial may still not be due solely to the direct effects of the drug or procedure, but rather due to positive indirect effects triggered by the placebo response to non-specific side effects (Peper and Harvey, 2017).

Lack of evidence is not proof that it does not work. Lack of replicable evidence for some cases implies that a positive response will not occur in all cases. Unfortunately, commercial interests may bias interpretations of research studies when the efforts to replicate a study had limitations in the first place, or the replication efforts did not retain transferable conditions to the next study. In other words, ‘ceteris paribus’ may not apply as all things are not always equal during replication studies. Similarly, individual differences that are outliers or extreme values during a study (e.g. positive benefit from placebo) can be ‘explained away’ with statistics because statistics may also skew the interpretations based on the biases of the researchers.   

Clinical trials are very expensive. The average clinical trial for a new therapeutic agent, 2015–2017, was $48 million dollars (Moore et al., 2020). The cost of achieving Food and Drug Administration (FDA) approval is so high that it is often out of reach for small companies. It is no wonder that most clinical trials are funded by the pharmaceutic industry and only for those drugs for which they foresee significant profits.  The estimated research and development investment to bring a new medicine to market is estimated to range between $314 million to $2.8 billion (Wouters et al, 2020). To be financially viable, this usually means that drugs must be used by a large consumer base and ideally be taken for the rest of the individual’s life. Non-drug approaches may be less profitable, so without a profit incentive, investigations of non-drug efficacy accumulates less evidence compared to multi-million dollar trials.

Human beings are not rats, mice, or monkeys.  The findings from animal studies in numerous studies provide some useful insights into the effects of medications or procedures on living organisms. Unfortunately, many results from animal studies could not be replicated in humans or, the findings may not apply to human beings. The basic assumption that animal studies could mimic human studies may not be valid since almost all test animals are not typical of normal animals, implying test animals are ‘abnormal’ in terms of results. For example, the animals such as rats are usually housed in small cages 24 hours a day which is analogous to a human being held in solitary confinement without social contact or ability to move for a lifetime. Thus, their physiology and their response to interventions are often different from healthy free ranging animals (Shaw, 2023).

Even when animal studies show that the medications are not harmful, they could be harmful for some human beings. For example, thalidomide was approved for use in Germany, so doctors prescribed it to treat morning sickness in pregnant women. However, in humans Thalidomide interfered with embryonic and fetal development in ways not observed in rodent tests (Tantibanchachai & Yang, 2019).

Statistical significance may not indicate meaningful clinical improvement. Many clinical studies demonstrate that the studied interventions have contributed to improvement. However, does the improvement make a quality of life (QOL) difference and/or clinically relevant difference for the person? For example, a successful study that demonstrated lowering of patients’ systolic pressure by 5 mm from 175 mm/Hg to 170mm/Hg may be statistically significant, but is not clinically meaningful, since, a resting systolic blood pressure of 170 mm/Hg is still a cause for concern.

Similarly, in the recent systematic review by Arciero et al. (2021) of approved oncology therapies, 40% of Food and Drug Administration (FDA)-approved as well as 58% of European Medicine Agency (EMA)-approved indications had published QOL evidence. However, only 6% of FDA- and 11% of EMA-approved indications had clinically meaningful improvements in QOL beyond minimally clinically relevant differences. This means that medication therapies are often approved without demonstrating Quality of Life improvement for the long term.

Statistics which describe how large an effect is may be referred to as an ‘effect size estimate, which is a better index of efficacy compared to other statistics such as a difference in statistical mean values. The effect size can be assessed by using various statistics such as Cohen’s d-statistic (Mean A minus Mean B divided by pooled standard deviation; Cohen, 1988)[i].

Number of people need to be treated for one person to benefit. Effect size calculation estimates the average number of people in a trial needing treatment so that one of them experiences benefit.  This statistic can be referred to as the number needed to treat (NNT) (Mendes et al., 2017). To calculate the NNT, divide 1 by the control event rate (CER) minus the treatment event rate (TER) or 1/CER-TER. For example, the number patients needed to be treated for five years with cholesterol lowering (e.g., statin drugs) medications to prevent one coronary heart disease event ranges from 53 (high risk group) to 146 (low risk group) (Rossignol et al., 2018). This means that many of the participants could experience negative side effects related to the medications while only one participant benefits from the prevention of a heart attack.

Focus on short-term versus long-term benefits. Many studies measure outcomes under highly controlled conditions of a study and are conducted for a relatively short time period—often for less than 3 months. However, effects that may be beneficial in the short term may not be beneficial or may even be harmful in the long term. For example, opioid medications are very useful in the short term to alleviate intense pain. However, over time, drug dependency may develop, contributing to addiction, inability to function, or death. Shockingly, opioid-related deaths in the U.S. numbered more than 100,000 people in 2022 (CDC, 2022).

Benefits do not enhance quality of life.  If the data indicate benefits of treatment, do the interventions improve quality of life and not simply prolong life for a few days, weeks, or months? Does the patient or client value quality of life over quantity of days lived (e.g., ”palliative care with shorter life, but some relief from pain and suffering versus prolonged life with pain and suffering”)?

Results may only apply to a select group. Biochemical individuality means that each person is unique to some degree, differing genetically, biochemically, and physiologically. Similarly, responses vary widely to medical procedures, medications, and other substances. A common example is alcohol sensitivity— the genetic predisposition to metabolizing alcohol breakdown— manifesting in highly visible facial flushing which occurs in 47%-85% of Asians and 3%-29% of Caucasians (Chan, 1986). In the context of medicine, individual differences that influence clinical outcomes include genetic predisposition, as well as age, gender, income, education level, job status, geographic region (e.g., climate and food sources) and other demographic factors, individually or in combination.

Consider that many interventions and medications have only been tested on narrowly defined subgroups such college students (true of most psychological studies), or men (true for most pharmaceuticals since women could be pregnant or in different phases of their menstrual cycle). The promise of personalized or ‘precision’ medicine will likely advance in the coming years, making medications more tailored to individual differences based on age, sex, and other demographic factors. 

There is no free lunch. Similar to the concept of short-term versus long-term benefits, when a drug offers a quick improvement, it may be effective, but may cause long-term harm. A representative example is the use of high-dose and multi-doses of anabolic steroids to increase muscle mass and athletic performance. There is a potential cost: “High and multi-doses of anabolic steroids used for athletic enhancement can lead to serious and irreversible organ damage” (Maravelias, et al., 2005).

Risks of hazardous exposures and risks associated with the treatment.  Could the procedure or medication result in loss or harm? Given bio-individuality, there can be broad “variability” in response and outcome, which depends on the vulnerability of a given individual (their adaptive capacity) and the risks involved. Additionally, there are sometimes important variables that have not been investigated deliberately because those important variables complicate interpretation, and or, there may be important variables that are missed The most obvious example of omission is when animal studies were or are conducted exclusively on male animals because interpreting results can be more complicated given female reproductive hormones. Beside sex variables other important variables that may be missed include covert illnesses and co-morbidities which are unknown at the time of the study. Taken together, studies which oversimplify variables may make it difficult interpret the results for individuals.

Below is a set of images relevant to climate change and farming, depicting the relationship between the exposure to hazards of systemic climate change and the vulnerabilities, sensitivities, and adaptive capacities of individuals and the community (Wilhelmi and Hayden, 2016).

Use “Uncommon” Sense

The attractive look and feel of a website are not evidence of accuracy or credibility. Rather, good design simply means it was developed by a skilled web designer or that the client paid a great deal of money to have it created. It does not make the content valid. The comments of Yucha (2002) and Yucha and Montgomery (2008) remind readers to increase literacy regarding ”health claims” made on websites, especially a commercial website intended to sell products or services.

Evaluating dietary supplements. If you’re thinking about using a dietary supplement, check the recommendations from reliable sources. Make it a point to purchase a reputable brand, since some supplements contain ingredients not listed on the label. In addition, they may interact with medications or other supplements. Share and discuss all your supplements you are taking with your healthcare provider. For example, vitamin E acts as anticoagulants and may increase clotting time and bleeding especially if one is taking “blood thinners.”

Follow the money. Ask who would financially benefit from the product or service? For example, physicians increase their referrals for lab testing, MRIs (Magnetic Resonance Images), or other diagnostic procedures if they have ownership in those testing centers or, if they receive significant reimbursement for those services, although there is no evidence that patients benefit more (Bishop et al, 2010).

Beware of advertised claims. Most highly advertised drugs are largely no better at treating a disease than generic medication or other options (Patel et al., 2023). Pharmaceutical companies in 2021 spend $6.88 billion for direct to consumer advertising (Faria, 2023). The advertisement suggests that their branded medication is better; however, generics are about 80% cheaper and have the same active ingredient and are similar in their action (AAM, 2020).

If the claims seem unbelievable, they are probably are unbelievable. If it is too good to be true, it probably is not true. Historically, Thomas Lupton (1580) wrote a thoughtful inquiry about religion and utopian societies, introducing a skeptics point of view, describing people and societies that are ”too good to be true.”  Modern skeptics consider the preponderance of evidence based on scientific replicability (the replication of findings in subsequent clinical trials) as proof of what they believe to be true.

Source: Indiana University of Pennsylvania, last accessed March 3, 2023 https://www.iup.edu/instructional-design/images/assessment.jpg

Assessing Online Information

What do we know about the accuracy of online health information? A skeptical viewpoint is that bias exists in sources of information from a wide range of commercial, organizational, governmental and educational institutions (identified by ending with .com, .org, .gov and .edu, respectively). Most all institutions set out to prove their own bias; however, people working in educational institutions by and large require their investigators go through a peer-review process, so they tend to be more trusted as sources of information. Commercial, organizational, and governmental institutions all have biased perspectives. However, they are less likely to reveal their biases, simply stating that “a study was conducted” without providing enough information who funded the study or the importance of positive results to achieve academic recognition.

“A lot of what is published is incorrect … much of the scientific literature, perhaps half, may simply be untrue.  Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.”  Dr. Richard Horton (2015), Editor-in-Chief of The Lancet.

Be skeptical of university or published research findings that are directly or indirectly funded/influenced by industry or commercial sources. Government research published in highly respected scientific journals may not be replicable because the investigations were narrowly designed to favor a particular bias. For example, industries that produce pharmaceuticals and medical devices, as well as agribusinesses that produce tobacco and sugar products, have been accused of a ”profit-first” bias (Bruening, 2019; Hill et al., 2019). These industries often support studies conducted by “independent” researchers at universities. However, grant funding quickly disappears if the findings are negative which may affect the career of the researcher because many university faculty positions and promotions depend upon the faculty member’s ability to garner grants.

Compare US safety guidelines to those of the EU. In many cases, the acceptable values are different. The safety limits for herbicide and pesticide residues in foods are often much lower in the EU than in the US (e.g., safer with lower exposure levels). For example, the US allows six times as much residue of the pesticide, Round-Up, with a toxic ingredient, glyphosate, in foods consumed in the American diet (Tano, 2016).  The USA allows this higher exposure even though about half of the human gut microbiota are vulnerable to glyphosate exposure (Puigbo et al., 2022).  A skeptical view of research could adapt a precautionary principle such as “if you think it could cause harm then do not use it until proven safe.”

Government guidelines and directions may not always be accurate. For example, after 9/11 the CDC initially announced that the particulate dust from the World Trade Center collapse was not harmful The CDC made this claim without any data, in efforts to reassure the public. In fact, the dust was harmful. More recently, some of the politicization of the CDC COVID-19 recommendations have raised questions. For example at the beginning of the pandemic, the CDC publicly recommended “If you are NOT sick: You do not need to wear a facemask unless you are caring for someone who is sick (and they are not able to wear a facemask)” which suggested that masks were not necessary. (McReynolds, 2020). Most likely, the statement was made so that more masks would be available for medical workers. The statement would have engendered more trust if the CDC had stated, Masks are useful; however, please make your own, since the medical-grade n95 masks are in very short supply and needed to protect the frontline health professionals who are most at risk.

The National Personal Protective Technology Laboratory (NPPTL) describes the types of masks needed to protect medical and other types of workers such as fire fighters, where the numbers represent the percentage of particulates filtered (e.g., 95%, 99%, 100%) and the letters represent the types of particles (e.g., N = not resistant to oils, R = resistant to oils and, P = strongly resistant to oils). Other countries have similar mask standards, where a N95 mask in the United States (i.e., N95; NIOSH-42 CRF 84) is equivalent to masks in Europe (FFP2; EN149-2001), China (KN95; GB2626-2019), Australia (P2; AS/NZ 1716-2012), Korea (KF94; KMOEL 2017-64), Japan (DS2; JMIHLW 214-2018) and, Brazil (PFF2; ABNT/NBR 13.698.2011). The reason for including the technical details about masks is to remind the reader that both governments as well as other sources of health information may hide some of the information about potentials for loss or harm behind lots of technical details, so knowing how to compare information becomes relevant when making health decisions.

Patient population in the research study may not represent the average patient (referred to as Berkson’s bias). Research study subjects may have multiple co-morbidities or may all be healthy young males. In either case, they may not be representative of the general patient population nor of individuals (Westreich, 2012).

The data does not discuss or excludes outliers? Positive findings, even in randomized, placebo-controlled studies, mean that the treatment approach is more beneficial than the control condition. In almost all cases, some participants respond extremely well and some very poorly, often referred to as statistical outliers. What is usually not reported are the characteristic of the ‘super responders’ or ‘non-responders.’ Have more trust in studies that provide a full range or a wider range of information about the positive and negative responders, rather than simply reporting about the average response.

The research review is highly selective. Meta-analyses and review articles evaluate the outcomes of multiple research studies. However, typically they include only well designed randomized controlled trials. In many of these studies, 95% of the published articles are excluded because they did not fit the narrow criteria of the randomized selection. Thus, these meta-analyses may exclude conditions under which the treatment approach would be highly beneficial to a specific set of people. When the meta-analyses identify the studies that are excluded and why, it is possible to learn of the biases of the meta-analyses.

Funding for research or clinical trials favors products or technologies which can be patented, commercialized and support industry profits. There is extensive funding for new drug development for the treatment of COVID-19 or hypertension, but limited funding for diet or lifestyle changes that could optimize the immune system. If a product or drug is beneficial however not patentable, it is unlikely that a pharmaceutical company will further develop and market it because competitors could easily produce it. For example, pharmaceutical companies do not advertise vitamin D₃ supplements because it is not patent protected even though a preponderance of independent research has clearly demonstrated that the incidence of symptoms following metastatic cancer diagnosis is reduced with vitamin D₃ supplements (Chandler et al., 2020).

Be aware of the revolving door. The top administrators of numerous US regulating agencies such as the Food and Drug Administration (FDA) and the US Department of Agriculture (USDA) are often rewarded with well-paying jobs in the pharmaceutical, healthcare, and agribusiness industries after leaving jobs in the US government. For example,

• Scott Gottleib, MD was head of the FDA 2017-2019 before leaving the government in 2019 and then has been a board member of Pfizer pharmaceuticals as independent director. (https://www.pfizer.com/people/leadership/board_of_directors/scott_gottlieb-md).
• Stephen Han was the FDA commissioner under President Trump and approved the emergency use of the Moderna vaccine. Six months later, he quit and joined Flagship Pioneering, the firm behind Moderna. (https://www.fiercebiotech.com/biotech/six-months-after-granting-moderna-covid-19-eua-ex-fda-commish-joins-biotech-s-founding).
• Joby Youn, who served as chief of staff to former U.S. Department of Agriculture Secretary Sonny Perdue, went on to work for the American Farm Bureau Federation. (McVan, 2022).

A skeptical question to be raised is to what extent does the promise of well-paying jobs impact the decisions of administrators who are in charge of regulating industries that may offer a high paying job in the future.  Would you avoid antagonizing those companies thereby risk a future financial windfall? Similar conflicts of interest may be at play in other industries. For example, Boeing’s close relationship with the Federal Aviation Administration (FAA) by giving initial approval of Boeing 737 Max airplane that may have contributed to the two fatal airplane crashes (Cassidy, 2020).

Use critical thinking and don’t rely solely on the first internet search results resources when making decisions about your health. Many commercial companies (e.g., internet resources ending with ‘.com’) will pay to be on the first page of an internet search. Consider using more advanced internet search results that access ‘scholarly’ information, often available from ‘.edu’ sources. Consult with your health care provider when you are considering complementary health care approaches if you have a medical condition.  Remember that some health providers may have personal biases as well as financial incentives in keeping you as their patient. Request evidence on which the provider is making their judgements and be sure to discuss the following two kinds of questions: (1) What are the risks, costs and benefit as well as potential for loss or harm? (2) Does the product or service interfere with other treatments? If not, then do what you think is useful. At worst, all you will lose is money.

Source: http://library.med.utah.edu/blog/eccles/files/2011/08/logoHealthLiteracy.png

Finding Health Information on the Internet

The following guidelines have been adapted from an online paper from the National Center for Complementary and Integrative Health (NCCIH) entitled, Finding and Evaluating Online Resources. The text in italic is reproduced directly from the online paper (NCCIH, 2023).

“Your search for published and online health information may start at a known, trusted site, but after following several links, you may find yourself on an unfamiliar site. Can you trust this site? Here are some key questions you need to ask.”

When checking online sources of health information, ask the following questions:

Who operates and pays for the website? Can you trust them? Any reliable health-related website should make it easy for you to learn who is responsible for the site. You should be able to find out who runs a website and its purpose on the “About Us” page. For example, on the NCCIH Website, each major page identifies NCCIH and, because NCCIH is part of the NIH, provides a link to the NIH home page.

Does the site sell advertising? Or Why does the site or app exist? Is it sponsored by a company that sells dietary supplements, markets drugs, provides other product, or services? Confirm any information you find on a site that sells products with an independent site that is not a commercial site.

What is the source of the information? Many health or medical sites post information collected from other websites or sources, and that information should be identified. For example, the Health Topics A-Z page on the NCCIH site provides links to documents that NCCIH did not create—but names the sources of the documents.

How do you know if the information is accurate? Is it based on scientific research? The site should describe the evidence (such as articles in medical journals) on which the material is based. Opinions or advice should be clearly set apart from information that is evidence-based (based on research results). For example, if a site discusses health benefits you can expect from a treatment, look for references to scientific research that clearly support what is said. Keep in mind that testimonials, anecdotes, unsupported claims, and opinions are not the same as objective, evidence-based information. [It is important to remember that this does not mean that it is incorrect; it just may mean the appropriate study was not done as there was no funding for it.]

Is the content a sales pitch masquerading as a news report? Some of these reports are reliable, but others are confusing, conflicting, misleading, or missing important information. For insight on how to evaluate news stories about health, wellness, and complementary therapies, visit our interactive module Know the Science: The Facts About Health News Stories.

Has the information been reviewed by experts? You can be more confident in the quality of medical information on a website if health experts reviewed it. Some websites have an editorial board that reviews content. Others put the names and credentials of reviewers in an Acknowledgments section near the end of the page and declare any conflict of interest. [Yet, even this is challenging as stated in the previous quotes by the Lancet journal editor-in-chief Horton. Thus having sign-off by someone with an advanced degree may not guarantee veracity.]

How current is the information? When was the information written or reviewed? Outdated medical information can be misleading or even dangerous. Responsible health websites review and update much of their content on a regular basis.  Content such as news reports or meeting summaries that describe an event usually is not updated. To find out whether information is outdated, look for a date on the page (it’s often near the bottom). [However, old information does not mean that it is incorrect. Information from the past may be valid and even fundamental and foundational. Sometimes an older medication may be more effective; however, it is no longer recommended because it has outlasted the time period of its patent protection and, the pharmaceutical company has created a slightly new variation which may or may not be more effective.]

What is the website or smartphone app promising or offering? When claims seem too good to be true, the claims probably are not true.

Useful websites for information resources. Start with one of these organized collections of quality resources suggested by the University of Utah (2023):

• Google Scholar ( https://scholar.google.com/ ) provides access to many peer-reviewed resources.
• MedlinePlus, (https://medlineplus.gov/) sponsored by the National Library of Medicine, which is part of the National Institutes of Health (NIH)
• healthfinder.gov, sponsored by the Office of Disease Prevention and Health Promotion in the U.S. Department of Health and Human Services.
• National Center for Complementary and Integrative Health (NCCIH), (https://www.nccih.nih.gov/) the Federal Government’s lead agency for scientific research on complementary and integrative health approaches. Keep in mind that many integrative health and complementary techniques have not been assessed because of a lack of research and funding, however, the procedures can be highly beneficial. The absence of controlled studies does not mean the absence of benefit.
• Follow NCCIH on Facebook, Twitter, Pinterest, and Instagram. These accounts are updated and managed by NCCIH and provide the latest resources on a variety of complementary health approaches.
• For information on dietary supplements, visit the NIH Office of Dietary Supplements website (https://ods.od.nih.gov/factsheets/list-all/). [Remember that many of the dietary values were initially identified as the minimum value to prevent the develop of the vitamin deficiency disease. This value may only prevent an obvious disease. It may not be the appropriate value for optimum health. Most of the data was based on healthy young Caucasian males and the values may not be accurate for women, other age groups, or genetic phenotypes and most likely need to be significantly higher.]

Finding Health Information on Social Media

Credible sources of health information may be found on some social media websites. One suggestion by Kington et al., (2021) is to apply the ‘CRAP’ test developed originally by librarian Molly Beestrum at Northwestern University using four major considerations labeled: “Currency/Credibility, Reliability, Authority, and Purpose/Point of View.” Also, consider the following:

• Check the sponsor’s website. Health information on social networking sites is often very brief. For more information, go to the sponsoring organization’s website. On Twitter, look for a link to the website in the header; on Facebook, look in the About section.
• Verify that social media accounts are what they claim to be. Some social networking sites have a symbol that an account has been verified. For example, Twitter uses a blue badge but people now pay fee for this badge. Is it really verified or only demonstrates that the person paid a fee. Use the link from the organization’s official website to go to its social networking sites.

Finding Health Information on Mobile Health Apps

The National Academy of Medicine (NAM, 2023) builds on the Kington et al. (2021) article about identifying credible sources of health information. Some of the reminders suggested in the NAM website (cf. https://nam.edu/identifying-credible-sources-of-health-information-in-social-media-principles-and-attributes/) are paraphrased below:

There are thousands of mobile apps (a software program you access using your phone or other mobile device) that provide health information you can read on your mobile devices. Almost 20 percent of smartphone owners had at least one health app on their phones in 2012. Keep these things in mind when using a mobile health app:

• The content of most apps is not written or reviewed by medical experts. The information could be inaccurate and unsafe. In addition, the information you enter when using an app may not be secure in terms of protecting personal or private health information (PHI) . [Even if the content is written by medical experts, remember they most likely got paid for it or received university grants from these companies.]
• There is little research on the benefits, risks, and the impact of apps as a source of health information. For example, the ketogenic diet has been found to improve certain medical conditions such as intractable epilepsy. However, for individuals who tend to put weight on easily with a high fat diet, the sudden versus gradual use of a ketogenic diet may be potentially harmful and could shorten lifespan.
• How secure is the technology?It’s not always easy to know what personal information on an app will accessed by third parties or how personal information will be stored or transferred in an unsecure manner.
• Consider the source. Before you download an app, find out if the store you get the app from says who created it. Don’t trust the app if contact or website information for the creator isn’t available. Health apps created by Government agencies can be found by visiting: Centers for Disease Control and Prevention.
• What is the site’s policy about linking to other sites? Some sites don’t link to any other sites, some link to any site that asks or pays for a link, and others link only to sites that meet certain criteria. You may be able to find information on the site about its linking policy. (For example, NCCIH’s linking policy is available on the NCCIH Website Information and Policies page.) Unless the site’s linking policy is strict, don’t assume that the sites that it links to are reliable. You should evaluate the linked sites just as you would any other site that you’re visiting for the first time.
• How does the site collect and handle personal information? Today, most websites track what pages you’re looking at. They may also ask you to “subscribe” or “become a member.” Any credible site collecting this kind of information should tell you exactly what it will and won’t do with your information.
• Will they sell your data? Many commercial sites sell aggregated data about their users’ demographics to other companies (for example, information such as the percentage of their users that are men over 40 or under 25). In some cases, they may collect and reuse information that’s “personally identifiable,” such as your ZIP Code, gender, and birth date. Read any privacy policy or similar language on the site, and don’t sign up for anything you don’t fully understand. You can find NICCIH’s privacy policy on the NCCIH website.
• Is the site encrypted? See if the address (URL) for the site starts with “https://” instead of “http://.” Sites that use HTTPS (Secure Hyper Text Transfer Protocol) are encrypted, less likely to be hacked, and more likely to protect your privacy.
• Can you communicate with the owner of the website? You should always be able to contact the site owner if you run across problems or have questions or feedback. If the site hosts online discussion forums or message boards, the site should explain the terms of use.

Are You Reading News or Advertising?

The Federal Trade Commission (FTC) has warned the public about fake online news sites. The site may look real, but is actually an advertisement. The site may use the logos of legitimate news organizations or similar names and web addresses. To get you to sign up for whatever they’re selling, they may describe an “investigation” into the effectiveness of the product. But everything is fake: there is no reporter, no news organization, and no investigation. Only the links to a sales site are real. Fake news sites have promoted questionable products, including weight loss products, work-at-home opportunities, and debt reduction plans. You should suspect that a news site may be fake if it:

• Endorses a product. Real news organizations generally don’t do this.
• Only quotes people who say good things about the product (includes only positive reader comments, and you can’t add a comment of your own).
• Presents research findings that seem too good to be true. (If something seems too good to be true, it usually is too good to be true.)
• Contains links to a sales site.

Use common sense and incorporate an Evolutionary Perspective in making decisions

To make sense of the flood of information use critical thinking and ask yourself whether the claims make sense in context of human evolution. Over millions of years of evolution, nature has “performed” ongoing experiments through natural selection to improve reproductive fitness. As (Talib, 2014) stated, “It [is] an insult to Mother Nature to override her programmed reactions unless we [have] a good reason to do so, backed by proper empirical testing to show that we humans can do better; the burden of evidence falls on us humans.”

Source: https://www.publicdomainpictures.net/pictures/130000/velka/darwin-evolution.jpg

How can we improve health with some simple procedures or drugs when nature has experimented for millions of years. Adapt the rules to maintain health as described by Talib (2014) in the book, Antifragile: Things That Gain from Disorder (2014), summarized with the following points:

• Anything that was not part of our evolutionary past should be viewed with healthy skepticism. There is a good possibility that it is harmful, because there has not been sufficient time for humanity to adapt genetically to the new variation. For example, the addition of altered trans fats to commercially available foods, which are not recognized by the human immune system and a result, can promote inflammation, cardiovascular disease, and cancer.
• We do not need evidence of harm to claim that a drug or an unnatural procedure involves potential risk.  Take a cautionary approach with a healthy dose of skepticism. If possible then wait until more evidence is discovered. If evidence of harm does not exist, that does not mean harm does not exist.
• Only resort to medical techniques when the health payoff is very large (i.e., to save a life). Does the intervention exceed its potential harm, in cases such as emergency surgery or a lifesaving medicine (e.g., penicillin).

Take charge of your health—talk with your health care providers about any complementary health approaches you use. Together, you can make shared, well-informed decisions.

Key Background Source material for the NCCIH (2023) article, “Finding and Evaluating Online Resources“

• Albrecht, U.V., Von Jan, U., & Praman,  O..(2013). Standard reporting for medical apps. Studies in Health Technology and Informatics. 190, 201-203. https://pubmed.ncbi.nlm.nih.gov/23823422/
• Office of Dietary Supplements. How to Evaluate Health Information on the Internet: Questions and Answers. Office of Dietary Supplements website. Accessed July 10, 2023 https://ods.od.nih.gov/HealthInformation/How_To_Evaluate_Health_Information_on_the_Internet_Questions_and_Answers.aspx
• Plaza, I, Demarzo, M.M., Herrera-Mercadal, P., et al. (2013). Mindfulness-based mobile applications: literature review and analysis of current features. JMIR mHealth and uHealth, 1(2), e24. https://doi.org/10.2196/mhealth.2733
• Subhi, Y., Bube, S.H., Rolskov Bojsen, S., et al. (2015). Expert involvement and adherence to medical evidence in medical mobile phone apps: a systematic review. JMIR mHealth and uHealth, 3(3),e79. https://doi.org/10.2196/mhealth.4169

Recommended sources for the NIH (2023) article, “Finding and Evaluating Online Resources“

• Boulos, M.N., Brewer, A.C., Karimkhani, C., et al. (2014).  Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online Journal of Public Health Informatics, 5(3), 229. https://doi.org/10.5210/ojphi.v5i3.4814. eCollection 2014
• Buying health products and services online. Federal Trade Commission website. Accessed July 10, 2023  https://consumer.ftc.gov/articles/0023-buying-health-products-services-online
• Carissoli, C., Villani, D., & Riva. G. (2015).  Does a meditation protocol supported by a mobile application help people reduce stress? Suggestions from a controlled pragmatic trial. Cyberpsychology, Behavior and Social Networking, 8(1), :46-53. https://doi.org/10.1089/cyber.2014.0062
• Comprar productos y servicios para la salud en internet. Federal Trade Commission website. Accessed July 10, 2023  https://consumidor.ftc.gov/articulos/s0023-comprar-productos-y-servicios-para-la-salud-en-internet
• How to Spot Fake Online Reviews. Consumer Reports website. Accessed July 10, 2023. https://www.consumerreports.org/money/customer-reviews-ratings/how-to-spot-fake-online-reviews-a1345282053/
• Find quality resources. HealthIT.gov website. Accessed July 10, 2023. https://www.healthit.gov/topic/health-it-resources
• Mani, M., Kavanagh, D.J., Hides, L., et al. (2015). Review and evaluation of mindfulness-based iPhone apps. JMIR mHealth and uHealth, I(3), e82. https://doi.org/10.2196/mhealth.4328
• Marcano Belisario, J.S., Huckvale, K., Greenfield, G., et al. (2016). Smartphone and tablet self management apps for asthma. Cochrane Database of Systematic Reviews. (11) :CD010013. https://doi.org/10.1002/14651858.CD010013.pub2
• Martínez-Pérez, B., de la Torre-Díez, I., & López-Coronado, M.  (2015). Privacy and security in mobile health apps: a review and recommendations. Journal of Medical Systems, 39(1), 181. https://doi.org/10.1007/s10916-014-0181-3
• Wallace, L.S. & Dhingra, L.K. (2014). A systematic review of smartphone applications for chronic pain available for download in the United States. Journal of Opioid Management, 10(1), 63-68. https://doi.org/10.5055/jom.2014.0193

References

• AAM (2020). 2020 Report: Generic Drug and Biosimilars Access and Savings in the U.S.,” Association for Accessible Medicines, last accessed March 3, 2023,
  https://accessiblemeds.org/2020-Access-Savings-Report
• Angell, M. (2009). Drug companies & doctors: A story of corruption. The New York Review of Books, 56(1), 8-12. http://www.nybooks.com/articles/2009/01/15/drug-companies-doctorsa-story-of-corruption/
• Arciero, V., Delos Santos, S., Koshy, L., et al. (2021).  Assessment of Food and Drug Administration– and European Medicines Agency–Approved Systemic Oncology Therapies and Clinically Meaningful Improvements in Quality of Life: A Systematic Review. JAMA Netw Open, 4(2):e2033004. https://doi.org/10.1001/jamanetworkopen.2020.33004
• Benbrook, C.M. (2016). Trends in glyphosate herbicide use in the United States and globally. Environ Sci Eur, 28(3). https://doi.org/10.1186/s12302-016-0070-0
• Bishop, T.F., Federman, A.D., Ross, J.S. (2010). Laboratory test ordering at physician offices with and without on-site laboratories. J Gen Intern Med.25(10, 1057-63. https://doi.org/10.1007/s11606-010-1409-7
• Bruening, Z. L. (2019). Profits over People: Flaws in Regulating the Pharmaceutical Industry and Controlled Substances (Doctoral dissertation, Utica College). https://www.proquest.com/openview/ae5178804f2e7a86bd2cf0db62ee2854/1?pq-origsite=gscholar&cbl=18750&diss=y
• Cassidy, J. (2020). How Boeing and the F.A.A. Created the 737 MAX Catastrophe, The New Yorker, September 17, 2020. https://www.newyorker.com/news/our-columnists/how-boeing-and-the-faa-created-the-737-max-catastrophe
• CDC Center for Disease Control and Prevention (May11, 2022). U.S. Overdose Deaths In 2021 Increased Half as Much as in 2020 – But Are Still Up 15%, last accessed 12/12/2022. https://www.cdc.gov/nchs/pressroom/nchs_press_releases/2022/202205.htm
• Chan, A.W. (1986). Racial differences in alcohol sensitivity. Alcohol Alcohol. 21(1), 93-104. https://pubmed.ncbi.nlm.nih.gov/2937417/
• Chandler, P.D., Chen, W.Y., Ajala, O.N., et al. (2020).  Effect of Vitamin D₃ Supplements on Development of Advanced Cancer: A Secondary Analysis of the VITAL Randomized Clinical Trial. JAMA Netw Open, 3(11):e2025850.  https://doi.org/10.1001/jamanetworkopen.2020.25850
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd ed. New York: Routledge. https://doi.org/10.4324/9780203771587
• Crescenzi-Lanna, L., Valente, R., Cataldi, S., & Martire, F. (2023). Predictors of young people’s anti-vaccine attitudes in the context of the COVID-19 pandemic. Public Understanding of Science, 09636625231179830. https://doi.org/10.1177/09636625231179830
• Faria, J. (2023). Pharama DTC ad spend in the U.S. 2012-2021. Statista,  last accessed March 2, 2023. https://www.statista.com/statistics/686906/pharma-ad-spend-usa/#:~:text=The%20pharmaceutical%20industry%20in%20the,invested%20into%20big%20pharma%20ads.
• Hill, J. A., Agewall, S., Baranchuk, A., Booz, G. W., Borer, J. S., Camici, P. G., … & Vrints, C. (2019). Medical misinformation: Vet the message!. Journal of the American Heart Association, 8(3), e011838. https://doi.org/10.1161/JAHA.118.011838
• Horton, R. (2015). Offline: What is medicine’s 5 sigma? The Lancet, 385(9976), 1380. 
  https://doi.org/10.1016/S0140-6736(15)60696-1
• Kington, R. S., Arnesen, S., Chou, W. Y. S., Curry, S. J., Lazer, D., & Villarruel, A. M. (2021). Identifying credible sources of health information in social media: Principles and attributes. NAM perspectives, 2021. https://doi.org/10.31478/202107a
• Koinig, I., & Kohler, S. (2021). On the Relationship Between Skepticism Towards and Reactance to Health Messages: The Special Case of Online Communication on Tick-Borne Encephalitis. Frontiers in Communication, 6, 170. https://doi.org/10.3389/fcomm.2021.737800
• Lupton, Thomas (1580). Siuqila too good, to be true : omen : though so at a vewe yet all I tolde you is true, I vpholde you, now cease to aske why? for I can not lye : herein is shewed by way of dialogue, the wonderful maners of the people of Mauqsun, with other talke not friuolous. Printed at London: By H. Bynneman, dwelling in Thames Streate, neere vnto Baynards Castell, 1580. https://www.amazon.com/Siuqila-too-good-true-vpholde/dp/1171274114
• Maravelias, C., Dona, A., Stefanidou, M., & Spiliopoulou, C. (2005). Adverse effects of anabolic steroids in athletes: A constant threat. Toxicology Letters, 158(3), 167-175. https://doi.org/10.1016/j.toxlet.2005.06.005
• McReynolds, T.  (2020). CDC: An about face on face masks? Last accessed February 22, 2023.  https://www.aaha.org/publications/newstat/articles/2020-03/cdc-an-about-face-on-face-masks/
• Mcvan, M. (2022), In Washington, agricultural policymakers circulate among Farm Bureau, USDA and industry. Missouri Independent, September 6, 2022. https://missouriindependent.com/category/energy-environment/
• Mendes, D., Alves, C., & Batel-Marques, F. (2017). Number needed to treat (NNT) in clinical literature: an appraisal. BMC Med, 15(1), 112. https://doi.org/10.1186/s12916-017-0875-8
• Moore, T.J., Heyward, J., Anderson, G., & Alexander, G.C. (2020). Variation in the estimated costs of pivotal clinical benefit trials supporting the US approval of new therapeutic agents, 2015-2017: a cross-sectional study. BMJ Open. Jun 11;10(6):e038863. https://doi.org/10.1136/bmjopen-2020-038863
• National Academy of Medicine, Identifying Credible Sources of Health Information, last accessed July 5, 2023. https://nam.edu/identifying-credible-sources-of-health-information-in-social-media-principles-and-attributes/
• NCCIH (2023). National Center for Complementary and Integrative Health. Finding and Evaluating Online Resources, Last accessed February 22, 2023. https://www.nccih.nih.gov/health/finding-and-evaluating-online-resources?nav=govd
• Patel, N.G., Hwang , T.J., Woloshin, S., & Kesselheim, A.S. (2023). Therapeutic Value of Drugs Frequently Marketed Using Direct-to-Consumer Television Advertising, 2015 to 2021. JAMA Netw Open, 6(1):e2250991. https://doi.org/10.1001/jamanetworkopen.2022.50991
• Peper, E. & Harvey, R. (2017). The fallacy of the placebo controlled clinical trials: Are positive outcomes the result indirect treatment side effects? NeuroRegulation. 4(3–4,102–113.https://doi.org/10.15540/nr.4.3-4.102
• Perrier, E. T., Armstrong, L. E., Bottin, J. H., Clark, W. F., Dolci, A., Guelinckx, I., … & Péronnet, F. (2021). Hydration for health hypothesis: a narrative review of supporting evidence. European journal of nutrition, 60(3), 1167-1180. https://doi.org/10.1007/s00394-020-02296-z
• Puigbò, P., Leino, L. I., Rainio, M. J., Saikkonen, K., Saloniemi, I., & Helander, M. (2022). Does Glyphosate Affect the Human Microbiota?. Life, 12(5), 707. https://doi.org/10.3390/life12050707
• Putois, O., & Helms, J. (2022). The Role of Denial in Vaccine Skeptics and “Anti-vax” Blame: A Psychodynamic Approach. Frontiers in Psychology, 13, 886368. https://doi.org/10.3389/fpsyg.2022.886368
• Rossignol, M., Labrecque, M., Cauchon, M., Breton, M., & Poirier, P. (2018).  Number of patients needed to prescribe statins in primary cardiovascular prevention: mirage and reality, Family Practice, 35(4), 376-382. https://doi.org/10.1093/fampra/cmx124
• Shaw, S. (2023 February 18). The case for free-range lab mice. The New Yorker. Last accessed  March 1, 2023. https://www.newyorker.com/culture/annals-of-inquiry/the-case-for-free-range-lab-mice
• Talib, N. N. (2014). Antifragile: Things That Gain from Disorder (Incerto). New York: Random House. https://www.amazon.com/Antifragile-Things-That-Disorder-Incerto/dp/0812979680#:~:text=Antifragile%20is%20a%20standalone%20book,world%20we%20don’t%20understand.
• Tano, B. The Layman’s Guide to Integrative Immunity. Integrative Medical Press; 2016. https://www.amazon.com/Laymans-Guide-Integrative-Immunity-Discover/dp/0983419299
• Tantibanchachai, C, &  Yang, J. (2019). Studies of thalidomide’s effects on rodent embryos from 1962-2008. The Embryo Project Encyclopedia. https://embryo.asu.edu/pages/studies-thalidomides-effects-rodent-embryos-1962-2008
• Westreich, D. (2012). Berkson’s bias, selection bias, and missing data. Epidemiology (Cambridge, Mass.), 23(1), 159. https://doi.org/10.1097/EDE.0b013e31823b6296
• Wilhelmi, O. & Hayden, M. (2016). Reducing vulnerability to extreme heat through interdisciplinary research and stakeholder engagement. In Extreme Weather, Health, and Communities (pp. 165-186). Springer, Cham. https://convergence.unc.edu/wp-content/uploads/sites/381/2017/03/wilhelmi-hayden_2016_reducing-vuln-to-extreme-heat-interdiscip-research-stakeholder-engagement.pdf
• Wouters, O. J., McKee, M., & Luyten, J. (2020). Estimated Research and Development Investment Needed to Bring a New Medicine to Market, 2009-2018. JAMA, 323(9), 844-853. https://doi.org/10.1001/jama.2020.1166
• Yucha, C. B. (2002). Problems inherent in assessing biofeedback efficacy studies. Applied psychophysiology and biofeedback, 27(1), 99-106. https://doi.org/10.1023/a:1014528622061
• Yucha, C., & Montgomery, D. (2008). Evidence-based practice in biofeedback and neurofeedback. Wheat Ridge, CO: AAPB. https://www.advantagehcs.com/wp-content/uploads/2018/11/EvidenceBasedYuchaMontgomeryW.pdf

[i]This is a measure of size of the  association as measured by as  statistic  such as  Cohen’ d; namely, if it is small–although statistically significant– it probably would not be clinically meaningful. Cohen (1988) suggested a ”d” statistic (e.g. Cohen’s d) comparing the group differences (e.g. treatment group vs. comparison group change scores; [M2 – M1]) divided by the standard deviation of both groups [square root of SD1+SD2]/2], interpreting moderate effects between 0.50 and 0.79 and larger effects above 0.80. Treatment group vs comparison group effects are also estimated by examining percentages.  Relative risk ratio or odds ratio is a single number that reflects the increased or decreased risk. For example, a doubled risk would be expressed as a relative risk of 2. Risk decreased by 50% would be expressed as RR 0.5. This number is calculated as the percent of people with clinically meaningful outcomes divided by percent of people without clinically meaningful outcomes. This provides a ‘relative’ estimate of effectiveness, where a ratio close to 1 indicates no difference between treatment and comparison groups, and ratios greater than 3 to 1 (e.g. treatment group was twice as effective as comparison group) are considered moderate effects and 4 to 1 are considered larger effects.

About the Authors

Erik Peper’s teaching and research focuses on self-healing strategies, illness prevention, the effects of posture and respiration, and how to use biofeedback and wearable devices. Each year he mentors undergraduate student researchers to create and complete studies that are presented at scientific meetings. He is an international authority on biofeedback and self-regulation and author of scientific articles and books such as Make Health Happen, Fighting Cancer-A Nontoxic Approach to Treatment, and Biofeedback Mastery. His most recent co-authored book is, TechStress: How Technology is Hijacking Our Lives, Strategies for Coping, and Pragmatic Ergonomics.  He publishes the blog, The Peper Perspective–ideas on illness, health and well-being (peperperspective.com). In 2013 was received the Biofeedback Distinguished Scientist Award in recognition of outstanding career & scientific contributions from the Association for Applied Psychophysiology. 

Richard Harvey has a Ph.D. for the UC Irvine Social Ecology program. His research includes developing stress-reduction interventions which promote psychological courage and hardiness. Before teaching at SF State, he was a research fellow at the UC Irvine Transdisciplinary Tobacco Use Research Center for five years, developed and ran the UC Irvine Counseling Center Biofeedback and Stress Management Program, and worked as a Maternal, Child and Adolescent Health Research Analyst in Orange County. He is the co-chair of the American Public Health Association, Alternative and Complementary Health Practices Special Interest Group, as well as a board member of the Biofeedback Society of California and the San Francisco Psychological Association. He has published in the areas of biofeedback, stress and computer-related disorders, tobacco cessation, and the psychology of hardiness and courage. 

Copyright © 2023 Townsend Letter