Intellectual Failures: Falsifying the Null Hypothesis

I’ve written a little about logic before (Intellectual Failures - Logic Fallacies). This post will try to explain the scientific (and statistical) logic of testing what is called the Null Hypothesis. In statistics it is referred to as Null Hypothesis Significance Testing (NHST). This technique should also be applied in EVERY debate but is rarely used.

There are two premises. First, science should be agnostic and unbiased. Given that, it makes sense to start from the premise that whatever hypotheses we are testing have equal validity. That is the null—no difference—hypothesis. Second, falsification rather than verification is the more informative and certain approach (e.g. If a hypothesis is that all swans are white, finding one black swan is more informative than finding a million white swans). Statistical tests of significance are designed, not to prove a theory, but to estimate the likelihood that a researcher has erred in falsifying or retaining the null hypothesis. This can be a bit confusing and apparently few grasp its logic.

The SARS-CoV-2 pandemic has been a glaring example of how most fail to follow this scientific approach. For example, the first null hypothesis that needed addressing was whether SARS-CoV-2 was different from any other virus on any of several important outcome variables. Initially the data was very unclear on several metrics, including its mortality rate, but many had already prematurely rejected the null hypothesis. To this day, there is serious disagreement on the nature of the pandemic due to measurement and treatment vagaries. The obligation to reject the null is not on any questioning public policy or health officials, it is on those arguing special measures need to be taken.

This is a key factor, often overlooked in any argument. Whoever is claiming something other than the null is true, is the one that must supply the evidence, not vice versa. They have the burden of proof. The U.S. judicial system claims to operate under this same principle. All are assumed equal, and not guilty (null hypothesis). The burden of proof lies on those claiming otherwise. For an example of pandemic response reasoning, those claiming masking is better than not masking are the ones obligated to disprove the null. They must supply the RCT trials and studies that have ruled out all rival hypotheses, not those refusing to engage in extraordinary behavior without evidence.

Even if you reject the initial null hypothesis, there are infinite more null hypotheses that need to be tested. What treatments are effective, and with who, and what techniques can mitigate transmission? Early on it was decided that universal masking, distancing, cleaning, and quarantining infected would be necessary to mitigate transmission. Treatments were scattershot but quickly resolved to only vaccination.

Science can be painfully slow. In emergencies scientific rigor may be impossible. If you are flying at 500 mph and 30,000 feet when a catastrophic malfunction occurs, there may not be time to systematically reject null hypotheses in a lab. You will have to rely on the expertise and authority of the pilot and crew to shortcut the process. However, this doesn’t mean ethics can be disregarded.

The pandemic has been such an emergency situation. Unfortunately it appears our “pilots” have not been so adept. They have too quickly rejected early treatment protocols, while simultaneously violating individual autonomy with mandates. This is an interesting case regarding the practical application of the null hypothesis. In this case, early treatment protocols may not have sufficient evidence to scientifically reject the null (that x treatment is better than no treatment/placebo), but when dealing with life or death matters a risk assessment is fitting. Do the possible benefits outweigh possible harms? Now on the opposite side, there is equally shaky evidence that mask and vaccine mandates are better than alternatives. Here it is tempting to say the benefits outweigh the risks too. However, the ethics of coercing others, seizing the sovereignty others have over their own bodies, must be considered too. I have argued in another post that no one has the authority to wrest bodily sovereignty from another unless that person is violating another’s autonomy (Intellectual Failures - Ethics).

I use an analogy with my students that many can relate to. You can’t start your car in the morning. What is your hypothesis? Dead battery? You test the battery voltage and it is insufficient. What do you do? Jump the battery, recharge the battery, or buy a new one? One of these strategies works and you drive away. Next morning your car doesn’t start. What is your hypothesis? Those with a more sophisticated knowledge of cars (and a keen sense of the value of money and time) might have several different hypotheses other than the battery alone (e.g. alternator has failed or current drain). Each of those should be tested before you go out and buy a new battery.

This illustrates two points. One, falsifying hypotheses, while tedious, is a more comprehensive way to solve a problem. Two, the depth and breadth of the researcher’s comprehension of the problem is essential to even begin proposing the full complement of hypotheses, let alone knowing how to test them. As I have contended before, this is where intellectuals flounder. They often simply do not have sufficient real world experience to know all the relevant variables to test.

In previous posts I’ve argued that during the SARS-CoV-2 pandemic, nearly everyone in the public sphere has failed to identify, test, and falsify all rival hypotheses. This results in erroneous probability calculations when population validity is violated and ignored treatments when construct validity is violated. To bring back the battery example, it’s like the intellectuals tested the battery and it was low. This was taken as evidence to falsify the null hypothesis. So, they went out and bought a battery. Problem solved, until the next day when the battery is dead again. Check the voltage, buy another battery, and so on every day. Early treatment protocols were not only abandoned early on, they have been forbidden. The CDC, FDA, WHO have no early treatment protocols. Neither does any U.S. university medical research institute. This is unprecedented. Population age stratification is ignored when analyzing the effects of SARS-CoV-2 and vaccination recommendations, which results in teens in good health receiving the same recommendations as an obese octogenarian with cardio-pulmonary disease.  Stupid is, as stupid does.