I train my students to analyze scientific manuscripts by assessing statistical, internal, and external validity. In the previous newsletter, I addressed one type of statistical validity failure. In this newsletter, I want to address external validity. External validity is the extent to which scientific research applies in the real world, outside the simulated world of the lab.
Many will be surprised (and disappointed) to learn that a great many scientists are completely uninterested in applying their research outside the lab. These intellectuals are like the character Sheldon from, The Big Bang Theory, who relies completely on intellect and has only tangential contact with reality. These intellectual “Sheldons” will unironically cherish that moniker as a badge of honor. However, this detachment from reality - which led Sheldon to believe he could devise a useful formula for romance - seems maladaptive to most people.
It’s true, intellectuals have prestigious titles, advanced degrees, and fine careers that many would deem as success. However, because most of their life is lived in their head, they often are viewed as having little “common sense”. This may be fine for researchers in physics, chemistry, geology, and biology, but it is a major handicap for social scientists. I had one professor in graduate school that studied child behavior. They commented that people assumed they must love children. Their reply was that they loved children in the same way an oncologist loved cancer. Young humans were a problem to be studied, largely because they didn’t understand the phenomena. To top it off, academic intellectuals also dress in really weird robes. And hats, don’t forget the even weirder hats.
In the social sciences, it is ironic and unfortunate that some of the people least familiar with human interaction are the people producing the science of human behavior. It is difficult to design research meant to apply in the real world when the researcher lives mostly in their head. Many wrongly assume that if research has good internal validity (follows the best practices of scientific investigation), and is replicable, the findings are true and applicable in the real world context. On its face, that is problematic. If all that matters is procedure, then any child, if capable of running the procedure, could be an accomplished scientist. In my experience, this failure to realize the importance of external validity, is one of the most subtle, overlooked, and significant problems in science.
The analogy I use with my students involves baking. If I told you I baked a chocolate cake, gave you the recipe, and you replicated it, that is internal validity. When mass producing a product this is a legitimate concern. However, that process in no way assures the cake is edible, esthetically pleasing, or marketable as a consumer good, let alone really a chocolate cake and not an angel food cake. Likewise, internal validity in no way confers external validity on research.
There is a golf analogy that can be used to illustrate the relationship between theory and empiricism, internal validity and external validity. Do you measure the quality of a golfer based on the characteristics of their grip, stance, club rotation, backswing, impact, and follow through, or where the ball lands? In practice, the real world, what ultimately matters is where the ball lands, on the green, in the hole, or in a trap. Because of the sheer number of variables contributing to any phenomena and their dynamic, theory and internal validity do not guarantee external validity.
There are several subtypes of validity that contribute to external validity. Construct, ecological, temporal, and population are the ones I emphasize and will explain here. Because there are no “best practices” or objective rules to follow to achieve external validity, in my opinion it is the most often violated, yet least contested aspect of science. External validity is truly the “stupid is, as stupid does” test of scientific research.
Construct validity is how well the variables manipulated and measured match the real world. Ecological validity is an assessment of how well the entire context of the simulation imitates the real world. Most often, the manipulations and measures in laboratory simulations are only proxies of reality. Temporal validity is a judgment of whether the results obtained apply in the past or future compared to only a limited time frame. Population validity is violated if the sample being studied does not accurately represent the population to which the results are inferred. Failure to stratify samples by relevant characteristics can egregiously distort calculations of probability, as I mentioned in the previous newsletter.
While there is much subjectivity in this area, science really can do better. For example, did you know that psychologists believe children under age 4 do not have what they call a “theory of mind”? Meaning there are adults that think toddlers don’t know other humans have minds. Without getting into the details, that is nuts. If you’ve ever spent more than cursory time with children, or pets for that matter, your intuition tells you that cannot be accurate. Survival for any species would be very difficult without a better than random probability of inferring another organism’s intentions and predicting their behavior. In my opinion, this failure is due to a mix of poor construct and ecological validity. Suffice it to say, if you think attempting to have a dialogue with 3 and 4 year old’s is a solid scientific method, you might be a Sheldon.
Sometimes failures of external validity are esoteric and inconsequential to the daily lives of anyone outside intellectual circles. However, the COVID pandemic has highlighted just how dangerous the failures of intellectuals can be. Policies developed during the current COVID pandemic have been the result of horrendous violations of external validity. There are several examples where research failed in population, construct, and ecological validity. As I said, these aspects are a bit more subjective than internal or statistical validity, but I will attempt to provide concrete examples of this in future newsletters.
Share this post