Replication used to mean testing the robustness of a theory. Scientifically, replication is actually just a failed attempt to falsify a hypothesis, that results in the retention of the current hypothesis. Philosophically, science is attempting to falsify rather than replicate or verify phenomena. It has been argued that falsification is more informative and certain than verification. There are some holes in that argument but the classic example is that if we have a hypothesis that all swans are white, finding more white swans (replication/verification) adds little explanatory power to the hypothesis. However, if we find even one, non-white swan (falsification) we now have non redundant information and can reject a hypothesis and perhaps an entire theory. We could be in error in that observation (our perception is off, the swan is covered in oil), and while that is a serious philosophical concern, it is often ignored.
In today’s scientific circles, replication means reproducing the same result using the same methods. Maybe because of our manufacturing boom, we became confused and thought producing science was the same as other consumer products like cell phones and chicken nuggets. Science may aid in reproduction of goods but its higher calling is to test hypotheses and theories we have yet to understand completely.
However, fraudulent research has caused many scientists to panic and they have promoted mass replication of method/procedure as the solution. This is ridiculous. First, deliberate fraud is very rare. Second, to assume you can eliminate fraud is naïve. Third, the tactics used to avoid fraud are insufficient and problematic. They only test the reproducibility of a particular method and are completely inadequate to test hypotheses or phenomena at a theoretical level. There have been a few good articles demonstrating these points. You probably can’t access these articles for several reasons I will discuss in another newsletter, but I will give you the links anyway.
DOI: 10.1177/1745691614528518
DOI: 10.1177/1745691613514450
My complaint is a bit different. While I agree with nearly all the criticisms that have been leveled against the replication effort, I haven’t heard anyone mention the need to be more careful in designing research to have better external validity. Reproduction is all about what scientists call internal validity. I’ve discussed this issue in other newsletters as well.
Here is an example of how I think the “leaders” of the replication effort are misguided. I ran into one of the chief proponents of the replication effort at a conference. In the lecture, they argued scientists will be viewed as truthful, able, and ethical depending on how well they respond to efforts to replicate their research. Immediately you should recoil at the idea that they view this process as the real arbiter of scientific truth. Even if the process was perfect - and it is not - that would be a faulty assumption.
Anyway, here is the rationale. If other scientists (about 30 labs) are able to replicate your method and find similar results, they believe it is confirmed that you are truthful, able and ethical. You have the stamp of approval! This chief instigator of replication took it as obvious that no one would have a problem with that initial statement. To his surprise, I had a very big problem. I pointed out that I would be horrified to think my research, if replicated, would now have some nearly unassailable status as confirmed because it was replicated by multiple labs. In fact, much of my research is devoted to debunking long held and oft replicated research. Again, being able to faithfully replicate has little to do with external validity. Mass replicating chicken nuggets in no way elevates their status as quality food.
Apparently, few seem to have a problem with that first possible outcome. Maybe few also see problems with the other possible scenarios, but to me they are all anti-science and incredibly scary. If your research cannot be replicated (there are several legitimate reasons this may not happen), your status as a researcher is now wholly dependent on your response. If you fall on your knees and beg forgiveness and admit your incompetence, then you will at least be viewed as ethical, although somewhat less able. If you dare challenge the replication, you are toast. You will be viewed as unethical, incompetent, and maybe a fraud. Majority wins. Isn’t that how science is supposed to work? It is not unlike Inquisition tactics. Once replication has failed (you are accused), whether you confess or deny, is of little consequence. Below is a simplified matrix of the one presented at the conference.
I am not going to spend the time here to delineate the many scientific and ethical violations with this approach. What is so worrisome is how entire fields of science have swallowed this flawed approach whole and their journals are supporting the effort. It seems we are doing a poor job training researchers in the foundational logic of the scientific method.
Scientific journals could do better in other ways too. The peer review process is hopelessly biased and flawed in other ways. An internet search will reveal I am far from alone in that judgment. Journals could expect more transparency from authors but they don’t know how to go about it.
As a scientist, I want my research to be understood by anyone, not just other scientists. Yet, from my scant publishing experience, I’ve found even other scientists are often ill equipped or unmotivated to adequately assess research validity. In wanting the public to have sufficient access to research, I am in the minority. Few would believe the public cares about science, let alone have the ability to assess validity. I think they are wrong.
I can only hope the misguided replication efforts will die a horrible death very soon. I’m not optimistic about that. In a future newsletter I will explain my proposed solutions to some of these problems. It involves changes in publishing and education that are not that difficult or radical. However, any deviation from the status quo is vigorously resisted. The Plato’s and Sheldon’s can’t imagine Gump’s having anything meaningful to contribute.
Share this post