The real gold standard of science is not “peer review”; it’s something called “reproducibility.” —Curt Devlin (Fairhaven, MA), July 1, 2013
Modern science has become a rarified business these days. It is conducted far outside the bounds of the average person’s experience. As a result, it’s easy to mislead people about how science actually works and what counts as good science or bad science. Most people know that evolution is considered to be good science and creationism is considered to be bad science, but they still would be largely at a loss to explain why.
When told that the “gold standard” of science is peer review, most people tend to accept this as gospel. If science has become so sophisticated that only experts in the field can understand it, then surely it makes sense to have any scientific conclusions evaluated by other experts in that field. Right?
Unfortunately, the idea that peer review is the gold standard of science is absolutely false.
To the extent that peer review is based on authority or expert opinion, it is completely contrary to the true spirit of science. Peer review is not a bad practice, but its true purpose is to improve the work and decide if it is worthy of being published. You could say that peer review is the gold standard of publication – nothing more and nothing less.
At its heart, the real core of science is a handful of simple ideas called the scientific method. It involves careful observation, precise measurement, and accurate reporting of both the conditions under which measurements were made, and the measured results themselves. The whole point of the scientific method is to eliminate human authority, opinion, or bias of any kind from consideration.
The real gold standard of science is something called reproducibility. Simply put, this means that if you do the same experiment under the same conditions and same measurement precision, you get the same results.
You could say the mantra of science is “see for yourself.”
If we apply the standard of reproducibility to the findings reported by Dr. Nina Pierpont in her book, “Wind Turbine Syndrome” (WTS), her conclusions hold up remarkably well because her work is based on very careful observation and measurement. Her reporting of experimental conditions and measurements is extremely detailed and meticulous – even to the point of publishing all the raw data in her book. (No one who is interested in selling a book puts raw data in it.) Presumably, Pierpont did this to ensure that serious defects would be obvious. This type of transparency and disclosure is a signature of scientific integrity.
Predictability is also a very important element of good science. The findings of a study should support specific predictions about outcomes under certain conditions. As a resident of Fairhaven, Massachusetts, where two 1.5 MW industrial wind turbines were sited in a dense neighborhood two years ago, WTS has proven to be an excellent predictor of the adverse health effects that have occurred since then. I have absolutely no doubt that the results of Pierpont’s study could be reproduced in Fairhaven tomorrow.
Pierpont’s critics within the wind industry could easily fund an independent study to determine whether her experiments can be reproduced, but they never have and never will. Perhaps they already know too well that these attempts will only result in confirming her findings. Pierpont’s findings are simple enough. When people live near wind turbines, they experience nausea, dizziness, sleeplessness and stress-induced illnesses. When they get away from them, they begin to feel much better and may recover completely.
Just the other day, I heard a science editor on public radio, Heather Goldstone, leveling criticism that WTS is not peer reviewed and parroting the claim that peer review is the gold standard of science. Anyone who has read WTS (as I have) knows that Goldstone is factually incorrect. Pierpont sought review, advice, and criticism from her peers throughout her research and publication. This group included highly regarded clinicians, acoustic experts, researchers in neurology and public health, among others. The referee reports (aka peer reviews) are all included in the book, for those who trouble themselves to actually read it.
Goldstone also claimed that Pierpont’s study was flawed because she studied subjects only in one small location. This is also factually incorrect, proving only that our “science” editor, Goldstone, had never bothered to actually read WTS herself. (Maybe she got this information about the book from “good authority,” perhaps a friend in the wind industry?)
As Pierpont explains in the book, she went to great lengths to identify subjects from other geographies and countries to avoid this limitation in her case study. That is why she had to restrict the participants to those who spoke English, to ensure that she could clearly understand their reports.
By contrast, the process of peer review does not stand up so well under close scrutiny. Several studies have shown that the peer review process can be fraught with petty professional jealousy, personal grudges, and other conflicts of interest created by ambition, academic competition, and so forth. Some studies have shown that this problem is even worse in blind and double-blind peer reviews, because reviewers can hide behind anonymity and offer reviews that they would not stake their professional reputation on.
So much for peer review as the ultimate standard of scholarly publication.
Advocates of wind energy would have you believe that anything that is not peer-reviewed should be discredited and disregarded. Let’s see how this idea holds up.
In 1904, if you had argued that apples fall from the tree to the ground because the immense mass of the Earth causes space to warp, you would have been treated to some strange looks. If you had claimed that time slows down as things speed up, or that matter and energy were really two forms of the same thing, you probably would have been diagnosed with dementia praecox (that’s what Alzheimer’s was called in those days). Truly, such ideas simply defy common sense. (Note: Good science often does.)
You would have been subjected to raucous laughter if you had mentioned that you learned all this from a third-rate clerk at a Swiss patent office in Bern.
And yet, as improbable as all this sounds, this is more or less what Albert Einstein did tell us in an article he published in a German periodical called Annals of Physics in 1905. It established one of the very pillars of modern physics for the next century.
Amazingly, Einstein’s work was not peer-reviewed at all. It was read by Max Planck, the preeminent physicist of the day, who gave it a wink and a nod. Then it was published. Since then, Einstein’s theories have been experimented with, scrutinized, and tested as much as any in history. Science must accept or reject it based on evidence alone, not a “peer reviewer’s” authority or opinion.
Einstein’s ideas – most, at least – have been confirmed over and over again.
Based on the “gold standard” of peer review, however, we are presumably expected to discard the theory of relativity until it has been properly peer reviewed.
In 1953, two Harvard biologists, James Watson and Francis Crick, published a paper in the journal Nature claiming that the chemical structure of DNA, the code for all life on Earth (and probably the universe), is a double helix – like a spiral staircase, which in fact gave them the idea.
Again, they did so without a single peer review. It would seem that we must disregard the foundations of modern biology and genetics, too. The “gold standard” of peer review demands it, correct?
When given fairly and honestly, peer review can be a powerful ally of science. Often, peer review can provide an invaluable exchange of ideas between researchers. Sometimes it can be the beginning of fruitful collaboration between scientists, each of whom is holding a different piece of the same puzzle. But the idea that peer review is the final arbiter of science is absurd. If there is such a standard, it is, and must be, reproducibility. Replicability.
Let’s face it. The chant of peer review coming from religious devotees of wind is becoming nothing more than lip service by those who have been turned into intellectual zombies by the incessant propaganda of a wind industry that places profits above health, politics before science, and opinion over genuine knowledge.
In the case of WTS, this chant has been used as a weapon of mass delusion, a device to dismiss a superb piece of science and a pioneering contribution to our knowledge about the impact of wind turbines on human health and well-being. This has been done because legitimate criticism and ground-level research only serve to strengthen the conclusions arrived at in this book.
If you are interested in some of the most cogent and legitimate criticisms of WTS that I have read, consider these:
» The study was done by interview and limited to available medical records.
» Participant memory limitations or distortions.
» Possible minimization or exaggeration effects.
» The study was limited to English-speaking subjects.
» Small case series sample.
» Limited duration of follow-up.
The details of these specific criticisms and limitations of the Pierpont report can be found on pages 124-125 of WTS. Pierpont herself wrote them to alert her peers and fellow clinicians, and to identify the limitations of her own work; undoubtedly realizing that the study should be done on a much larger scale to address them. This was a task she did not have sufficient resources to do herself.
Calling attention to the defects or limitations of your own study does not invalidate it. On the contrary, it is one of the hallmarks of good science and an invitation to further study by other scientists who may be in a position to eliminate those limitations and either confirm or reject its conclusions on the basis of the evidence alone.
At the end of the day, you must ask yourself why a study of such profound social importance has not been repeated on a large scale. Could it be that those with the most to lose are afraid of what they will find?
“Wind Turbine Syndrome” is good science. The devotees of wind turbines, who would challenge the results reported in its pages, must do so on the basis of more good science – or not at all. Either they must exercise the Principle of Reproducibility or accept Ludwig Wittgenstein’s famous caution, ”Whereof one cannot speak, thereof one must be silent.”
Editor’s note: Notice “Google’s” doodle image for July 25: celebrating Rosalind Franklin on what would have been her 93rd birthday. Her x-ray crystallography of DNA was key to determining its double-helix structure.
Dr. Heather Goldstone, Science Editor
WCAI, National Public Radio
Dear Dr. Goldstone,
I am writing to provide you with a copy of a recent essay by Mr. Curt Devlin regarding a dangerous fallacy that has been repeatedly promoted by you, and others, regarding the true nature of the scientific method.
This essay, which refers to you by name, explains in plain English why the obsession with “peer reviewed studies” is nothing more than a strategy employed by the wind energy industry – and its sympathizers – to create an artificial, and illegitimate, standard of “evidence of harm” from wind turbines whose practical effect – and obvious intent – is to dismiss an overwhelming torrent of evidence from all over the world that demonstrates, beyond a shadow of a doubt, that these adverse impacts are occurring on a global scale.
I, too, have long been troubled by your dismissive public remarks concerning the research of Dr. Nina Pierpont’s book, Wind Turbine Syndrome.
Specifically, I have been amazed that someone with your training should continue to insist that this work is: a) “not peer reviewed”; and b) that the gold standard of science is peer review.
In the first instance, it seems that you have not even bothered to read Dr. Pierpont’s book. If you had, you would have discovered, to your shock and amazement, that the work was actually read in advance of publication by a distinguished group of medical doctors, including clinicians, epidemiologists and other medical researchers, all of whom found merit in the work. And if you had bothered to dig a little deeper, you would find that many other distinguished scientists who read the book after its publication and considered the quality of the research had commented favorably on its merit.
Such minimal research would not seem like too much to ask of someone who regularly reports on science for the National Public Radio.
In the second instance, it is a source of constant dismay that you fail to understand the simple point that Mr. Devlin makes in his essay below – namely that peer review isn’t really worth squat, other than acting as a screen to publication in various journals.
As you know, scientists offer competing hypotheses all the time to explain events around us, including medical phenomena that make their presence known through the assertion of symptoms without immediately revealing all of the intricate details of the mechanism, or mechanisms, that produce the symptoms. In other words, the effects are always appreciated first; only later – typically after a considerable amount of hard work and ingenuity are brought to bear – are the causes of those effects completely comprehended.
Surely, you must be able to appreciate – as the Science Editor at NPR that his is how we came to achieve a detailed understanding of every disease or medical illness on earth, including malaria, polio, mercury poisoning or mesothelioma (from exposure to asbestos).
The gold standard of science is “TRUTH” – not the “truthiness” of “peer review” which is subject to all of the distorting, and falsifying, influences enumerated by Mr. Devlin – all of are familiar, and well-understood, to every practicing scientist.
As Mr. Devlin so eloquently explains, the way that we verify the “truth” of any scientific hypothesis is through “reproducibility” – not through “peer review.”
As Mr. Devlin explains, “peer review” is just an opinion; but “reproducibility is where the rubber meets the road.
If scientists of all persuasions are able to reproduce the findings of a study, that adds credibility to the accuracy of the hypothesis; conversely, if other scientists – such as the paid shills of the wind energy industry and sympathetic science editors – are unable to contradict the findings of the original study, this also confers credibility on the conclusions of the study.
I regret to inform you, Dr. Goldstone, that “peer review” is only the “Goldstone Standard” – and not the “gold standard” – of science. Nothing more; nothing less. And that the “Goldstone Standard” of science is a rather pathetic and misguided one, at that.
Please also consider – in this context of our mutual quest for the truth about the adverse health impacts from industrial wind turbines – that you, and your like-minded cohorts in the wind industry – have regularly sneered at first person testimonials from the victims of these installations, haughtily dismissing them as “anecdotal” and “unscientific” – and therefore largely irrelevant, according to you.
You habitually, and sanctimoniously, dismiss all of this information, notwithstanding the fact that these testimonials and the pattern of adverse health impacts that are reported are eerily similar – virtually identical – regardless of where they come from.
All of the complainants share the following circumstances, regardless of what country, or what continent, they inhabit:
— They live in relatively close proximity to newly installed wind turbines;
— They were fine before the wind turbines arrived;
— They are profoundly disturbed – sometimes physically ill – while the wind turbines are operating;
— Their symptoms disappear – or are greatly alleviated – when the wind turbines stop operating or when they remove themselves to a safe distance.
Even though there are literally thousands of these reports from residents located on six continents describing the same pattern of symptoms – and the same cause and effect – this is not “scientific” enough for you.
You want them to tell their symptoms to a scientist who will diligently record all of this information as “data.” God knows where you think that money for this exercise will come from, with no profit potential for anyone. And if the victims hired their own researcher, you would doubtless declare the results to be “biased” – and dismiss them as “invalid.”
Then you want some other group of scientist to review the “data” – the description of the symptoms that have been described by the victims and diligently recorded by the researcher – and to bless it as worthy of publication.
Only then will the “data” be considered to satisfy the “Goldstone Standard” of excellence – and be credible to anyone, in your view.
Do you not understand the absurdity of dismissing thousands and thousands of first person accounts in this way, based on the idea that it is “not credible” unless it is “peer reviewed” by the editors of some medical journal?
Let me try to explain this to you a different way.
I am not a trained scientist – at least no beyond a typical high school education in the subject. But I am a practicing Emergency Medical Technician (EMT).
As you can appreciate, EMT’s are pretty low on the food chain in the rarified realm of “science” that you inhabit. We respond to medical emergencies, we assess the condition of patients and provide emergency medical care to stabilize and improve their condition and we drive them to the hospital in an ambulance for further treatment, if necessary. We’re not PhD’s and we’re not brain surgeons, but we’re trained to provide basic medical assistance that sometimes proves to be vitally important.
Guess what we do when we first encounter a patient in distress: we ask them to tell us their symptoms. Why did they call 911? What’s wrong? What’s different? What are they feeling?
And do you know why we do this, Dr. Goldstone?
Because the patient is the only source of this information!
I can observe the scene of an accident. I can tell if there has been a significant “mechanism of injury” if a car has hit a tree at high speed. I can tell if someone obviously has a broken arm. But I can’t tell if his abdomen hurts; of if he is dizzy; or if he has chest pain or tingling in his legs.
Do we summon a doctor with a clipboard to interview the patient and tabulate the results in a fancy spreadsheet – and wait for him to compile similar data from 50 other car crashes – before addressing the immediate problem?
Do we dismiss the information that we receive from the patient until some other team of scientists has had a chance to subject the first scientist’s findings to a “peer review” – the “Goldstone Standard” – to confirm our suspicion that the patient might be in serious jeopardy if we do nothing?
Do we tell the patient that we’d like to believe him – that we certainly sympathize with his predicament – and that we think that it is very possible that he might be telling the truth – but that, unfortunately, we can’t really do anything until the scientists all get together and publish their findings?
Or do we tell him – as in the Town of Falmouth – that we’d like to take him to the hospital – God knows he’s in bad shape – but that the Board of Selectmen has decided to limit ambulance trips to one a day – as part of their “balancing of interests” and to save the taxpayers’ money – and, unfortunately, we already made a trip to the hospital earlier in the day – so he’ll just have to suck it up?
No. We don’t do any of these things.
a) There is only one expert on “symptoms” – and that is the patient. We cannot see, or hear, his symptoms. He feels them and he reports them to us.
b) We have no reason to doubt the patient – especially when we know that similar patients in similar circumstances are likely to report the same symptoms. Why should I be shocked if a victim in a car crash reports neck pain? Or if a diabetic is dizzy and disoriented? I know from experience that one thing leads to another – without a “peer review.”
c) The patient’s “first person testimonial” seems credible – particularly if the patient has no reason to lie. Am I to suspect that victims of car crashes are part of some vast global conspiracy because they report similar symptoms after the same “mechanism of injury” in the absence of a “peer review”? Really? Why do they do it? How did they manage to communicate with each other and make sure that they got their stories straight – time after time after time, a thousand times, in a thousand places?
The reason we don’t apply the “Goldstone Standard” and insist upon a “peer review” of this information is simple: if we did, we would have a lot of dead and dying patients.
People would be in agony; walking around with broken arms; in a diabetic coma; suffering anaphylactic shock from an acute allergic reaction; experiencing the initial stages, or the warning signs, of an imminent heart attack – while we all awaited the “peer review” of our initial assessment.
It is the same with “wind turbine syndrome” – a simple term which you, and others, love to villify. This is a straightforward reference to “a concurrence of symptoms” that tend to occur in people who live too close to wind turbines.
How do we know this? Because the victims report this same “concurrence of symptoms” – or some variation thereof – over and over and over again in the presence of wind turbines.
How did Dr. Pierpont discover this?
She went to a wind turbine site and she asked the residents to describe their symptoms. Then she recorded the results of these “first person testimonials” – as they were described to a scientist; she organized and compiled them in a manner that passed “peer review” with flying colors (fulfilling the rigorous terms of the “Goldstone Standard”); and then she published them to alert the world that she thought that there was a burgeoning problem that merited our attention – and further study.
As Curt Devlin notes, Dr. Pierpont also called attention to the limitations of her study – primarily the modest sample size – and invited others to “reproduce” her studies to validity the truth of her findings.
In fact, other researchers have validated her findings; and no researcher has invalidated her conclusions.
In fact, there is a growing consensus – even at the Massachusetts Clean Energy Center (which never laid eyes on a wind turbine project it couldn’t love – including proposals to install 500 foot wind turbines less than 600 feet from residences in Eastham and Bourne) – and even at the Massachusetts Department of Environmental “Protection”, which is headed by Ken Kimmell – author of the ill-fated Wind Energy Siting Reform Act to ram wind turbines down the throats of recalcitrant communities and who engineered the infamous MA DEP sham science study – that wind turbines cause serious adverse health impacts.
Largely because it is increasingly difficult – if not impossible – to ignore the voices of thousands of people who have suffered the direct consequences of these installations – the patients whose symptoms even you will one day grudgingly acknowledge – no doubt with a wistful sigh, saying:
“If only we had had the peer-reviewed studies then that we have today, we could have avoided this whole problem.”
But the main reason is that, sooner or later, over time, with or without a “peer review”; with or without any reference to the “Goldstone Standard”; the truth always asserts itself.
I hope that you don’t mind that I am circulating this letter to some of your peers in the press who, like you, regularly report on wind energy in Massachusetts. I will also be posting it on the same website where Curt Devlin published his critique of the fundamental bankruptcy of your religious devotion to the concept of “peer review” – even as you ignore the vast wealth of evidence right in front of you on this important subject (see link below).
I hope that you will take the opportunity to respond, so that we can all gain a better understanding of your point of view, and that you will post your response on the same website for the benefit of all.