Ask Ethan: What is the true purpose of scientific peer review?

Every so often, a new scientific result, theory, idea, or claim starts making headlines: not just in scientific circles, but in popular media as well. Most often, the one question all people know to ask is whether or not that paper has successfully passed peer review or not. If it hasn’t, people often dismiss the work, noting that we should remain skeptical because it hasn’t yet been vetted by anyone else with the appropriate expertise. But if it has passed peer review, people often assume that means everything that’s written in the paper — the methods of the study, the analysis performed, the results obtained, the conclusions drawn, and other assertions that the authors might make — must be correct. Even if it flies in the face of conventional wisdom, the fact that it has passed peer review means that everything that’s written in the paper need to be taken seriously.

But is that really what peer review means? Moreover, does that even reflect a proper understanding of what the purpose of peer review actually is? This week’s Ask Ethan question isn’t drawn from a layperson, but rather from a fellow science journalist, Ashley Balzer Vigil, who was reporting on a controversial (but successfully peer reviewed) study, and asked me,

“I guess what’s tripping me up the most is how could someone who seems so knowledgeable in the field write a paper that was then published by a respectable journal if the conclusions are blatantly incorrect? Should the journal have rejected it?”

At the heart of these questions is what the process of peer review is actually intended to do: both in theory and in practice. By understanding what this is all about, we can better make sense of the latest science news that comes across our radar: whether we’re experts in that field or not.

The event horizon of a black hole has been considered an important factor in the generation of Hawking radiation around black holes in many previous studies, but a new one from 2023 suggests that this radiation can still be generated outside of an event horizon even if the “horizon” itself does nothing more than forbid light from escaping from within it. This picture is non-rigorous as formulated, however, and others have subsequently disputed these claims.

Here are some claims that have been made, recently, in bona fide scientific papers that have passed peer review.

• That exoplanet K2-18b, a cold, Neptune-like world, was claimed to have biosignatures of dimethyl sulfide and dimethyl disulfide in its atmosphere.
• The Universe is 27 billion years old, not 13.8 billion years old.
• There are structures that we’ve discovered that are more than 10 billion light-years long present in the cosmos, defying the cosmological principle.
• Hawking radiation happens even for objects that aren’t black holes, meaning everything, from protons to the entire Universe, will decay.
• We’ve discovered organics on Mars, and all non-biological sources for their origin have now been ruled out.
• And that the recurrent nova, T Coronae Borealis, is about to brighten once again, and we can even provide specific dates that we should look for it.

I chose these claims, specifically, not because they’re outliers, but because these are examples of a very, very common phenomenon: a scientific paper making a strong but dubious claim that would be revolutionary for our understanding of reality, if true.

They also share something else in common: these claims are almost certainly not correct. We’ll return to each one of these claims in detail to unpack why, but first it’s worth addressing the core questions here: how could each of these papers, if they contain claims that are almost certainly not correct, have passed peer review? How does peer review work, and what is its purpose if not to prevent incorrect claims from getting published to the greater world? What does it mean when a paper has passed peer review, and why is that an important, but fundamentally limited, step in the scientific process?

The high-redshift galaxy RXJ2129-z8HeII contains an ionized helium feature as well as a severely blue tilt to its stellar spectrum. However, it contains extremely large quantities of oxygen, making it a terrible candidate for pristine material, despite the assertions of the authors of the relevant paper, based on the current evidence.

When you have a scientific idea, discovery, or result, we have to remember what the whole purpose of the enterprise of science is: to enhance and improve our collective understanding of reality. This is not the enterprise of one person, a group of people, a set of researchers, a consortium of universities, or of an individual nation. No, this is a collective enterprise: the enterprise that represents perhaps the greatest aspect of human civilization of all. Science is something that is for everyone, everywhere, as the lessons we learn about reality are for all of humanity to share in and reap the benefits of equally.

Because of this, as scientists, we want those creative ideas (even the wild ones that are unlikely to pan out), those new discoveries (even the ones we don’t fully understand or are able to make sense of), and those novel results (even the ones that defy our expectations, and even the ones that come in at low confidence) to all see the light of day.

That’s why the main purpose of peer review isn’t to keep papers that may contain statements that don’t hold up, over time and with further scrutiny, from making it into the scientific discourse. On the contrary, the purpose of peer review is to determine whether a paper has enough merit that it’s worthy of becoming part of the scientific discourse of any field.

The whole basis of the gravitational lensing study that claims to favor wave-like dark matter is encapsulated in this diagram. The authors simply model the normal and dark matter as shown, show the standard lensing predictions with crosses, and the actual observations with circles. Where the crosses and circles don’t overlap, they claim it disfavors particle-like dark matter. With 75 realizations of possible wave-like dark matter solutions shown (color-coded points), they assert that these points fit the data much better. Is this convincing?

In other words, if a paper passes peer review, that means that at least two people — the editor of the journal to which it was submitted, and the referee (or referees) of the paper to whom the editor referred/assigned the paper to — have decided that yes, this paper and what’s presented within it is worthy of being discussed and considered by others in the field: peers. It means that:

• If there’s a new idea, even an idea that’s very likely to be wrong, that at least those two people have deemed the idea has sufficient merits such that it ought to be considered by the broader community.
• If there’s a new discovery, even a discovery that we don’t fully understand, even if we can’t make sense (yet) of what’s been found, that discovery should be reported and taken into consideration by others in the field.
• And if there’s a new result, even a result that’s unexpected or that contradicts our theoretical expectations, that result should be investigated further and brought onto the radar of anyone who’s interested in that field.

In other words, passing peer review doesn’t mean that a paper is right. It doesn’t mean that what’s presented in the paper is going to hold up to further study or scrutiny. It doesn’t even mean that everything in the paper is sound and correct. It just means that two entities, the editor of the journal who’s charged with stewarding the respectability and reputation of that journal, and the referee who reviewed the paper, have deemed that the work is important enough that anyone who reads their journal should read and consider this paper, too.

This graph, from Lee and Kim’s first paper on LK-99, shows the applied current (x-axis) versus the measured voltage (y-axis) across their LK-99 sample at several different temperatures. The y-axis is displayed at insufficient resolution to tell whether the voltage truly goes to zero or is merely small, and the lack of any data points along the vertical lines, which should show the transition to a superconducting state, are serious omissions that should raise the hackles of anyone inclined toward skepticism about this result.

It’s also important — because of the concept of scientific freedom-of-thought — to allow the authors leeway to be provocative. Although the leading ideas in science, particularly if a consensus has already been achieved on a set of particular issues, are profoundly powerful, science thrives in an environment where those ideas can be legitimately challenged on all fronts: theoretically, experimentally, observationally, or on the grounds of self-consistency. If we only allowed results that agreed with (or are at least consistent with) what has been previously been established, science would lose the capability of advancing and refining, or even revolutionizing, our understanding of how things work.

When a paper passes peer review, that’s the paper receiving the “green light” signal to go forth into the community, and to be discussed and considered, and debated and potentially refuted, by the remainder of the community. That can include checking and attempting to reproduce the work, independently. It can include bringing in other lines of evidence that either support or contradict the work in question. It can include reanalyzing and reinterpreting the raw data, particularly in ways that the original authors neglected. It’s part of the scientific process, and why we so often say that science is a self-correcting enterprise: because claims that are flimsy will, and must, get appropriate scrutiny.

Deviations of iron isotopic ratios from terrestrial ratios in nine spherules, reproduced from Figure 12a of Loeb et al. The pink line adjoining them is the terrestrial fractionation line (TFL) on which samples should array if they have a Solar System origin and experienced chemically-driven mass-dependent isotopic fractionation by gain or loss of Fe, for example by vaporization of Fe. All nine spherules are exactly consistent with a Solar System origin.

This seems to imply that there’s an inherent contradiction in how we conduct science:

• on one hand, we rely on peer review to separate research that lacks merit from meritorious research that deserves a second (and third) look, and a follow-up,
• but on the other hand, peer review can often let through papers filled with ideas, results, analyses, or conclusions that absolutely cannot be supported by the data that we have.

For science itself, this actually isn’t much of a problem. Peer review is not the final line of defense in separating fact-from-fiction; it’s the first line of defense: sort of a “you must be this tall to ride this ride” bar that they set up at your local amusement park.

It’s only when those papers filled with unsubstantiated, unsupportable claims start making the popular rounds that misinformation flies, and that people start getting the impression that something is fundamentally wrong with how science is conducted. When people without the requisite expertise to analyze a scientific result start opining about it — and, in particular, when they become enamored of those results and become credulous about them — that we run into problems. While scientists are typically extremely careful, demanding a high standard of evidence to convince them that a currently held position needs re-evaluating, the general public, lacking the necessary expertise, cannot make such a determination with any accuracy. Their misplaced confidence, coupled with a firehose of information they’re ill-equipped to evaluate, is their downfall.

Although the JWST MIRI spectrum of exoplanet K2-18b is consistent with a series of light molecules like methane and carbon dioxide along with DMS and/or DMDS, the “significance” of 3-sigma was only obtained because all other possible gas species that could exhibit a strong absorption feature beginning at 9 microns were excluded from the analysis. There are other strongly viable scenarios that must be considered as well, and when they are, the evidence for DMS and DMDS disappears to be insignificant.

It’s why, earlier this year, a claim that we detected biosignatures on exoplanet K2-18b was of such interest. All at once, it:

• represented a bold, revolutionary claim,
• that passed peer review,
• that relied heavily on a very dubious analysis of the data performed by the authors,
• that was coupled with a very splashy, uncritical press release,
• that made worldwide headlines,
• before being widely debunked by other researchers in the field, who noted the shoddy analysis, the lack of appropriately considered alternative explanations, and a gross overconfidence in sets of assumptions made by the original paper’s authors.

The problem is not that such a paper passed peer review; the paper itself was “fine” by scientific standards. The poor analysis, based on wildly optimistic and restrictive assumptions, was something that the authors were (and should be) granted leeway to do. It’s telling that the broader community wasn’t upset at the paper at all; they were upset at how the researchers basically exploited a gullible media ecosystem to, as they say, “try their case in the court of public opinion,” despite knowing full well that they were using their platform to write checks that the scientific data couldn’t cash.

They were exploiting a hole in the way information gets disseminated to increase their own profile. It had the effect of grossly misleading the general public: an effect that serves to erode trust in the scientific process, which no amount of subsequent debunking by the rest of the community could repair.

This image shows a blackbody spectrum at 2.998 K (blue line) and how that spectrum shifts if the Universe expands: to eventually yield a cooler blackbody of 2.725 K (black line), which is what the COBE data from the 1990s indicated. (As shown with 400-sigma error bars.) The red line corresponds to how the 2.998 K CMB would have changed under a tired light scenario: losing energy but failing to maintain its blackbody character. We can therefore determine that no more than 0.001% of the CMB can be composed of “tired light” photons.

Is the Universe 26.7 billion years old, does it not contain dark matter or dark energy, and instead does it contain tired light and evolving fundamental constants? No, of course not. All of those points, both separately and in tandem, can be (and have been) debunked on scientific grounds, but the initial reporting — as well as a substantial amount of follow-up reporting — focused on the provocative claims, not the scientific merits (or lack thereof) of the research.

A lot of attention, back in 2024, was given to the “discovery” of large collections of galaxies, discovered by looking along the line-of-sight of background quasars, that appeared to make patterns like rings and arcs on the sky that were extraordinarily large: as in several billion light-years in length. This is a case where the science is interesting, in the sense that using the absorption signatures of the background quasar light to reveal signatures of star-forming galaxies that lie along those lines-of-sight is a spectacularly successful new technique. But it’s also a case where the interpretation of the data, and stating that those galaxies are actually clustered together as part of a real structure, is riddled with flaws. Using quasars or gamma-ray bursts to map out the Universe is an exercise in selective incompleteness, and until those gaps are filled in with deep galaxy surveys, asserting “there’s a structure there” is unsupportable by the evidence.

The blue points, in the background, represent identified quasars, which “backlight” the clouds of matter in front of them. The grey points represent absorption signatures of that quasar light by singly ionized magnesium, which is only produced in the presence of heated gas. The alleged “Big Ring” is shown near the center of the image, and makes more of a helix-like shape in 3D space than a ring. The distance between even adjacent sources, here, is greater than the distance from the Milky Way to the Virgo cluster.

A little farther back, in 2023, a fascinating paper came out asserting that perhaps Hawking radiation isn’t just for black holes, but rather should apply to any massive object that led to spacetime curvature. The arguments were straightforward and the calculations were simplistic, and it, too, garnered lots of attention. But it was only months later that various corners of the physics community came together and realized, “oh, the lack of an event horizon prevents such radiation from being emitted from atom-based matter,” even though that result was initially demonstrated back in 1975. It deserved the attention from the community in a profound-but-underappreciated way: to understand why the claim was wrong.

And most recently, just earlier this month, the claim that there are organics on Mars (which is true), and that those organics are likely a biosignature because all known abiotic explanations have been ruled out (which is less true), has spread rampantly. The authors did consider some abiotic pathways for creating the compounds that were found in the Martian rock, and determined that those pathways were non-viable. But did they, as was claimed, rule out all abiotic scenarios? Of course not; they only ruled out the scenarios they considered. Plenty of other ones, some of which may yet be plausible, still await examination.

Plenty of other examples abound: where peer-reviewed studies have made claims that took the public by storm, only to be demonstrated to be false later on. The star system T Coronae Borealis, although it did go nova twice in history, once in the 19th century and once in the 20th century, is not well-enough understood for us to predict when the next nova will be. Although a few attempted to make predictions as to when it would occur and those predictions received a lot of attention in popular media those dates have all passed, and no nova has yet taken place. It’s a case where speculation, rather than robust science, rose to popular prominence. And let’s not get started on whether the interstellar interlopers that we’ve spotted passing through our Solar System are aliens or not. (They are clearly not, as the data has overwhelmingly shown.)

This Hubble image, from July 21, 2025, shows the interstellar comet 3I/ATLAS, with the telescope tracked on the (moving) comet to keep it stationary in-frame, while the stars in the background blur behind it. Hubble’s imaging reveals a relatively small size for the central nucleus, with a diameter no greater than 5.5 kilometers, while revealing an enormous extent to the coma, or halo, surrounding it: extremely consistent with a traditional comet.

So, yes, we do have a problem, here in our world, that lots of very dubious conclusions, on the strength of their scientific merits, are constantly streaming out into the world. It’s driven by a mix of factors, including:

• zealous researchers who are eager to get their names and work out there,
• contrarians with enormous blind spots (the crowd who are guilty of fooling themselves),
• press officers at research institutes and universities who see their job as increasing publicity, not as disseminators of accurate, truthful information,
• a media landscape that rewards “revolutionary-if-true” claims, even when the claims themselves are not accurate or true,
• and a general public who lacks the necessary expertise — and that too often believes that those possessing such expertise mean to unscrupulously silence dissenting views, rather than to scrupulously smackdown unsubstantiated assertions — to discern the merits and criticisms of a research work for themselves.

But the solution to these problems is not to raise the bar as to what peer review needs to accomplish. You cannot replace expert analysis of an idea by the scientific community, which takes lots of time, effort, and expertise, with a process as simple as peer review. In fact, I would go as far as to assert that there is no single peer reviewer, in any field, that is capable of being the perfect gatekeeper; no such singular expert exists. Instead, it is only through the collective efforts of those possessing the requisite scientific expertise, over long periods of time, that the scientific consensus bends towards the most accurate representation of reality. Peer review is a wonderful resource, but it is never the final arbiter of whether a claim will turn out to be true or not. That’s not necessarily a flaw in the system; it’s simply a part of how the real, messy scientific process works.

Send in your Ask Ethan questions to startswithabang at gmail dot com!