You are viewing a read-only archive of the Blogs.Harvard network. Learn more.

The Longest Now

The Kostoff knowledge: Elsevier fakes peer review of COVID click-bait

The Kostoff knowledge v.14

Updates: Elsevier retraction (5/9), concern (12/17). EIC Tsatsakis removed. (~3/25).
Analyses by Schneider (10/6) & Morris (10/14). Kostoff’s article is top 1% by Altmetric.
K. publishes 3× more extreme version (10/13). Tox.Rep’s CiteScore grows 5% in Oct.
15 of Kostoff’s last 18 papers written w. Tsatsakis, the other 3 in Tsatsakis journals.

Earlier this month, Elsevier‘s Toxicology Reports (CiteScore 6.4, top quintile) published a special issue on the COVID-19 pandemic.  Its includes a remarkable article by Kostoff, et al., claiming that getting a COVID-19 vaccine is, “extremely conservatively“, 5x as likely to kill people over 65 as it is to save them, and even more harmful to younger people. (Kostoff, et al., Tox. Rep. (2020), 7, 1448-1458)

This echoes the fraudulent claims of German homeopath Harald Walach, who briefly published a similar article in MDPI Vaccines in June, before it was promptly retracted.  A few of the most outrageous claims are listed below. None of this is subtle – unbelievable assertions start in the second paragraph of the abstract; the lead author has no past experience in the field; and the article puts “pandemic” and “vaccine” in scare quotes, and makes regular use of bold italics to emphasize points that are exaggerated.

This is why we have peer review, and editors, to distinguish research from polemic. Access to a reliable + competent body of reviewers is, in theory, a primary service that giant publishers like Elsevier offer to editors. Another is their name: being an Elsevier journal means you will be taken seriously out of the gate, and added to the major indices.

We should all be concerned that our publishing model allowed such a deceptive essay to be given the veneer of legitimacy – for weeks now, without correction.  And we must hold both journals and publishers accountable for fraud that they support or legitimize – through deceptive practice, lack of claimed review, or inaction.

I want to come back to this, and discuss ways to remedy this, and some current steps in the right direction.  But first let’s look at this instance in detail – as the errors were the most obvious that I’ve seen, related papers have been retracted in recent months, and it is impossible to imagine even casual peer review missing them.  And because, as we will see, this particular Elsevier journal has been gaming the system for some time.

Article-level fraud (by the authors)

1. Extensive misuse of VAERS data: VAERS is an open public registry of unvetted self-reports of health events occurring after vaccination. Most events are not caused by vaccines, but this is a starting point for further analysis. Doctors are supposed to report any deaths or hospitalizations occurring within a week of vaccination, regardless of potential causal link.

The very openness of this data has led to it being widely cited in anti-vax propaganda, misinterpreting VAERS as a catalog of known harms and side-effects. (“Don’t Fall for VAERS scares“)

  • 1a: The article mentions that VAERS data is not causal; but then after a brief hand-waving assumes it is causal in the calculations. VAERS reports are not causal: most reported events have no connection to getting vaccinated (this mistake inflates their risk calculation by a factor of ~1000)
  • 1b: A 2010 working paper suggested that 1% of all adverse events, and perhaps 3-40% of serious adverse events, temporally following a vaccination, are reported to VAERS. The article cites this to suggest that 1% of causal deaths are reported, although detailed studies show that most potentially-causal deaths are reported. (this mistake multiplies their calculated risk by a factor of 100)
  • 1c: VAERS reports of all kinds fall off quickly in the days after a vaccine is administered. However the article assumes reporting rates are constant over time, leading to a large undercounting of the base rate of non-causal events, and an overcounting of events in the first week as causal. (this mistake is used to justify the inflations above)

2. Misstating risks of COVID:

  • 2a: The article suggests that when COVID-19 is listed alongside other comorbidities on a death certificate, COVID is not the cause of the death; and ignores all such deaths in calculations. This claim is false on its surface, and seems copied directly from anti-vax propaganda. (this mistake multiplies their calculated risk by a factor of ~20)
  • 2b: Throughout the article + appendices the authors drop in vague and counterfactual claims to claim these numbers are conservative, and actual death rates are higher.  These claims only start to make sense if you read anti-vax conspiracy theories. (They misstate how PCR tests are run, they claim no long-term effects from COVID (which has crippling and well-documented long term effects), they claim terrible and unknown long-term effects from vaccines (which are designed precisely to avoid those effects), they ignore the communicability of the disease & related public health impact). (these mistakes add a factor of 2-20)

Overall, this is an exercise in motivated reasoning. Basic statistics is misused, sources misquoted, and standard knowledge and practice misrepresented, to confirm a desired result. The topline numbers claimed in the article differ by a factor of 5 million from the best serious estimates of risk/benefit analysis for the vaccines. (Walach, in his previous retracted paper, only managed a fudge factor of 500,000.)

The authors, in compiling this pure thought experiment, also do not review or reference existing studies of the subject. The potential adverse effects of COVID-19 vaccines is an important topic, which has been studied extensively by epidemiologists and virologists (which none of these authors are). From the top page of a quick scholarly search:

  • Klein, et al (with a group of statisticians and epidemiologists) published a statistical analysis of 12 million shots: “Surveillance for Adverse Events After COVID-19 mRNA Vaccination“. They found exactly two significant effects – a higher risk of temporary myocarditis the week after the shot, and a lower risk of appendicitis in the 3 weeks after.
  • Canada published a comprehensive overview of the adverse effects from their first 20M vaccination shots, attributing 1 death, and contributions to 3 other deaths to those 20M shots.

Journal-level fraud (by the editors)

How did an article that was problematic by the end of the abstract, and fraudulent just beneath the surface, get published in a modestly successful Elsevier journal?

Background radiation:

  • The editor-in-chief of Toxicology Reports (Aris Tsatsakis) who took over in its fourth year, regularly appears as co-author on articles in his own journal.  The current special issue on COVID, in which Kostoff published two articles, lists Tsatsakis as co-author of 4 of the 8 articles published; three of these listed him as the ‘handling editor’ responsible for recommending acceptance.  (The journal has since issued two three updates, removing his name and instead listing one of his frequent co-authors, though that means little post-acceptance.)
  • This has happened before. In May, 2020, Kostoff and Tsatsakis published a similarly conspiracy-tinged thought experiment about the harms of 5G mobile networks, titled “Adverse health effects of 5G mobile networking technology under real-life conditions”.  It was published in a special issue of Toxicology Letters, a sister journal to Tox.Rep., also edited by Tsatsakis.
  • This has happened before… a lot. Kostoff wrote 14 of his last 17 papers with Tsatsakis, all published in journals Tsatsakis is involved with.
  • Other authors published regularly by Tsatsakis are independent researchers who focus exclusively on discrediting the efficacy of vaccines. For instance Neil Z. Miller, author of anti-vax bestseller Miller’s Review of Critical Vaccine Studies, is the sole author of the most-downloaded article in the journal’s history.

Editorial misconduct here:

  • The editor-in-chief (Tsatsakis) is a co-author of the paper, and the handling editor who accepted it for publication.  Weeks after publication, he was replaced as handling editor by his frequent co-author Poulas.  Poulas has been an outspoken champion of Walach’s earlier fraudulent analysis.
  • Tsatsakis and another co-author (Calina) are also co-authors of another article in the same volume, evaluating the risks associated with vaccination in the EU, whose assumptions and conclusions directly contradict those in this article. Each article has a title and thesis crafted to appeal to a different audience.
  • Peer review does not seem to be happening.  Most articles in this special issue are by frequent co-authors of the current editors, or by the editors themselves.  Multiple articles in this volume (a low-N ivermectin study whose conclusions misstate its results, a data-mining paper) suffer from methodological, statistical, or textual oversight that even a brief peer review would catch.
  • The direct implication of this paper is that noone, particularly no children or elderly people, should get a COVID-19 vaccine. By all but the most fringe scholarship on the matter, this is a deadly recommendation that will lead to widespread death and chronic illness, and an ongoing degradation in public health.  It was immediately picked up by explicit anti-vax campaigns. There has been no recognition of this potential impact, through an accompanying letter from the editor or some other response to the public outcry against this work.

Publisher-level fraud (by the managers)

In the last three months, we have seen three very high-profile examples of COVID conspiracists publishing dangerously flawed research as fact, under the banners of MDPI, JAMA, and Elsevier – some of the giants of the industry.  Moreover, while extensive complaints by hundreds of readers can sometimes lead to retractions, many flawed papers are never retracted, and the visibility of retractions is always a fraction of that of the initial flawed work.

Not only is there no accountability for the publishers, they may benefit from the attention of anti-vax and anti-science networks. And they directly benefit from the high publication volume, and volume-driven reputation metrics, that lets such fraud thrive.  This is a crisis at a grander scale, defrauding us all of an effective knowledge commons.

How did we get here?

Elsevier runs a popular and profitable journal factory. They encourage people to start new journals. A legitimate and respected scholar founded this one, led it for a few years, then passed on the reigns to someone else. The second lead editor began using it to publish himself and his circle of colleagues. A journal’s peer review is only as good as its reviewers and process, and there is no way to tell if opaque reviews are happening at all. Current models of scholarly recognition and tenure privilege raw publication and citation counts over most other things, so running your own journals (either publishing yourself or publishing people who can publish you) and randomly adding the names of others to your papers, directly advances your career.

Peer review, when done properly, is a scalable way to improve quality, annotation, and context of new research.  But it can also be faked, as a way to grant undue reputability to poor or downright harmful work.

How do we remedy this?

  • We need transparent descriptions by journals of their peer-review process (Transpose!), what responsibility they take for retractions and following up on misinformation they publish.
  • We need accountability for authors, editors, and journals, when major mistakes make it past review – and a step up in accountability when there is intentional fraud or systematic manipulation.
  • We need effective responses to fraud and retraction that invest energy in propagating the retraction proportional to the harm and misinformation propagated by the initial publication. (see NISO’s CORREC recommendation)

Kostoff and co-authors should not be welcome to post other less visibly-deceitful papers without passing a higher bar for diligence and review.  Toxicology Reports should not be included in mainstream metrics until it has changed its editiorial process.  The responsible editors should be held accountable by their co-editors and fellows in their discipline.  Elsevier, as the umbrella publisher giving their work legitimacy, should be accountable for allowing conflict-of-interest editing, poor science, and poor review to continue for years in this journal, until it reached this extreme state.  And metrics maintainers like Scopus and other maintainers of journal metrics should take responsibility for capturing this sort of fraud and manipulation in their metrics, not only after the fact for the small percentage that are caught by the public, but actively (through systemic review) and retrospectively (identifying past patterns that match recently-caught fraud).

There have been a few tiny steps towards this sort of review lately, but for the most part the authors, journals, and publishers have downplayed responsibility, blamed ‘hackers’, and evaded any accountability.

  • This month, Springer Nature added “expressions of concern” to 400 articles in Special Issues of “Arabian Journal of Geosciences” and “Personal and Ubiquitous Computing
  • In July, Elsevier issued “expressions of concern” for 400 articles in Special Issues of Microprocessors & Microsystems
  • In 2019, Scopus stopped indexing two of the four Impact Journals, Oncotarget and Oncoscience, both high-volume publications. However they declined to share a reason, so it is unclear whether the same reasoning might apply to the other two IJ journals (which share an Editor-in-Chief).

Dear Sam
Well done. Far too many crap articles are being published. Didn’t the tone of the article–even its Abstract–raise any suspicions from the reviewers?


Comment by Patrick Moriarty 10.04.21 @ 4:01 pm

[…] Center for Internet and Society at Harvard, began tweeting criticism of the paper, which he has now gathered at his blog. One […]

Pingback by Author Claims COVID-19 Vaccines Kill Five Times More People 65+ - Health News 24 - Health News 10.05.21 @ 3:43 pm

Patrick – judging from the replies of Kostoff and Editor-in-chief Tsatsakis reported by Retraction Watch, they both renewed their insistence that the article is legitimate.

If the reviewers that the editors choose are already anti-vaccine, and interested in publishing work that confirms their beliefs, the tone might have seemed a positive signal. That’s not surprising in itself — there are journals devoted to all sorts of non-science; Infinite Energy Magazine is going to continue to publish reports of perpetual motion machines.

The question is why a top publisher such as Elsevier has health-related journals with such editors.

Comment by SJ 10.07.21 @ 7:34 pm

Right on! My experience in paper production involved biology research long ago. The SARS-2 pandemic has led to my reading a many papers and, even without specialized knowledge, it is clear that the quality is poor — and this includes many honest and basically sound,but poorly worked up papers. And then there is the really bogus stuff. Something truly awful has happened in the world of science and medical research.

Comment by Bernard Biales 10.12.21 @ 11:56 am

When I saw your section heading Background Radiation I expected a reference to a paper in the Elsevier publication Toxicology Letters. Specifically, I expected a reference to the paper “Adverse health effects of 5G mobile networking technology under real-life conditions” with a list of authors including Kostoff and Tsatsakis in a special issue edited by a team including Tsasakis.

I must have been thinking of the wrong type of radiation.


Toxicology Letters, Vol 323, 1 May 2020, pp 35-40.


Comment by Bob 10.18.21 @ 5:31 pm

A good point, one I will update this post to include.

There is a small constellation of papers on the manifold harms of 5G, the benefits of nicotine, and the harms of vaccines, which have been published in Tox.Reports, Tox.Letters, and a few related journals, all by this group of authors + editors.

Comment by SJ 11.01.21 @ 12:52 pm

An erratum [0] was added to Kostoff’s paper by the publisher:

“The publisher wishes to clarify that Dr Aristidis Tsatsakis, the Editor-in-Chief of Toxicology Reports, was not involved in the peer-review of this article. Full responsibility for the editorial process for this article was delegated to the Handling Editor Dr Konstantinos Poulas.

The publisher would like to apologise for any inconvenience caused.”


Comment by Rising Actions 10.24.21 @ 4:52 pm

Thank you for the update. They have now published at least three such errata for various articles in this series. As Poulas is a frequent co-author of Tsatsakis, and himself frequently publishes in this journal, that still goes against the best journalistic practice.

But the cycle of in-group citation, review, and publication — while subverting the intended function of peer review — is difficult to recognize or stop for a sufficiently large in-group, precisely because there is no longer a single step of COI, but rather a slightly longer chain.

Comment by SJ 11.01.21 @ 1:03 pm

[…] The paper is on the Retraction Watch website, a website that tracks the process of retraction, the process by which a paper is pulled by the publisher.11 Sam Klein wrote a pretty comprehensive and fairly accessible critique on the paper on a Harvard blog, The Longest Now.13 […]

Pingback by Sunny's Blog 11.01.21 @ 7:39 pm

What I don’t understand is why nobody simply looks at countries without covid in the community (or very, very little) that have high vaccination rates. Australia is an example, so is my home country New Zealand. We barely have any community cases in New Zealand and practically no one dying with or due to covid. 7 million doses of Pfizer have been administered so far, vaccinating 12+ year-olds throughout our country of 5 million people. Data shows that there is no excess mortality.
How can anybody publish a paper about a huge number of people dying from the vaccine if it can be so easily disproven by just looking at statistics? Shouldn’t this be the go-to response to any of those absurd claims? You don’t even have to be a scientist, a brain in your skull is all that’s needed.

Comment by Sarah 11.05.21 @ 1:51 am

You are right, of course. These conspiracists have to disbelieve excess-mortality data, and believe all vaccine harms are classified as something else, to hide them … except in the public VAERS / Eudra / DAEN databases. It doesn’t make sense.

But a convoluted multi-step argument based on dozens of different errors allows for a Gish gallop, which is good enough to convince many…

Comment by metasj 11.07.21 @ 12:49 pm

Thank you for mentioning my article!
But i have something much better: Smut Clyde taking on the Tsatsakis gang in this new post:

Comment by Leonid Schneider 11.23.21 @ 4:44 pm

[…] to its newsroom for further questions, instead of the editors.The article was the subject of a critique by Samuel Klein, and was corrected sometime after we posted on it in October to say that a different editor — […]

Pingback by Elsevier subjects entire special issue of journal on COVID-19 to an expression of concern 04.08.22 @ 8:28 am

[…] Center for Internet and Society at Harvard, began tweeting criticism of the paper, which he has now gathered at his blog. One passage:Overall, basic statistics is abused; sources misquoted, and standard knowledge and […]

Pingback by Author defends paper 04.10.22 @ 11:35 pm

Bad Behavior has blocked 150 access attempts in the last 7 days.