A correspondence with The Lancet about an apparent subtraction error
COPE guidelines in practice
This is my correspondence with The Lancet on the 2023 paper "Does repeated influenza vaccination attenuate effectiveness? A systematic review and meta-analysis" following my comment on PubPeer from January 2024.
The editors and authors have been notified of this post and given the opportunity to respond here or on PubPeer.
Subject: Does repeated influenza vaccination attenuate effectiveness
Alex Byrnes
Dec 15, 2024, 10:33 AM
To: Emma Grainger
Dr. Grainger,
I have been waiting for a reply on a simple error in this article for almost a year now. The equation that "column A + column B = column C" is wrong in many cases. I have run this by virologists and I think I was very clear and polite. It should take no more than 20 minutes to see if there's anything there.
Thank you,
Alex
Jan 20, 2025, 10:40 AM
Dr. Grainger,
Following up with this after a month. In a few weeks, I will contact COPE with a complaint and document this exchange on Pubpeer. Please respond as soon as you can. I believe the time to resolve this should be minimal and I'm happy to drop it if I'm incorrect.
Thank you,
Alex
Jan 20, 2025, 12:22 PM
Dear Alex,
Many thanks for your email. I don’t think I have received this query submitted to the journal, I don’t monitor PubPeer. Please can you outline to me the details of any proposed error(s) and I will reach out to the corresponding author accordingly for their response.
All my best,
Emma
Jan 22, 2025, 11:41 AM
Dr. Grainger,
No problem and thank you for the reply. By "this query" you mean my original email? It includes a link to pubpeer: https://pubpeer.com/publications/CD503DF0831804823348007B2DAAFE Are you able to access the link? That's the proposed error. I'm sorry if this sounds litigious. I want to be very clear because these things can get muddy quickly.
Best,
Alex
Jan 22, 2025, 1:22 PM
to Emma, me
Dear Alex
Many thanks for your email. We will explore the query, discuss in the editorial team, and reach out to authors of the paper. Please allow us some time to follow-up and we will be back in touch.
We do not have a record of the initial email you sent so apologies for any delays.
Kind regards,
Diana
Dr Diana Stanley MD
Deputy Editor
The Lancet Respiratory Medicine
Jan 24, 2025, 9:37 AM
to Diana, Emma
Thank you, Diana. I will allow some time and check back.
Best,
Alex
Feb 21, 2025, 10:29 AM
to Diana, Emma
Diana,
It has been a month since our last email. You said there would be exploration, discussion, and emailing the authors. Can you say what point you're at now?
Thank you,
Alex
Feb 22, 2025, 9:00 AM
to me, Emma
We have reached out to the authors of the paper and are awaiting their response.
Many thanks for your patience.
Diana
Sent from my iPhone
Feb 24, 2025, 1:07 PM
to Diana, Emma
Diana,
No problem. Thank you for your work on this.
It's hard to tell exactly what you mean, but the process doesn't depend on the authors, only whether or not you are convinced of the miscalculation.
COPE guidelines say that you (The Lancet) should update the post-publication site yourselves. I will update the site next week monday. You have the full right to reply, of course, either here or on pubpeer.
Specifically, I think The Lancet is declining to enforce policy. The error is a miscalculation, if not more, and posting an expression of concern is appropriate. It should be done "quickly." The paper has been cited several times since our last email.
Best,
Alex
Feb 25, 2025, 3:02 PM
to me, Emma
Many thanks for your email. We have our statistician and authors reviewing the numbers and will respond as soon as possible.
Feb 26, 2025, 10:06 AM
to Diana, Emma
Diana,
Great! Looking forward to it. Do you mean the authors of this paper? Or other authors?
Best,
Alex
Feb 26, 2025, 10:59 AM
to me, Emma
Authors of this paper.
Feb 26, 2025, 12:20 PM
to Diana, Emma
Lovely! Good luck with it.
Mar 26, 2025, 9:07 AM
to Diana, Emma
Diana,
It has been a month since our last contact. I'm going to update pubpeer with the information I have on the case next week Wednesday. If you would like anything withheld, please let me know before then. These emails are the only documentation of the journal's response so I think they are relevant. You're free to respond there after next week and I will try to address any disputes.
Best,
Alex
Mar 26, 2025, 11:59 AM
to me, Emma
Many thanks for the email and I share your frustration in the delay.
We have chased the authors and our statistician.
If we do not hear back from them in the next few days we will seek a further statistician opinion. We wanted to use the one who reviewed the paper if possible though.
I hope to have a follow up for you soon.
Stanley, Diana (ELS-NYC)
Mar 31, 2025, 1:33 PM
to me, Emma
Our statistician thinks that these are estimates based on pooled models which account for study weighting which is not a simple average (see below). He suggests we wait on the authors reply as there may not be an error. The stats person on the paper is looking at the query now and will be in touch this week.
Apologies again for the delay in getting back to you. I had a look and this query is worth addressing by the authors. It is possible there are errors in the table if the formula should be used directly as suggested by the authors who raised the comment. However, my understanding is that these are estimates based on pooled models which account for study weighting which is not a simple average. If the differences are supposed to be simple differences based on the data in the table, the query may be correct (this is not my understanding but having read the query, I am not sure!). In summary, I suggest asking the authors of the paper to address the query.
Kind regards,
Diana
Dr Diana Stanley MD
Deputy Editor
The Lancet Respiratory Medicine
Apr 1, 2025, 10:56 AM
to me, Emma
Dear Alex
We have now heard back from the Authors of the paper and from stats and they think that there is not an error in the paper but that in your calculation you did not meta-analyse the virus/study/season/age group-specific delta VEs to calculate your answers. The authors of the paper said they would be very happy to discuss any queries you have and so they can better understand your query. Are you happy for me to share your email and introduce you over email so we can resolve this concern?
Kind regards,
Diana
Dr Diana Stanley MD
Deputy Editor
The Lancet Respiratory Medicine
Apr 3, 2025, 11:21 AM
to Diana, Emma
Diana,
All of this is consistent with there being an error in the paper. If the null hypothesis is that there's no error, observing what The Lancet and the authors have said over the past year is very unlikely.
Unfortunately, I may have to keep coming up with publicly verifiable facts in order to comment on pubpeer. Other than continuing to audit this paper and putting my comments on pubpeer or another public place, what are my options according to The Lancet? I do not consent to giving up my anonymity to the authors either directly through you or indirectly.
The reason I'm asking formally is the COPE guidelines are flowcharts where I have options at each point. If there are no further options then I'd like to get it from you officially. This is also relevant outside of publishing matters if the authors want to know who I am. Please let me know formally if I have any further recourse through The Lancet.
I am still open to the explanation you mentioned. Preferably, you or the authors would post it on pubpeer. I will wait two weeks for a response or a post there. Please let me know if that will not be enough time.
Alex
Apr 8, 2025, 10:36 AM
to me, Emma
Hi Alex
As explained in the previous email both the statistician and paper authors think that there is not currently an error. They felt you have likely recalculated the estimates using a method different to what they have described. The estimates in question are pooled delta VE estimates. For each virus/study/season/age group they calculated delta VE as the absolute difference in VE from two different vaccination groups (vaccinated in the current season only versus vaccinated in the current and prior seasons). Those delta VE estimates for each study were then meta-analysed to arrive at the pooled delta VE estimates for a virus across studies within a season and across all seasons and studies. They suspect you have calculated the difference in meta-analysed pooled VE estimates; e.g. 33-45=-12, rather than meta-analysing the virus/study/season/age group-specific delta VEs.
The normal course for public discussion of errors/queries is through our correspondence section which we would be happy to explore if you want to submit a formal letter to us.
The authors of the paper are also to liaise with you directly to share the data and scripts to work through the calculations and copy in the journal to any correspondence.
Do let me know how you would like to proceed.
Apr 9, 2025, 10:16 AM
to Diana, Emma
Diana,
Thank you for the offer of a correspondence. I would be happy to see these issues in a correspondence but I am not interested in writing one. They tend to take a long time, and there is already a correspondence out in The Lancet on this paper.
I agree, though, that making things public is the right thing to do. COPE recommends putting a response in the same place as the original report, meaning PubPeer.
PubPeer discourages comments that haven't made an effort to understand the paper to protect authors from "homework." I think I should be given the same courtesy. You're welcome to include questions you have with any response, however.
Although The Lancet has not ruled out replying on PubPeer, I think you would have responded on that question here or there by now. So I doubt this will happen. I'm open to the response, though. You have an explanation of some kind from the authors now.
As I said, I can't respond on PubPeer without publicly verifiable facts like your response, or new questions about the paper. I believe there are other comments to make but I don't want to add complexity to what I think is a straightforward question. In the future, I will most likely be sending you and the authors a blog post to review and respond to. I follow the norm used by other post-publication reviewers to give the authors the right to reply and I would publish the responses along with the post. The Lancet is obviously involved so I would extend that to you as well.
All the best,
Alex
Apr 9, 2025, 2:01 PM
to me, Emma
Hi Alex
I will ask the authors if they wish to respond on pub peer. Our journal prefers to do exchanges in an open format through email or correspondence.
Do let us know if you decide to change your mind and submit a correspondence.
Addendum: Emails with the authors
Added June 2nd, 2025.
Does repeated influenza vaccination attenuate effectiveness
Tue, Apr 29, 2:17 PM
to Adam Kucharski, Annette Fox, Sheena G. Sullivan
Doctors Kucharski, Fox, and Sullivan,
From what I understand from Diana Stanley, at least some of you have been corresponding with her about my pubpeer comment on your paper: https://pubpeer.com/publications/CD503DF0831804823348007B2DAAFE.
I am planning to post my correspondence with Dr. Stanley on my blog redteamofscience.com in two weeks. The post is attached and I'm asking you for a reply that I'll publish underneath, or a response on pubpeer that I will link to.
I'm hoping you will respond on the pubpeer comment, but feel free to respond to the correspondence as well since that involves you too.
I would also like to reach the first author. If she agrees, can you forward her email address or send this message to her?
Thank you for your time.
Best,
Alex
Tue, Apr 29, 10:26 PM
from Sheena Sullivan
to Elizabeth J Robinson, Elenor Jones-Gray, Diana Stanley, Adam Kucharski, Annette Fox, me
Dear Alex,
Thanks for your interest in our paper.
Please refer to the paper’s supplementary material which provides the R script for our calculations. You should be able to reproduce the analysis using the information provided in the forest plots.
For each study/age-group/virus/season combination the delta is indeed calculated as the difference between the two vaccination group estimates (e.g. current and previous season - current season). However, the summary delta estimates you have highlighted are not a simple difference between two meta-analysed summary VE estimates for each age-group/virus/season combination; they are instead a meta-analysed estimate of the individual deltas.
I have posted this information in pubpeer.
Regards
Sheena
Thu, May 8, 1:15 PM
to Elizabeth J Robinson, Elenor Jones-Gray, Diana Stanley, Adam Kucharski, Annette Fox
Thank you for the reply! I am interrupting my scheduled posts for a conference, but I expect the Lancet thread I attached will be published on the 26th and I will have a response to your pubpeer comment the week after that. I will attempt to post a link on pubpeer so it will go out to anyone who has an email on the thread here: https://pubpeer.com/publications/CD503DF0831804823348007B2DAAFE
If you have any comments you would like to appear alongside a post instead of on pubpeer, please let me know.
Alex
My next post, a response to the authors’ PubPeer comment, comes out Monday, June 2nd, 2025.