All that glitters is not gold

This blog post is not an accusation of misconduct, and reflects my personal opinion.

This paper has it all. The authors are from Stanford University’s School of Medicine and Harvard Medical School. It was published in 2002 in Nature Medicine, and has been cited over 900 times, as per Google Scholar. It even received a prestigious F1000 Recommendation. And it was supported by six NIH grants.

Akbari O et al. Antigen-specific regulatory T cells develop via the ICOS–ICOS-ligand pathway and inhibit allergen-induced airway hyperreactivity. Nature Medicine 8(9), September 2002. DOI: 10.1038/nm745.

But all that glitters is not gold. Let’s take a look at some of the flow cytometry images in this paper.

Figure 2 appears to have some unexpected repetitive patterns. The mice in the left panels got sprayed with Ovalbumin in their noses to induce an allergic reaction. The mice in the right panels received only PBS (control experiment). But unexpectedly, some groups of cells, marked by me in pink, lime green, light orange, and red, appear to be visible multiple times within the same panel or across panels.

Several Flow Cytometry experts assured me that this is very unusual. Even if you would run the same samples again in the same machine, you would expect to measure dots at slightly different positions. The overall distribution (the “clouds”) might look similar, but the individual data points (called “events”) would be in a different configuration. It would be comparable to tossing sugar repetively on the floor; the grains will end up at different spots, even if you throw the exact same amount of grains at the same force, in the same direction.

So it is scientifically correct to be raise concerns about flow cytometry plots with putative repeating groups of events. Some folks on Twitter even called this phenomenon Flaw Cytometry (coined by Kevin Vervier) or Faux Cytometry (by Konrad Rudolph).

Figure 3G appears to suffer from the same repeats. The Pink Cloud visible in Figure 2 (see above) might also appear here, but in different positions, and representing different experiments.

Figure 4 shows some “stripy” clouds of dots that were all gated separately. The numbers above the gates represent the percentages of cells in that gate. However, these narrow gates all appear to look …. unexpectedly similar.

Moving on to Figure 5 which almost looks like it might have been generated from parts shown in Figures 2 to 4, such as the pink, red, light orange, and blue groups of events.

You can read my concerns about this paper here on PubPeer.

Additional papers with concerns

The first author of the paper discussed above has written some other articles that were honored with posts on PubPeer.

These papers span this author’s career from working as a PostDoc in a lab at the Department of Pediatrics at Stanford University School of Medicine, then becoming an Assistant Professor at Harvard Medical School, and his Professor position at the University of Southern California.

Several pseudonymous users, including Bolboschoenus Maritimus, Hoya Camphorifolia, Pandanus Discostigma, and Rhagovelia Zela raised concerns about these papers. Some of these concerns were posted more than two years ago.

Here are the additional papers, in chronological order:

1. Akbari O et al. Pulmonary dendritic cells producing IL-10 mediate tolerance induced by respiratory exposure to antigen. Nature Immunology (2001). DOI: 10.1038/90667. [PubPeer link]
2. Akbari O et al. Essential role of NKT cells producing IL-4 and IL-13 in the development of allergen-induced airway hyperreactivity. Nature Medicine (2003). DOI: 10.1038/nm851. [PubPeer link]
3. Stock P et al. Induction of T helper type 1–like regulatory cells that express Foxp3 and protect against airway hyper-reactivity. Nature Immunology (2004). DOI: 10.1038/ni1122. [PubPeer link]
4. Stock P et al. CD8(+) T cells regulate immune responses in a murine model of allergen-induced sensitization and airway inflammation. European Journal of Immunology (2004). DOI: 10.1002/eji.200324623. [PubPeer link]
5. Maazi H et al. Lack of PD-L1 expression by iNKT cells improves the course of influenza A infection. PloS one (2013). DOI: 10.1371/journal.pone.0059599. [PubPeer link]
6. Maazi H et al. ICOS:ICOS-ligand interaction is required for type 2 innate lymphoid cell function, homeostasis, and induction of airway hyperreactivity. Immunity (2015). DOI: 10.1016/j.immuni.2015.02.007. [PubPeer link]