Face value: sunscreen, lack of control group, and conflict-of-interest

A reader pointed out a 2016 paper published in Dermatologic Surgery.

Daily Use of a Facial Broad Spectrum Sunscreen Over One-Year Significantly Improves Clinical Evaluation of Photoaging – M. Randhawa et al. – Dermatologic Surgery: December 2016 – doi: 10.1097/DSS.0000000000000879

In this study, 32 women aged 40-55, all with light skin, were recruited. They all had to apply a sunscreen to their face in the morning, and a simple moisturizer in the evening. They had to do this for an entire year, and were also advised to avoid excessive sun exposure during that time.

After 3, 6, 9, and 12 months, the women’s faces were evaluated by a dermatologist for photo-aging, as assesses by skin clarity, texture, fine lines, and pigmentation spots.

Amazingly, all women showed spectacular improvement of their facial skin. The authors concluded that the daily use of sunscreen improved the appearance of skin in all women.

I disagree.

There are lots of problems with this paper. I posted these on PubPeer today, and decided to elaborate a bit more here.

No control group

Here is the first problem: the study does not have a control group. All women had to apply the sunscreen and the moisturizer. There was no control group in which women did not have to apply the sunscreen.

A good control group, for example, could have been another set of women who had to apply the same cream as the sunscreen formulation, but without the photo-active ingredients. Or, a group of women who only had to apply the evening moisturizer, without applying the morning lotion at all.

Incorrect conclusions

The second problem is that the authors jumped to conclusions.

The authors concluded that the daily use of the sunscreen significantly improved the “photo-aging” of these women’s skin. However, other ingredients in the sunscreen or in the provided evening-moisturizer might have also contributed to that. The paper lists the photo-active ingredients in the sunscreen, but presumably this was some sort of cream or lotion, and other ingredients might have been added that could have been done some good for the skin. The evening-moisturizer could also have played a big role in restoring the skin.

In addition, all women were advised to stay out of the sun. Maybe that advice alone could have helped their skin to improve? Without control group, it is really hard to know if this advice alone made any impact.

A better conclusion would have been that the women’s skin improved because of the daily application of sunscreen in the morning and moisturizer in the evening, and staying out of the sun. All these factors could have played a role. But, the lack of control group makes it impossible to tease out these confounding factors.

Conflict of interest

Let’s look at the third – and probably biggest – problem: Conflict of Interest.

The sunscreen was provided by Johnson & Johnson, the US-based multinational that makes products like Neutrogena, Band-Aid, Tylenol, and baby products.

Three of the six authors of this paper were J&J employees, while the other three were consultants or employees who received compensation for their work on this paper. These are clear conflicts of interests, and although they are mentioned in the paper, one may ask the question how objective these authors might have been in evaluating the women’s skin during the study.

The J&J-paid dermatologist (JJL) stated that 100% of the women had much clearer skin and that 90.6% had a more even skin tone and improvement in discrete pigmentation. Wow, what a perfect result. One could wonder if receiving money for this statement made any difference in the dermatologist’s assessment.

In contrast, only about half of the women thought their skin looked better, and only one third thought they looked younger.

Given the subjective scoring by a doctor with a huge conflict of interest, one can become a bit suspicious if the results are so perfect. Looking at Figure 4, almost all subjects improved “significantly” in all parameters at all visits. Note that the doctor looked at these faces while knowing which visit it was (12 weeks vs 24 weeks, for example) and which product they had used (since there was no control group).

There are no error bars to be found in this paper, but all results were, if we believe the authors, perfect and significant. The lack of proper statistical analysis is very disappointing, because the acknowledgments mention “David Lewin, PhD of Statistically Speaking Consulting LLC” for “the data statistical analysis”. One could have hoped that with such expertise advice, standard deviation or error bars could have made it into the bar graphs.

Before and after photos

Let’s take a look at the two sets of “Before and after” photos that the authors provide in this paper. It is unclear if the women portrayed here gave permission to use their photos in the paper (they consented to the study, but there is no word on photo usage) but let’s assume they did.

According to the paper “after 52 weeks of sunscreen use, the face showed significant smoothing of texture and significant decrease of crow’s feet fine lines” (Figure 2) and “significant decrease in overall skin tone and localized pigmentation” (Figure 3).

I looked really hard, but other than the slightly smoothened area in the red circle in Figure 2, I don’t find the skin of these faces significantly improved. In fact, I find these before/after photos remarkably similar. Much more similar than I would have expected after 1 year.

Industry papers

In principle there is nothing wrong with papers funded by, or co-authored by industry partners. But if a study does not have a control group, if subjective assessments are done by industry-paid doctors, and the results are too perfect and all claimed to be significant without any error bars, we have the right to be skeptical.

As I wrote above, the study would have been much more believable if it had included a control group. Some other suggestions would be to score the faces by at least 2 different dermatologists, and to score them blindly (without knowing which group or timepoint the faces/photos belong to).

As a peer reviewer, we should be paying attention to these conflict-of-interests and subjective, unblinded assessments as well. This should not have passed peer review in my opinion.

You can read my comments about this study on PubPeer. Feel free to pitch in on PubPeer or in the comments below.