Thoughts on the Prevent Senior study

A new study from Brazil is reporting significant positive effects of hydroxychloroquine (HCQ) + azithromycin (AZM) on early-stage suspected COVID19 cases. The study was posted as a preliminary manuscript draft on Dropbox a couple of days ago. I just tweeted my thoughts in a long thread, but here is a bit more polished version.

Update 20 April: It was announced today that the study described below has been suspended because of ethical violations. As pointed out by Natalia Pasternak and Carlos Orsi and Ricardo Parolin Schnekenberg (see Additional Reading links below), the study had already started before the ethical approval had been obtained. This could be figured out by looking at the disclosed study days in the preprint and the trial registration at the Clinical Trials website.

A preprint, not a peer-reviewed paper

The first thing to note is that this manuscript was posted on Dropbox, not onto a regular preprint server such as bioRxiv or medRxiv. This is NOT a peer-reviewed paper. It is draft meant for others to comment on.

Some people were worried that this might have been a leaked private link, but others have confirmed that the authors posted the link in public.

Summary of the study

Here is a short summary of the study:

636 senior patients (average age = 62.5 years) were enrolled by “telemedicine” (I assume that meant that they called their doctor as opposed to visiting the hospital) or after they visited an ER doctor. In order to be enrolled, they had to have COVID-19 like symptoms, such as a fever, cough, muscle aches, etc.

The enrolled patients were offered treatment with HCQ+AZM. If they accepted that treatment, the patients were assigned to the treatment group. If they refused, they were assigned to the control group. The endpoint is a bit unclear, but after a certain number of days the number of hospitalizations was reported.

The HCQ+AZM treated group (n=412) had 1.9% hospitalizations, while the untreated control group (n=224) had 5.4% hospitalizations. In the treated group, early treatment was associated with fewer hospitalizations. So, it is concluded that the HCQ+AZM treatment results in fewer hospitalizations in COVID-19 cases, in particular if the treatment is started early.

The authors are all affiliated with the Prevent Senior Institute in São Paulo, Brazil. The study was pre-registered at Clinical Trials under identifier NCT04348474, “Efficacy and Safety of Hydroxychloroquine and Azithromycin for the Treatment of Ambulatory Patients With Mild COVID-19” although the current status is “Not Yet Recruiting”.

No randomization

The 2-armed study has an unusual set up. This study was not randomized at all. Patients could chose in which group they wanted to be. This could create all kinds of problems, in particular because patients knew if they were being treated, and because the decision to choose the treatment might have been influenced by how sick they felt or how long they had been sick.

Luckily, Table 1 (see below) shows that the two groups do not differ in terms of average age, gender ratio, and prevalence of certain chronic diseases or smoking. That is good.

Unclear if these patients had COVID-19

Patients were recruited if they had persistent flu-like symptoms for 2 or more days, with a suspicion for COVID-19 infection, but no PCR test was done. This is a big red flag. We are not sure if they had COVID-19 or if they had another illness, such as the common cold.

Typical symptoms for COVID-19 are fever and cough. Yet, at baseline, of these 636 patients only 6.6% had a fever and 43% had a cough (see Table 1). These 2 symptoms also differed significantly between the 2 patients groups. If you add up the list of Flu-like symptoms provided in Table 1, the totals add up to 116.9% in the total group. Of course, a patient could have had multiple symptoms, so the summed percentages can easily be >100%, but this suggests that on average, most enrolled patients had only one of these symptoms.

Which brings me to the next point.

The two groups were not equally sick

In terms of how sick these patients were, there are significant differences between Treatment and Control group as shown in Table 1 above. In general, the Treatment group had a higher incidence of fever, cough, diarrhea, muscle aches, etc. They appeared to be a bit sicker at baseline than the Controls. On average, they had 1.4 of the listed symptoms.

But look at the Control group: the summed percentage of the “flu-like symptoms” in the Control group is only 71%! Because that is below 100%, that suggests that several Control group patients did not have any of the listed COVID-19 symptoms. So why were they enrolled in the first place?

Table 2 (below) also appears to suggest that the Treatment group had more COVID-19-like clinical features than the Control group.

Of the 412 Treatment patients, 251 went in for a CT scan. That is about 61% of the patients. Of these, 70% had COVID-19 suggestive results.

In contrast, only 24% (54 among 224) of Control patients were ordered a CT scan by their doctor, and only 41% of these had a scan that suggested they had COVID-19.

These combined findings suggest that some people in the control group might have had an illness different than COVID-19. If the treatment was so successful, why were more CT scans performed in the Treatment group, and were more of these suggestive for COVID-19?

Something is not right here.

Unclear time intervals

There are some other inconsistencies/unclarities in the paper that need to be fixed.

• The Methods on page 7 say that patients were enrolled if they had 2 or more days of COVID-19 symptoms, while on page 8 it is stated that they needed to have had those symptoms for over 3 days (so presumably 4 or more).
• In the Methods it is stated that patients were followed for 14 days after symptom onset (by telemedicine). However, the Results state that average follow up was 5.0 days. It is not clear how these two numbers match.

It seems important to do a longer follow up on these patients. Five days seems very short. If they were already sick for 2-3 days, and you follow them for only 5 days, who knows if they maybe got sicker.

Here is a very interesting finding: the earlier patients were treated, the lower the chance they ended up in the hospital. However, it is not stated in Table 1 how/if the 2 treatment groups differed in days-of-symptoms at baseline. This is a big omission. If the treatment group had a significantly shorter sick period than the control group, that could be an important, not disclosed, confounding factor. As of now, this important piece of data is missing.

Unclear hospitalization reasons and number of deaths

Another big omission from this draft is the reason for admission to the hospital. This study takes hospital admission as the endpoint, and shows that the admission rates are lower in the Treatment group, but it is not listed why patients were admitted.

In light of the lack of COVID-19 symptoms in the Control group, it is very important to know why they ended up in the hospital. Could they have had another disease?

In addition, the number of deaths among the enrolled patients is not clear. The manuscript notes that two patients in the Treatment group died of other reasons, i.e. acute coronary syndrome or metastatic cancer. There is no mention of any deaths in the Control group in the paper, but the authors apparently told online that 5 control patients had died. It is not clear why, and neither is the time between onset of illness and death provided.

Incorrect p-values

Another possible error in the paper was noted by Twitter user @houndcl (Lu Chen). Figure 1 shows the very small p-value of a chi-square test (misspelled) of p<0.0001, while the actual numbers appear to give a p=0.034.

Lu Chen also showed that the p-values listed for Figure 2 are not p<0.0001 over all groups and p<0.0001 comparing the first two groups, but 0.03 and 0.16, respectively (Fisher’s test).

And, as pointed out by Ricardo Parolin Schnekenberg on his blog Notes On Covid, the p-values reported in Table 1 of this paper also appear to be incorrect. All p-values in Table 1 listed next to the “flu-like symptoms” are p<0.0001, but if you actually calculate them, they are much less significant. For example, the Coryza (inflamed nasal mucosa) p-value is actually 0.1350 (two-tailed Chi-square test, no Yates correction).

Summary of problems

• It is unclear how many of the enrolled patients actually had COVID-19. No PCR test was performed, and on average, study subjects did not have a lot of the typical coronavirus-associated symptoms.
• The study was not randomized; patients could pick their group themselves
• Even though the 2 groups were similar in age, gender, etc., they differ in their symptoms. On average, the Treatment group had a higher percentage of COVID-19 symptoms and COVID-19 suggestive CT scans than the Control group. The Control group might have had a higher percentage of another disease that was not coronavirus related.
• Time intervals between onset of symptoms and enrollment, or between enrollment and endpoint measurement are not clear
• Reason for hospitalization or deaths are not disclosed. Since this is a senior population with underlying diseases, and because people could chose their own group, this should be provided.
• The reported p-values (measure of significance) do not match with the reported data. The significance is not as high as the authors claimed.
• A large additional problem with the study (see links below) was that the study got ethics approval after the patients had already been treated.

Additional problems and reading

Here is are some other critical evaluations of this study (will update list if needed):

A twitter thread by Gaetan Burgio @GaetanBurgio , done earlier today: https://twitter.com/GaetanBurgio/status/1251476181989208066

Uma aula de como não se deve testar um medicamento – Natalia Pasternak and Carlos Orsi – Revista Questao de Ciencia (in Portugese)

Crítica ao Estudo da Prevent Senior – Notes On Covid – Ricardo Parolin Schnekenberg