Air cleaning technologies and the battle against Randomised Control Trials

RCTs are not a 'gold standard' methodology where we can actually measure outcomes

A recent paper in JAMA reported the conclusion of a large scale RCT trial on the use of portable HEPA filters in care homes. The study found little significant evidence that they helped reduce infections, and although there have been numerous criticisms of the study, most of these have focused on the lack of rigour in ensuring adequate effective air change rates and the rates of compliance within the intervention arm.

While these are valid criticisms of the study, and do largely render the results meaningless, a more fundamental critique of the study design is that RCTs are the wrong method for assessing a physical intervention such as HEPA filtration.

RCTs have to be used where outcomes are not directly measurable, like vaccine or drug trials, and where the intervention is predetermined and constant (eg a patient receives a vaccine, and that vaccine stays with the patient throughout the study). Where RCTs fail for measures like filtration is that even if all the technical questions are adequately addressed, because we are looking at treatment of a physical space, rather than an individual, unless the study participants stay within their respective control/intervention locations for the entirety of the study period, the results become largely meaningless. For example, in the JAMA study, we have no idea if residents left the care homes, staff could have become sick at home, and there was no way of understanding what the impact of the HEPA filters was on pathogen concentrations where they were used.

The alternative approach, and much better option, is to simply measure how effective the intervention has been. In the case of HEPA filters, this would involve measuring the pathogen load with and without HEPA filters running.

Tackling the RCT obsession is a long haul, but I submitted the following comment to JAMA, which has now been printed alongside the original article.

Dear Editor,

It is deeply disappointing to see another major study on air cleaning technologies succumb to the inappropriate application of the randomised control methodology.

Far from the ‘gold standard’ that some assume, RCTs are an unfortunate necessity for situations where we cannot directly measure physical or biological outcomes, such as drug effects inside the human body. Where we have physical phenomena that can be directly measured, such as viral persistence in air, RCTs are unnecessary and can be misleading. We don’t rely on RCTs to justify whether we need to purify drinking water; we measure pathogen and pollutant loads to check whether the purification process is functioning correctly.

We have solid evidence that HEPA filters can effectively capture viral particles. In laboratory studies HEPA filtration at an effective rate of 2 air changes per hour (ACH) reduced measured SARS-CoV-2 load by 96.03% (1). The question is whether we can replicate this ACH rate effectively in a real world situation. This should be assessed via physical measurement, not RCTs.

In the current study, if staff and residents were free to move in and out of the study areas, the entire RCT methodology breaks down. Unless the participants were restricted throughout the full duration of the study to the intervention and control settings, it becomes impossible to identify where infections actually occurred, and the results therefore become effectively meaningless. You could, in theory, construct a logically valid RCT design. This would require strictly restricting participants movements exclusively to intervention/control areas, within functionally identical buildings, and then physically introducing equivalent concentrations of pathogens into the study areas, in order to derive any indications of the efficacy of the HEPA filters. However, such an RCT approach becomes an impractical and unethical construction.

It remains potentially the case that it may prove impossible to replicate the laboratory efficacy of HEPA filters in real world situations such as care homes, due to practical issues such as noise, ACH rates, and human control, but that cannot be assessed via RCTs. We can, however, measure pathogen load under HEPA/no HEPA conditions, and that’s the data that needs to be gathered.

Beyond HEPA we are witnessing remarkable technical advances in the arena of air cleaning technologies (2). RCTs will never be an appropriate protocol to measure the efficacy of such physical devices, given the restrictions required to make such studies logically valid. The alternative is the simple approach; install these devices, switch them on, and then measure the pathogen concentrations. As we do with water.

1) https://journals.asm.org/doi/10.1128/msphere.00086-22
2) https://www.spiedigitallibrary.org/conference-proceedings-of-spie/13890/3080065/Aluminum-nitride-integrated-photonics-for-far-UVC-second-harmonic-generation/10.1117/12.3080065.full