Pandemic’s Ecological Fallacy

Data become studies.

Studies draw conclusions.

Conclusions create interpretations.

And interpretations lead to clinical decisions.

A process most would have hardly noticed pre-pandemic. But over the past year, we have all become ad hoc epidemiologists extrapolating our own conclusions on the clinical studies that formed the foundation for healthcare policies the states used to enact social distancing guidelines and mask mandates.

The heightened scrutiny of the data and the attention placed on the data ushered in an unprecedented level of pressure to report data and publish studies.

The faster we needed data, the more we abbreviated the clinical studies. The more abbreviated the studies, the greater the error – a process that highlights a fundamental limitation in the clinical study design process. When the fluency or the rate of progress increases, the basic checks and balances in the studies deteriorate.

We get faster results, but the quality of the results is lower. When the quality of the results are lower, there is greater error in extrapolating conclusions from those studies. Something the journal, Nature, has been covering extensively.

They found that more than 4% of the articles listed in the Dimensions database and around 6% of those listed in the PubMed database were dedicated to COVID-19 in 2020. Dimensions and PubMed are the two most referenced databases for clinical studies across the world.

Perhaps more alarmingly, the pandemic also saw a sharp rise in preprints (articles posted online before peer review). According to Nature, more than 30,000 of the COVID-19 articles published in 2020 were preprints — somewhere between 17% and 30% of the total COVID-19 research papers. And more than half of the preprints appeared on one of three sites — medRxiv, SSRN and Research Square. These are distinctly different from the more commonly referenced sites that have greater oversight in the quality of the publications listed.

Unsurprisingly, this has led to an increase in the number of retractions of clinical studies as well. Typically, it would take three years to retract a paper, but during COVID-19, it has taken just months — another effect of the increased fluency on clinical study development.

The rapid appearance and disappearance has a destructive effect on public policy, the least of which is the damaged credibility. Public policy experts and elected officials make decisions based upon this data, and the quality of their decisions is based upon the quality of the data.

So when we pump out publications with questionable data, we get questionable decision-making.

The most cited COVID-19 study is a publication that studied 41 patients at a hospital in Wuhan, China. The most cited preprint was a study on social distancing measures, which had a major impact on public policy in Britain. The latter attracting the most attention on social media, according to internet monitoring firm, Altmetric.

While it is too early to generalize, we see a clear disparity in the quality of the articles focusing on public policy and the articles focusing on other aspects of COVID-19. Articles that focus on public policy, or issues that affect the public, tend to be published more quickly and present with lower quality of data.

Recently, multiple European countries have discontinued use of the Oxford/AstraZeneca vaccine due to the risk of blood clot among those receiving the vaccine. According to some reports, only five (5) in thirty (30) million developed blood clots. And earlier studies that evaluated the safety of this vaccine found no risk of blood clots.

“There is currently no indication that [Oxford/AstraZeneca] vaccination has caused these conditions, which are not listed as side effects with this vaccine,” a statement issued by European Medicines Agency (EMA) read. “The vaccine’s benefits continue to outweigh its risks and the vaccine can continue to be administered while investigation of cases of thromboembolic [blood clots] events is ongoing,” it added.

Yet countries nevertheless have stopped administering the vaccine, based upon the anecdotes of select individuals. Preventing millions of high risk individuals from receiving the vaccine throughout Europe just as the continent is immersed in another wave of COVID-19 infections.

When anecdotal evidence trumps statistically significant findings from regulated clinical studies, we find decisions are made more as impromptu reactions than as comprehensive evaluations of the data.

But when the quality of the data is already perceived to be suspect, even robustly designed studies lack credibility. Soon the all data points from all sources are given equal consideration, largely because most of the public, even many health policy experts, lack the nuanced understanding of clinical study design –how to evaluate the quality of the data from the study design, a critical skill academic physicians devote much of their career towards.

This becomes a problem when trying to decipher the results of different clinical studies. Even the clinical studies used to evaluate the many vaccines ranged widely across the world. Some studies used relatively few clinical subjects or had limited endpoints at which the studies drew their conclusions. Some only included subjects with mild symptoms while others included a wide range of clinical presentations. The variability in the study designs led to variable confidence in the studies – rightly or wrongly.

We saw how the Pfizer and Moderna vaccines had different results compared to the Johnson and Johnson vaccine. Much of that was due to the study design. And the Indian and Chinese vaccines had even greater variations in the study design as many of these studies used fewer subjects compared to the Western study counterparts.

What would have been helpful is a uniform approach to modifying study design that expedites the research for each vaccine in a more consistent manner. Adjust the studies, but adjust them similarly.

Rather than allow for variations in the fundamental study design, we should develop novel study designs that can be used in times of heightened fluency of publications.

The pandemic showed that there will be times when clinical studies need to expedite their process. Rather than implementing the same clinical study design models and modifying them in haphazard ways, we should implement adaptive techniques that increase the fluency of clinical study design while maintaining the quality of data needed to optimize the extrapolation of key interpretations from that data.

We must optimize the ecological fallacy of data – the extrapolation of data to individuals. Something we fell well short of during the pandemic, leading to moments of spectacular failure.

In the early days of the pandemic, we needed to understand whether masks were truly effective, and we needed to understand this over the course of weeks, not years. That added fluency led to a slew of half-baked clinical studies that were more or less glorified narratives about the value of masks, which used numbers as fillers to substantiate a preexisting belief about masks held by the authors of the studies.

As more such narrative-driven studies came about, the overall credibility of all studies – even well designed studies – fell in the eyes of the public. But more importantly, the public policy experts making the decisions failed to discern the differing quality of the data when citing studies to make public policy decisions – leading to decisions made more out of politics than science, and revisions that prompted reactions of disbelief and disillusionment.

Health should develop rapid clinical studies designs that can be completed in a matter of weeks instead of months. Yet retain the fidelity of the data needed to make medically appropriate decisions.

The very concept of clinical research, how we obtain new information or systematize existing information into clinical decision-making, should change post-pandemic, become more adaptable.

When the fluency of research increases, it should not come at the cost of the quality of data. There is a balance, and it is to be found in appropriately designed clinical studies that account for the heightened fluency of research.

Now that would make for a great clinical study.

Message Board

Leave a Reply

Your email address will not be published. Required fields are marked *


Vaccine Passports: what we need to know

The COVID-19 pandemic has killed more than half a million people in the US and has seriously impacted our daily lives. The granting of Emergency Use Authorizations for COVID-19 vaccines has been a game changer in helping to reverse the pandemic onslaught. Demand for the vaccines currently far exceeds supply nationwide. They have become the golden ticket that can transport us back to a time when routine activities, such as attending school or taking vacations, didn’t seem fraught with danger. What’s the best way to go about re-establishing these activities? One idea that’s getting discussed seriously is giving those who have been immunized a vaccination passport (VP).

What is a vaccination passport?

Let’s start by clarifying some terminology. VPs are documents that show that someone has been given a vaccine(s) and is therefore presumed to be immune from getting and sharing that disease. These are distinct from diagnostic tests to determine if a person is or is not infected with a particular virus (PCR and viral antigen tests) or has been exposed to a virus (antibody tests). VPs are the modern day equivalents of the “letters of transit” that played a key role in the film Casablanca. They’ll enable you to travel freely.

Why give vaccine passports?

Vaccinated people are unlikely to transmit the virus that causes COVID-19 with others, though this has yet to be definitely proven. So, what’s the primary rationale for giving vaccination passports? It seems the goal is to incentivize people to get vaccinated. Folks might be more likely to seek out the vaccine if it bestows upon them certain privileges, such as being able to travel freely. While the idea might sound appealing at one level, the road to vaccine passports is pockmarked with political potholes and littered with logistical land mines.

Administrative issues

There are a lot of important questions that need to be answered before VPs can become widespread; it’s unlikely there will be a “one size fits all” solution. What might VPs look like? Pieces of paper (that could easily be lost or counterfeited), or a digital key or document you store on your smartphone (that many older folks still don’t have)? If you lost one, how would you get it replaced? Would your vaccination records be tied to your other medical records, and if so, what kind of privacy and security protections would be needed to safeguard your data? How would the passport administrator verify that you got vaccinated in the first place? Who is responsible for correcting any errors that crop up? Given how much we’ve struggled as a nation to simply solve the scheduling of vaccinations, VPs might be significantly more difficult to manage.

Who’s going to run the program?

A coalition of health tech leaders (including Epic, the Mayo Clinic, Microsoft and Salesforce) has started the Vaccination Credential Initiative to create an internationally accepted digital health card. Its vision – as stated on its website – is “to empower individuals to obtain an encrypted digital copy of their immunization credentials to store in a digital wallet of their choice. Those without smartphones could receive paper printed with QR codes containing World Wide Web Consortium (W3C) verifiable credentials.” Some European countries (e.g. Denmark) are working on developing their own immunization passports, and the President of the European Union Commission has voiced support for them.

We already have a hodgepodge system designed to keep folks with COVID-19 from traveling. Many airlines are requiring proof of non-infectivity to board a plane – these are distinct from VPs. The methods adopted vary from airline to airline: United uses Travel Ready Center, American uses VeriFly while others use Common Pass. Saga cruises are the first in the UK to demand proof of vaccination, but not everyone is on board with the concept. The World Travel and Tourism Council stated its opposition to allowing travel only by those who have been vaccinated, on the grounds that it is discriminatory.

Discrimination concerns

The concept of gaining advantage from one’s immunological status is not a new one. In New Orleans in the mid 19th century, great economic value was attached to those who survived the onslaught of epidemic waves of yellow fever. A mix of structural racism and a system valuing “immunocapital” ranked men and women highly if they were yellow fever survivors. Those that had not had the disease yet found it difficult to obtain work or obtain credit and women could not marry. Slaves were similarly revalued, with larger assessments attached to survivors. This legacy of viral discrimination casts a long shadow over current discussions surrounding vaccination passports.

Also problematic is the fact that not everyone will be eligible for VPs. Some people can’t be vaccinated for medical reasons but would still like to participate in whatever activities the passports enable (such as travel opportunities). These individuals will likely file complaints if they are denied VPs, arguing that they’re discriminatory in nature. Vaccine opponents will no doubt also file lawsuits seeking to block the use of VPs, claiming they represent an attack on their freedoms and personal choice.

Others may be shut out of VP programs because they don’t have the computer skills to register online, a problem that’s turned out to be widespread with the vaccinations. Don’t have any ID? In some places this prevents the most vulnerable from getting the COVID-19 vaccine, and therefore VPs as well. Many Black and marginalized communities are already deeply suspicious of the medical establishment, and wary of receiving a COVID-19 vaccine. Restrictions that prevent these groups from participating in various societal activities because they don’t have VPs will echo the Jim Crow days of poll taxes and literacy tests that were required to vote.

Enforcement issues

VPs bring up a similar issue as mask mandates: who is going to enforce them? It is one thing for a cruise line to do so, since reservations are required and extensive paperwork needs to be filled out in advance. Who is going to keep non-vaccinated people out of restaurants or concerts that require proof of vaccination? We’ve already seen anti-maskers storm department stores, ignore requirements to wear masks on planes and even kill a security guard who confronted them.

Black market for vaccine passports

Many airlines are now requiring proof of a negative COVID-19 test before they will let their customers fly. This has led to a black market for fake negative COVID-19 test results for those who are looking to game the system. Similarly, bogus cards claiming that the cardholder was exempt– for medical reasons – from having to wear a mask, have been manufactured and distributed. It’s not hard to imagine that a similar market will emerge for VPs.

Overcoming vaccine hesitancy

Given the multitude of issues and concerns listed above, moving forward with issuing vaccination passports seems highly problematic. If the primary purpose of having VPs is to drive up immunization rates, that can be better accomplished by running public service announcements and ads illustrating how liberating it is to start doing normal activities once vaccinated. A nationwide campaign to do just that is already in the works. I personally can think of a better use for the time and money that will be spent on establishing VPs: use them to focus efforts on overcoming vaccine hesitancy in concert with combatting the high tide of online vaccine misinformation and disinformation.

Source: Technology Networks

News Briefs


The African origins of the small pox vaccine

In one of history’s most fascinating twists, the Africans who were forcibly transported across the Atlantic helped the nascent United States develop early versions of the small pox vaccine.

Twitter Handle

Copyright © 2020 I Daily Remedy
Powered by: YES IT Labs LLC