As the Covid-19 pandemic rages around the world, we should not let our desire for clear hard numbers blind us for the pitfalls of public health statistics. The havoc caused by SARS-CoV-2 is increasing every day since its terror began in Wuhan, China. Understandably, the public wants something to chart this novel coronavirus’ terrifying progress around the globe and to hold to account their leaders. Several massive failures compare abysmally to some success stories. Numbers help quantify that difference and give a feeling that you can grasp what is going on. Sadly, things are not so simply. Countries’ numbers of Covid-19 cases and fatalities are a tempting source of clarity. However, the raw numbers are false prophets. You should not compare country numbers directly. Journalists who continue to do so are misleading their audience.
No number in the complex field of public governance will ever be 100% accurate. Nevertheless, if numbers are inaccurate in similar ways, they can still be usefully compared. However, the Covid-19 numbers produced by different countries are inaccurate in such different and unique ways that no meaningful direct comparison is possible. There are five reasons why.
The first and most important reason is differing access to tests. When the first people became sick with Covid-19, there were no tests. Even once the SARS-CoV-2 virus was isolated, it still took time to develop reliable tests and criteria for diagnosis. Medical staff then need to get access to these tests in their place of work and need to learn how to use them. You need a way to diagnose someone before you can even count them. A lot of poorer countries have very low numbers of Covid-19 because they simply do not have any tests.
Once they are available, it matters who gets tested. The Netherlands does not test every single suspected case, making one Dutch health official argue that when the country had 1,000 confirmed cases the real number was around 6,000. South Korea’s aggressive test campaign leads to much higher confirmed numbers. Early on Singapore tested all pneumonia patients in its hospitals as part of a wide sweep that made it an early outlier in the tables. Some people worry that Japan has such low numbers because it simply does not test enough. Germany points to its high testing rate to explain its low number of deaths. After initially not testing many people, the UK has only begun ramping up testing recently. There is a lot of variety between countries. But regardless of the policies that determine who gets tested, a country’s health care system also determines if they are able or willing to. A widely-shared report accounted how a Miami resident was faced with a US$3,200 bill after voluntarily asking for a test. Given the prohibitively high costs of medical care for many in the US, a lot of Americans will avoid looking for help and thus never show up in the statistics. In general, if you do not test people, you will not have many confirmed cases. That obviously does not mean there aren’t any.
Lastly, when the system is completely overwhelmed—as happened in and around Wuhan and might now be happening in Bergamo and New York City—many people simply cannot be tested, even if everybody involved would want them to be. In Hubei, people most likely suffering from Covid-19 were unable to be admitted to the hospital as the health care system could no longer cope the sheer volume of patients. Many sufferers were never counted, even when put up in government-organised hotels and other isolation centres. If you do not test people, your numbers will not show them.
The second reason for the incommensurability of country data is the fact that criteria for diagnose differ between jurisdictions. You do not simply produce a confirmed case with a clear black-and-white Covid-19 test. It is not an automaton. Different public health authorities have different criteria for making a formal diagnosis. Tests that prove the presence of certain anti-bodies or RNA are part of that, but symptoms also play a role. Many of the tests, rushed to the market, are not completely reliable and so procedures have been developed that involve multiple tests as well as CT scans and observing the patient’s general condition. A diagnosis is the product of a set of rules. Those rules can and do differ between places.
China’s Hubei province caused consternation in February when it suddenly reported a one-off bump of 14,000 new cases. These numbers concerned patients who had the symptoms and CT scan results consistent with Covid-19 but had failed to test positive or even get tested at all. This big one-time move allowed them to access government help for Covid-19 patients anyway. It also shows how changing the criteria for a formal diagnosis can radically alter the number of confirmed cases, making one wonder what would have happened elsewhere. But this mass addition to China’s formal numbers has only happened once.
China also got people worried after recent revelations that more than 43,000 asymptomatic cases were not included among the roughly 80,000 confirmed Covid-19 cases reported by Beijing. Most other countries do seem to include these cases in their numbers. Using that approach, China’s number of officially confirmed cases would be closer to 120,000 than 80,000. In general, much is still unknown about the number and nature of asymptomatic cases.
When systems are overwhelmed, the ability to make a formal diagnosis is also affected. There is an incentive to make the criteria stricter to ensure that the limited available care only goes to the most seriously affected patients. The chaos of an overwhelmed medical system, when the medical staff gets tired and new outsiders are rushed into action without much training, further affects the reliability and consistency of how diagnoses are made.
The third reason why you should not compare the data is that rules for attributing deaths differ. Establishing the cause of death of a patient is challenging in many cases, especially when limited pathology is available in a stressful situation. It is understandable that priority is given to caring for those still alive. However, if no attempt is made at all at establish a cause of death, it will not show up in the statistics. Many of the people who passed away in their homes in Wuhan have never been tested and were cremated before a cause of death could be formally established. They do not show up in the statistics.
Moreover, even in the best of times, it can become an almost academic question what the real cause of death was when the deceased was weak and suffered from multiple illnesses. The death rate then becomes heavily affected by wha the rules happy to be for fitting a complex reality into simple numbers. There are many reasons why Italy has a high number of death. However, one of them is the fact that Italy counts anyone who dies while hospitalised with Covid-19 as a Covid-19 victim, even when other factors were also at play. Apparently, China does not do that, increasing suspicion that its reported number of victims is way too low. In some countries with a large private healthcare sector, cooperation with government agencies collating numbers may also be affected.
Moreover, as the health care system becomes overwhelmed, health care in general will suffer. People will die of causes unrelated to Covid-19 who would not have died if there had not been a pandemic. The Covid-19 statistics do not include these cases and it would be very hard to judge the pandemic’s role in other mortality.
The fourth reason why you have to be wary of data is that not all governments are honest. Some leaders prefer to keep the numbers down because a large amount of confirmed cases makes them look bad. US President Donald Trump initially did not want the cruise ship Grand Princess to dock because it would lead to more patients on the American ‘score card’. If he brought that instinct to other decisions related to the pandemic response, it could lead to a lower number of reported cases in a myriad of small ways.
More high-profile are the attempts by the Chinese party-state to keep the CCP’s political interests at the forefront. The reliability of Chinese statistics is always a thorny question. The fact that China does not include asymptomatic cases could be seen as proof of its reluctance to be transparent. But there are broader concerns about China’s numbers, especially now that President Xi Jinping has visited Wuhan and the Party propagandists have launched a triumphant narrative. ‘Speaking politics’ has always been in command in Wuhan, which of course began this whole pandemic by covering up cases. Kyodo cites an anonymous doctor claiming that Wuhan began manipulating case numbers starting with Xi’s inspection. RTHK also reports on Wuhan residents claiming they are refused testing now. We know that Wuhan continues to find asymptomatic cases, despite the official insistence on zero domestic transmissions.
Although initially Chinese journalists used the short window of relaxation to produce impressive works of investigation and reporting, the Party has now closed down that avenue and the propaganda lines have been set out. As the focus shifts to propagandising the narrative that China has saved the world—lying about the way the party-state’s cover-up allowed this pandemic to get underway in the first—place, it essential that the increasingly dire news from the rest of the world is not matched by bad domestic news. China’s numbers before Xi Jinping’s Wuhan trip had some issues, but we now live in the period after the trip, when the pressure each local official faces to keep them down must be immense. This cannot be good news for their reliability, in Wuhan or elsewhere in China.
The fifth reason why you should not compare countries’ Covid-19 numbers without caveat is that each country is different. Even converting the numbers to per capita will not be enough to account for this. Italy has relatively many elderly and that fact combined with the habit of multiple generations living together under one roof has been blamed for its higher death rate.
There is also the matter of a country’s political culture and the attitude of the people towards their government. In some places, people are more likely to turn to the state for help and report if they do not feel well. In other places, the state might be underdeveloped or people prefer to fend for themselves or have no choice to do so, and stay at home when unwell, never entering the formal health care system.
Within countries there are also differences. Shanghai has been held up within China as a standard for how to deal with the pandemic. The country’s various provinces are not all equally developed or capable. This will also show in their policies for tests and criteria for formal diagnoses. Shanghai and Guangdong may have been stricter than other provinces.
Of course, the Covid-19 numbers people are bandying around are not completely useless. Especially within countries they can still tell a story—accounting for major political shifts such as Xi Jinping’s Wuhan visit. If the inaccuracies are systemic rather than random, the resulting trend is still useful.
However, the different countries all face the pandemic in different ways and produce data in different ways. The neat tables and graphs presented by Johns Hopkins University and other aggregators simplify reality. This is misleading. The data they show in deceptively clear format is the product of countless small and big decisions by countless people at countless points in the massive human endeavour to stop this awful pandemic. Small differences at lower levels can aggregate to big difference in the final data. Each country’s data is the product of different testing regimens, different diagnostic criteria, and different degrees of transparency. People should now stop making direct comparisons.
Leave a Reply