Behind the Collapse of Xi'an Covid-19 App

January 4 was the deadline for the municipal government in Xi’an, the ancient capital of China with a population of 12 million, to reach its goal of zero new daily Covid-19 case. On January 1, there were 122 new infections; On January 2, the daily count fell to 90; On January 3, the figure was 95.

The new cases in 24 hours were not many compared with thousands in many cosmopolitan regions in North America during the ongoing fifth wave of the coronavirus pandemic since early 2000. But with the zero-case policy in place across China, the doulbe-digit new coronavirus infections have made the outbreak in Xi’an the second catastrophic heathcare event in two years, only after that in Wuhan, the once epicenter of the pandemic.

While many residents in the city rushed to the Covid-19 testing centers to find out if they were infected, their app that stores personal travel history and Covid-19 test results stopped working.

In China, everyone must show their travel records in the past 14 days in order to get on a bus, shop in a store or depart a city. If they visit an area with a single Covid-19 infection, the status of their app is showed in the color of yellow or red, which means they should quarantine for two weeks even if they have no symptoms at all.

Many residents affected by the app crash complained online that they could not even leave home to get tested. Let alone the government’s zero new Covid-19 case goal by January 4.

Every resident in high-risk neighbourhoods of Xi"an is required to get tested for Covid-19 every two days thanks to China’s enormous manufacturing capacity and the centralized administrative system. Many doctors and nurses from other cities were dispatched to the city to help.

Amid the fury of the public, it took about two hours for engineers to frantically solve the problem. Actually it was not the first app crash. On December 20, a similar breakdown occurred during the current round of outbreaks in Xi’an that started on December 9, when the first local case, as opposed to infections of overseas travellers to China, was reported. Last time, it took about 24 hours for the app to resume working following hectic repairs.

Rumors quickly circulated online that the municipal government spent only 270,000 yuan (about US$38,600) to build such a vital administrative system. A snapshot of a document showing the government’s invitation for offers from suppliers surfaced.

After a probe in the snapshot by TMTPost, the snapshot turned out to be the document for a different government procurement project.

The app system was developed in three days, according to a news story published in May 2020. However, amid the outcry over the crash, Xi"an Telecom said three days were just the time for developing a prototype, not the final version of the product. 

Based on the information accessed by TMTPost, the procurement project of the app named “One Barcode to Anywhere”, which was designed to curb the spread of the coronavius, was won by China Telecom’s subsidiary Xi’an Telecom. On February 18, 2020, about two months after the outbreak in Wuhan, the app was launched for its tral operations.

Xi’an Telecom divided the project into various parts, which were subcontracted by nearly 10 information technology companies. The project consists of development, operation and maintenance, Internet security-related products and services, server products, text message services and big data among many others.          

The app is built on another project named “Xi’an Digital Administrative Platform”, known as the “Administrative Cloud”. According to the information available to TMTPost，the storage of data, internet and computation services used by the app are supplied by the platform. The services bought by Xi’an Telecom are generally called “application services.”  

According to public information released by government agencies, Alibaba Cloud is the supplier of Administrative Cloud and text message services; Meilin is responsible for back-office algorithm; Xi’an Dongruan System Integration Co. Ltd. supplies software and maintenance services.

Meilin Data was blamed by some internet users for the crash on December 20. The company responded by saying that it provided algorithm, not operational services.

Dongruan were involved in platform software and customized development services. The platform was originally not designed for displaying the results of mass Covid-19 antigen tests on the app. After the first breakdown, the capacity was expanded to 40,000 QPS (Queries Per Second). However, it was still not enough when all 12 million residents were tested within two days.    

Liu Jun, the head of Xi’an Big Data Resources Administration, apologized for the December 20 breakdown and cautioned residents not to use the app unnecessarily.

His advisory provoked a wave of online mocks. After all, who would use the app when not needed? One netizen said sarcastically: “Don’t show your barcode when you don’t have to; But when you have to, the barcode is dead.”

Liu also gave the reason for the systematic glitch. Too many users were using it at the same time, he said.

As to the cause of the January 4 breakdown, the authorities attributed to “a sudden surge in the number of users”, which was ten times as high as that in the pre-December 9 period. The unusually high volume of visitors led to “online overcrowding,” which triggered the firewall to block information packages. The core issue is that the architecture is not suited for a large number of simultaneous visits.

On January 5, Liu was publicly reproached by Xi’an Municipal Committee of the Communist Party of China in the wake of two breakdowns in two weeks. He was suspended from his position and ordered to hand in a self-criticism report.

According to public information, the “One Barcode to Anywhere” app amassed downloads of over 30 million by over 200,000 stores and companies as well as 50,000 operators and users of the subway system, buses and taxies. The app has been used for a total of 3 billion times in 43,000 neighborhoods. The app, along with big data, has been applied to conduct contact tracing of nearly 400,000 persons and for over 400,000 Covid-19 tests or vaccinations per day.  

Some netizens pointed fingers at Alibaba Cloud after the two crashes. Alibaba Cloud appeared not only on the list of suppliers for the “One Barcode to Everywhere” app, but also on the list of suppliers for “Administrative Cloud.”  

A source close to the procurement project told TMTPost that Alibaba Cloud made a mistake in the analysis of DNS, or Domain Name Service. There are two VIP routes reserved for the app but one error occurred on one of the two routes. RDS, or Relational Database Service, contains many slow SQLs, which caused traffic congestion. The two problems were quickly solved in the wake of the breakdown.  

However, a spokesman of Alibaba Cloud denied the existence of two issues. “Alibaba Cloud provided underlying structure to the “One Barcode to Everywhere” app, but was not involved in the creation of architecture. During the Covid-19 outbreak in Xi’an, Alibaba Cloud platform has been running smoothly. There were no glitches in either DNS analysis or RDS. Our technical team has been on the spot. Alibaba Cloud is willing to make more contributions to the fight against the novel coronavirus,” he said.

An expert in cybersecurity and information technology told TMTPost on the condition of anonymity that traffic congestion triggered a blockage on the firewall and consequently failures of requests for the data. Therefore the personal information of users could be matched with the information on test results or travel history in the cloud, resulting in the failure of information display on a user’s app. Ultimately it was a problem of archteicture, not the firewall, he added.

On January 4, the city government failed to reach its goal. As of the midnight of January 10, Xi’an reported 15 new daily infections. On December 23, 2021, the ancient capital of China was locked down following an outbreak of the Delta variant－related infections in early December.

关键词： English