The Challenge of Forecasting Demand of Medical Resources and Supplies During a Pandemic: A Comparative Evaluation of Three Surge Calculators for COVID-19

Ever since the World Health Organization (WHO) declared the new coronavirus disease 2019 (COVID-19) as a pandemic, there has been a public health debate concerning medical resources and supplies including hospital beds, intensive care units (ICU), ventilators, and Protective Personal Equipment (PPE). Forecasting COVID-19 dissemination has played a key role in informing healthcare professionals and governments on how to manage overburdened healthcare systems. However, forecasting during the pandemic remained challenging and sometimes highly controversial. Here, we highlight this challenge by performing a comparative evaluation for the estimations obtained from three COVID-19 surge calculators under different social distancing approaches, taking Lebanon as a case study. Despite discrepancies in estimations, the three surge calculators used herein agree that there will be a relative shortage in the capacity of medical resources and a significant surge in PPE demand as the social distancing policy is removed. Our results underscore the importance of implementing containment interventions including social distancing in alleviating the demand for medical care during the COVID-19 pandemic in the absence of any medication or vaccine. It is said that ''All models are wrong, but some are useful'', in this paper we highlight that it is even more useful to employ several models.


Introduction
deaths, and the impact of social distancing 12 Unfortunately, forecasting what is most likely to occur 75 in the upcoming weeks during the COVID-19 pandemic is not available for all countries including 76 some European countries or few states in the USA 8 . Where available, different models are 77 providing widely varying numbers of needed medical resources and/or supplies which often lead 78 to an incorrect distribution of what is available due to inconsistency in numbers. For example, the 79 Centers for Disease Control and Prevention has reported that COVID-19 outbreaks in parts of the 80 USA have resulted in surges in hospitalizations and ICU patients 13 . However, providing accurate 81 predictions of the healthcare system capacity peak demand is controversial due to the scarcity 82 and/or unreliability of data in addition to challenges associated with forecasting the effects of the 83 rapid changes in mitigation policies 14 . So far, the efforts to accurately model any emerging 84 outbreak's trajectory for the upcoming days are limited due to variabilities in assumptions and 85 parameters including social distancing [14][15][16] . Accordingly, the use of a single forecasting model 86 may not precisely predict how the pandemic evolves 17 . 87 Regardless of all the challenges, COVID-19 has put forecasting at the top of global public 88 policymaking and developing effective preventive strategies 18,19 . Since there is no "gold standard" 89 for predicting thresholds, the reasonable evaluation of the outputs of various forecasting models 90 has remained an open question 20 . In this work, we highlight this challenge by comparing the 91 projected demand for medical resources and supplies from three surge calculators for  taking Lebanon as our case study. To this end, we adapted an available statistical model for 93 estimating the daily impact of COVID-19 on hospital services based on the COVID-19 Hospital 94 Impact Model for Epidemics (CHIME). The model was modified to incorporate longer projection 95 periods and different social distancing policies (%). We then compared the hospital beds, ICU 96 beds, and PPE demand over 200 days projected by the three surge calculators a) CHIME PPE 97 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020 Our estimations were based on the CHIME model that was initially developed by the Predictive

129
In this study, we have introduced some modifications to the CHIME application to make it 130 compatible with the input parameters available for the projections in Lebanon.

131
The modifications were twofold: 132 1. Automate the data projection process: 133 The application was updated allowing the user to enter the input parameters into an "Excel 134 workbook" and obtain the generated census compiled together in the same workbook. This 135 alleviates the data processing part where the user needs to obtain each projection separately. It also 136 enables faster simulations for different input parameters and most importantly different social 137 distancing measures (Table 2).

Allow projections over a longer period:
139 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020 The application initially allowed for a maximum 30-day projection period. Our contribution 140 enables for longer periods such as 200 days, as seen herein. This provides analysts and healthcare 141 workers with longer forecasts, thus giving them more time to prepare for periods of peak demand.

142
The CHIME projects the daily and cumulative number hospitalized, ICU, ventilated, and newly 143 admitted COVID-19 cases, with social distancing policy percentage being the variable.  Our projections were based on three social distancing policy scenarios (0%, 30%, and 50%) 152 based on the National Health Strategic Preparedness and Response Plan for COVID-19 153 pandemic lock-down management and exit strategy implemented by the Lebanese Government 6 . 154 We also simulated the strict social distancing policy (92%) based on the Wuhan-style 155 containment 25 . The latter was performed to assess how strict "lock-down" could help better 156 contain the COVID-19 outbreak and maintain the demand for hospital beds, ICU beds, were chosen as the per five-step re-opening plan implemented by the Lebanese Government 6 162 ( supplies to respond to the current COVID-19 pandemic. Also, the CHIME and COVID-19 ESFT 172 calculators are freely available online for use by governments, stakeholders, and healthcare 173 centers.

174
The CHIME PPE calculator 175 The CHIME PPE calculator was generated to work in parallel with CHIME-generated 176 projections. The calculator uses forecasted patient censuses to output daily and cumulative 177 projections for each type of PPE (including N95 masks, surgical masks, gloves, gown, and eye 178 disposable protection) quantities per day, and computes the cumulative PPE predictions 26 . This 179 tool also permits users to input their custom scenarios (standard, crisis, contingency, and 180 custom), tailored to the specific situations relevant to their hospital or healthcare system 26 . In 181 our study, we chose the values for the standard scenario assuming we do not have exact publicly 182 available data estimates of the exact number of staff and HCW (healthcare workers) in Lebanon.

183
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020 debated in the intervention strategies, our estimations were based on the four social distancing 185 policy scenarios mentioned above ( ICU beds occupied by the critical and severe COVID-19 cases per week and not day. Therefore, 205 the estimated peak for inpatient and ICU beds will be the same throughout the chosen period.

206
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020 Although the tool is suited for projections over a short period (12 weeks), it offers an option to 207 enter data manually and make projections over longer periods. COVID-19-ESFT does not 208 quantify or account for resources already available locally or those pending delivery. The 209 calculator projects the PPE quantity per person per day for inpatient care, cases in isolation, 210 screening, and laboratory, and the total daily costs (USD) of items over 28 weeks. Then, it adds 211 the total quantity for each per day. In this study, we use the default input parameters set by the 212 WHO COVID-19 ESFT calculator for Lebanon, including the population estimate, patients case 213 sensitivity, healthcare workers, and staff. To maintain consistency across the different 214 calculators, we manually input the cumulative projected COVID-19 cases as obtained from the 215 CHIME model application. The data is compiled in a weekly form (up to 28 weeks) to be 216 compatible with the COVID-19 ESTF calculator. This approach guarantees that all calculators 217 are using the same numbers for infections and the focus would be on the 218 discrepancies/agreements between the models on the estimates of resource demand.

219
The COVID-19 AUBMC surge needs calculator 220 The AUBMC calculator (https://www.aub.edu.lb/fm/vmp/Pages/calculators.aspx) was developed 221 by our team based on AUBMC and MoPH data and is implemented as an excel file that predicts 222 the average total of PPE needs (e.g., gloves, surgical face masks, face shields, and N95 masks)  (Table 3). CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint

243
Based on MoPH data, from March 29, 2020, Lebanon is estimated to have 2308 ICU beds and 244 11794 inpatient (hospital) beds. We assume that 32 % of ICU beds and 20% of hospital beds at 245 the national level are currently occupied by non-COVID-19 patients (Table 4). However, this 246 occupancy number is, in reality, higher due to the casualties from the devastating port explosion 247 in Beirut on August 4, 2020. Also, there has been a surge in cases nationally. One reason behind 248 this surge is that many people have been unable to follow precautionary measures, such as social 249 distancing, during the relief efforts in Beirut. Also, some of the major hospitals have been  In one scenario and based on our data, assuming strict social distancing policy or the Wuhan-255 style (92%), the modified CHIME model estimates that out of the remaining 9435 hospital beds, 256 a peak of 67 inpatient beds is needed by the COVID-19 cases or 0.7 % of the available hospital 257 beds. Of the remaining available 1569 ICU beds and 687 ventilators, the model estimates a peak 258 of 15 (1%) ICU beds, and 10 ventilators (1.5%) are needed by COVID-19 patients respectively, 259 with a decreasing pattern in the estimated COVID-19 infected patients over the projected period.

260
Assuming the same scenario, the COVID-19 ESFT calculator estimates a weekly peak of 11 261 inpatient beds (0.1%), 17 ICU beds (1.1%), and 9 ventilators (1.3%) are occupied. The COVID-262 19 AUBMC calculator estimates a different peak of 71 inpatient beds (0.8%), 18 ICU beds 263 (1.1%), and 11 ventilators (1.6 %) (Figs. 1.A, 2, and 3). Upon relaxing social distancing 264 measures to 50 %, the CHIME model estimates the same peak capacity for the available inpatient 265 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020.  (Fig. 4B). The projected 309 average daily demand for all PPE types by the three surge calculators was still revealing a spike 310 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020.   (Fig.4D). 323 We then compared the change in the forecasted PPE fold-demand from other scenarios with the 324 Wuhan Style scenario (92%) since we have the least contact rate, minimum COVID-19 cases, 325 and minimal projected hospital utilization and PPE demands. So, when we changed social policy 326 to 50%, the CHIME estimated an average of a 17-fold increase in the daily demand for all the 327 PPE types. However, the COVID-19 AUBMC and ESFT reveal a minimum of a 1-and 4-fold 328 increase in the average PPE demand per day. This difference in PPE demand projections 329 becomes more apparent as social distancing policy decreases to 30% where the ESFT and the 330 CHIME estimate up to 46 and 100-fold increase in the average daily demand for all PPE types 331 respectively. Although the COVID-19 AUBMC calculator projected a substantial increase in the 332 average daily demand for all PPE types, this pattern in fold increase was not observed. The

333
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint calculator, however, estimates only a 4-fold increase in the average daily demand for all PPE 334 types. The difference in estimations was more obvious with the relaxed social distancing 335 scenario (0%). While the CHIME estimates a significant increase in the average daily demand to 336 3000 folds for all PPE types, the COVID-19 AUBMC and ESFT forecast of around 100-and 337 200-fold increase in the average daily demand for all PPE types respectively (Fig. 5).

339
As the first wave of the COVID-19 pandemic sweeps across the Middle East and the world, with 340 evidence of a second wave emerging in some countries, the strain placed on the health system 341 services including hospital beds, ICU beds, and ventilators continues to escalate. To support the 342 preparedness for a pandemic outbreak, the capability to forecast the potential spread of a disease 343 is an utmost need for applying public health interventions and effectively allocating resources 29 .

344
Particularly, this is critical for low-and middle-income countries, as they often unevenly bear the CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review) preprint
The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint the USA and Europe during the COVID-19 pandemic and proposed some measures to increase 381 the supply of key products and services 8,9 . Interestingly, our data indicate that the output 382 predictions of all used surge calculators vary very widely upon relaxing the social distancing 383 policy measures leading to a rise in the margins of agreement.

384
The discrepancy in the obtained results could be related to several reasons. For PPE, some of the 385 items are subdivided into different categories such as the types of gloves, gowns, and masks used 386 by patients and staff as in the WHO COVID-19 ESFT calculator. Also, the ESFT calculator 387 gives a more detailed quantification for PPE, for example, the estimation for gloves refers to the 388 sum of those used for surgery, examination, and heavy duties. In the CHIME and COVID-19 389 AUBMC calculators, some of these categories are missing, which indicates that the PPE types 390 are not being included in the calculated demand. In the COVID-19 AUBMC calculator, the N95 391 masks are considered to be used only in intubations. However, the ESFT calculator does not 392 even give estimates for N95 masks. Moreover, the CHIME and ESFT calculators include PPE 393 needs for testing staff and HCW, which adds to the demand in its estimations. Considering the 394 need for PPE in healthcare departments related to non-COVID-19 patients and used for 395 additional precaution could also increase the calculated demand. Varying results across the three 396 calculators are also evident in the inpatient beds, ICU beds, and ventilators needed during the 397 peak demand period. This can be related to the way each calculator assesses the severity of the 398 cases. According to the COVID-19 AUBMC calculator, each ICU patient is assumed to require 399 ventilation. However, the WHO ESFT calculator assumes that only severe and critical COVID-400 19 cases are admitted to the hospital and assumes that only the critical cases in ICU require 401 ventilation, while the rest need oxygen tubes. As for the CHIME, it classifies the hospital 402 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review) preprint
The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020 admitted COVID-19 into three categories including hospitalized, ICU, and ventilated. Since 403 there is no absolute truth that we can compare to assess the accuracy, we only resort to the from the modified CHIME model application into all calculators to maintain consistency. Yet, in 413 addition to using already built models, we still lack accuracy as we do not have tangible data in 414 Lebanon on PPE consumption and the capacity of healthcare system infrastructure. Also, we are 415 not very sure how much our input data are up-to-date and reliable especially that the healthcare 416 capacity is subjected to change throughout an outbreak 29 .

417
The results of this study were based on showing the differences in estimations done by three 418 surge calculators for COVID-19 and a modified version of the CHIME model to measure needs 419 for different health system resources based on the total of predicted simultaneously active cases.

420
These calculations were carried out based on three important assets receiving significant 421 attention worldwide: inpatient beds, ICU beds,ventilators,and PPE 35,36 . Other items and human 422 resources required in the diagnostic and treatment chain can be forecasted including staff and 423 HCW in the frontline of COVID-19 response, and therapeutics for supportive treatment. We 424 believe that one of the major limitations for forecasting COVID-19 is based on the limited 425 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint evidence since neither the magnitude nor duration of the COVID-19 wave is known with 426 certainty. Another limitation is that we did not factor in PPE re-use measures such as sterilization 427 of used N95 masks. Therefore, the adequate management of medical resources (including PPE, 428 beds, ventilators, and health care providers) is highly recommended at this stage as some 429 countries have started to experience a resurgence of COVID-19 cases as the pandemic continues 430 to accelerate. Experience from several countries including China, South Korea, and Singapore in 431 addition to mathematical modeling has revealed that the pandemic can be contained even in the 432 exponential growth phase using a combination of interventions 34,37 . The latter mainly includes 433 social and physical distancing, public awareness, and wearing masks 34 .

434
In conclusion, the surge calculators used here, regardless of the variability in outputs, can be and averaging 29 . Therefore, the use of more than one model is recommended to generate more 440 accurate and better predictions of the pandemic's evolution 38 . In other words, policymakers can 441 use these calculators interconnected with each other based on the available data for each country 442 to understand the pandemic from all its angles to be able to generate policymaking frameworks.

443
This urges the need for a clear methodology that allows policymakers to decide which model is 444 more applicable or adaptable for their context 17 , and underscores the necessity of enabling 445 calculators to be adopted to local policies and behaviors beyond social distancing. Although gaps 446 in the present data streams provide a challenge for the current epidemic forecasting, recent 447 breakthroughs in this field afford the possibility for refining future predictive models 29 . Since 448 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. for the USA and European Economic Area countries. medRxiv (2020). is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint 19. Perc, M., Gorišek Miksić, N., Slavinec, M. & Stožer, A. Forecasting COVID-19. Front. 524 Phys. 8, 127 (2020. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020.  . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint

D. 0%
COVID-19 ESFT COVID-19 AUBMC CHIME . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint  . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10.1101/2020.09.29.20204172 doi: medRxiv preprint  . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted September 30, 2020. . https://doi.org/10. 1101/2020