Advertisement

eHealth for the prevention of healthcare-associated infections: a scoping review

  • Author Footnotes
    † These authors contributed equally to this work.
    R.G. Bentvelsen
    Correspondence
    Corresponding author. Address: Microvida Laboratory for Microbiology, Amphia Hospital Breda, P.O. Box 90158, 4800 RK Breda, The Netherlands. Tel.: +31 765953015
    Footnotes
    † These authors contributed equally to this work.
    Affiliations
    Department of Medical Microbiology, Leiden University Medical Center, Leiden, The Netherlands

    Microvida Laboratory for Microbiology, Amphia Hospital Breda, The Netherlands
    Search for articles by this author
  • Author Footnotes
    † These authors contributed equally to this work.
    E. Holten
    Footnotes
    † These authors contributed equally to this work.
    Affiliations
    Department of Medical Microbiology, Leiden University Medical Center, Leiden, The Netherlands
    Search for articles by this author
  • N.H. Chavannes
    Affiliations
    Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
    Search for articles by this author
  • K.E. Veldkamp
    Affiliations
    Department of Medical Microbiology, Leiden University Medical Center, Leiden, The Netherlands
    Search for articles by this author
  • Author Footnotes
    † These authors contributed equally to this work.
Open AccessPublished:May 03, 2021DOI:https://doi.org/10.1016/j.jhin.2021.04.029

      Summary

      Background

      The increase in smartphone use and mobile health applications (apps) holds potential to use apps to reduce and detect healthcare-associated infections (HAIs) in clinical practice.

      Aim

      To obtain an overview of available apps for HAI prevention, by selecting the clinically relevant apps and scoring functionality, quality and usefulness.

      Methods

      This scoping review of available apps in the iOS and Android app stores uses an in-house-developed tool (scraper https://holtder.github.io/talos) to systematically aggregate available apps relevant for HAI prevention. The apps are evaluated on functionality, assessed on quality using the ‘Mobile Application Rating Scale’ (MARS), and assessed on potential use in clinical infection prevention.

      Findings

      Using the scraper with CDC HAI topics through 146 search terms resulted in 92,726 potentially relevant apps, of which 28 apps met the inclusion criteria. The majority of these apps have the functionality to inform (27 of 28 apps) or to instruct (20/28). MARS scores for the 28 apps were high in the following domains: functionality (4.19/5), aesthetics (3.49/5), and information (3.74/5), with relatively low scores in engagement (2.97/5), resulting in a good average score (3.57/5).

      Conclusion

      Low engagement scores restrict apps that intend to inform or instruct, possibly explained by the often-academic nature of the development of these apps. Although the number of HAI prevention apps increased by 60% in 5 years, the proportion of clinically relevant apps is limited. The variation in HAI app quality and lack of user engagement, could be improved by co-creation and development in the clinical setting.

      Keywords

      Introduction

      Healthcare-associated infections (HAIs) are major threats to the safety of patients worldwide. HAIs cause excess morbidity, mortality and costs, at all levels of healthcare. Despite efforts in the prevention of HAI it has been estimated that 4% of hospitalized patients or 648,000 patients with 721,800 HAIs occurred in US hospitals in 2015 [
      • Magill S.S.
      • O’Leary E.
      • Janelle S.J.
      • Thompson D.L.
      • Dumyati G.
      • Nadle J.
      • et al.
      Changes in prevalence of health care-associated infections in U.S. hospitals.
      ]. In Europe more than 2.5 million cases of HAI occur every year, with a burden of 501 disability-adjusted life years (DALYs) per 100,000 general population [
      • Cassini A.
      • Plachouras D.
      • Eckmanns T.
      • Abu Sin M.
      • Blank H.P.
      • Ducomble T.
      • et al.
      Burden of six healthcare-associated infections on european population health: estimating incidence-based disability-adjusted life years through a population prevalence-based modelling study.
      ]. Over 2019, the Dutch National Institute for Public Health and the Environment (RIVM) registered a prevalence of HAIs of 5.2% (95% confidence interval 5.0–5.9) [
      Dutch Ministry of Health Welfare and Sport
      Referentiecijfers prevalentie ziekenhuisinfecties 2015 t/m 2019.
      ]. Therefore, new prevention strategies are pursued. As smartphone use increases worldwide, this holds potential to create awareness, to detect, and to reduce HAIs in clinical practice.
      Mobile applications (apps) are occupying an ever-growing role in our lives. One specific group of mobile apps in which this increase is apparent is the group of health-oriented mobile applications or mHealth. The World Health Organization's Global Observatory for eHealth (GOe) defines mHealth as “medical and public health practice supported by mobile devices” [
      Global Observatory for eHealth
      Global diffusion of eHealth: making universal health coverage achievable.
      ]. It is expected that mHealth and eHealth may improve access to care and reduce costs. The 2015 WHO global survey reported that 83% of countries were running at least one national mHealth initiative. Many health-related topics, such as infection prevention, could benefit from mHealth.
      However, literature on HAI prevention apps is scarce; as assessed by the most recent HAI app review in 2015 by Schnall and Iribarren [
      • Schnall R.
      • Iribarren S.J.
      Review and analysis of existing mobile phone applications for health care-associated infection prevention.
      ]. No studies were identified on the effectiveness or usability of HAI prevention apps. In a search in October 2019 of PubMed and Google Scholar, our group identified one mHealth hand hygiene compliance project and three projects on surgical site infection prevention apps, although none of those apps were publicly available in the app stores. Previously, searching the app stores, Schnall found 2646 potential apps, of which 17 were relevant for clinical infection prevention. In addition to primary focus area and features, app store user ratings were given as quality assessment. Because of the lack of evidence, this scoping review will use data from PubMed and ‘grey’ sources such as app stores and websites on app efficacy evidence [
      • Tricco A.C.
      • Lillie E.
      • Zarin W.
      • O'Brien K.K.
      • Colquhoun H.
      • Levac D.
      • et al.
      PRISMA Extension for Scoping Reviews (PRISMAScR): Checklist and Explanation.
      ].
      Studies on the quality of apps, referred to as ‘app store reviews’ (ASRs) are available on various subjects. A systematic review of ASRs published in 2016 evaluated a variety of ASRs on their methodology and concluded that most of these analyses are missing a systematic approach in the aggregation of data or fail to report one [
      • Grundy Q.H.
      • Wang Z.
      • Bero L.A.
      Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods.
      ]. Due to the presentation of apps by the stores, manually collecting a list of relevant apps is prone to error. Apps and their properties as data for an ASR that are not aggregated properly could lead to bias and finding the best method of accumulating these data will prevent such bias. To the best of our knowledge, there is no system available for researchers to generate a dataset as output of search results from app stores. One possible exception is the SARASA method, a methodology that describes the use of R-scripts to gather and filter apps [
      • Albrecht U.V.
      • Hasenfuss G.
      • von Jan U.
      Description of cardiological apps from the German App Store: semiautomated retrospective app store analysis.
      ]. However, SARASA software is not publicly available.
      The aim of this study was to systematically and qualitatively evaluate available apps aimed at the prevention of HAIs. In addition, we aimed to develop a tool to generate a detailed list of relevant apps.

      Methods

      App store search tool development

      To gather comprehensive and systematically organized app store search results, a tool was developed using Python. The tool for generating a collection of relevant data from a large database, is called Talos, a data aggregator or a scraper. See the GitHub repository at https://holtder.github.io/talos for download, installation and user guides. The user specifies search terms and a language region in a webform. The scraper generates a list of relevant apps by querying the app stores with the search terms provided by the user. It then exports a list of all relevant apps that are available in the specified region, sourced from the (Apple) App Store and the (Google) Play Store for apps. The data that app stores return is then converted to a readable format that can be imported by almost all spreadsheet and database management systems, such as Microsoft Excel, Microsoft Access, IBM SPSS or Stata. This data consists of application-specific output variables such as the name, and description of the application, the listed price, and the date of the most recent update.

      Search tool strategy

      Using the scraper Talos, three language regions were queried for relevant apps. Each search term was queried twice in English for the apps available in the USA and the UK. Subsequently, the search terms were translated for a query in the Dutch language region for the Netherlands and Belgium. All terms were based on seven subjects related to infection prevention and HAIs as defined by the US Centers for Disease Control and Prevention (CDC) in 2017 [
      Centers for Disease Control and Prevention
      HAI progress report 2017.
      ]. The types of HAIs include: ventilator-associated pneumonia (VAP), catheter-associated urinary tract infection (CAUTI), central line-associated bloodstream infection (CLABSI), and surgical site infection (SSI). Also, we queried for apps on hand hygiene, meticillin-resistant Staphylococcus aureus (MRSA), and Clostridioides difficile (Supplementary Table S2).

      Selection strategy

      The first selection was performed by filtering based on scraper data. Duplicate apps from different app stores or language regions were excluded, as were all apps not updated in the 18 months before the query date. Secondary exclusion was based on purpose of the apps: (1) games without any educational purpose; (2) non-medical or non-health-related apps; (3) apps without infection prevention purpose. These criteria were applied in two selection rounds. During the first round, apps were excluded based on title and description in the app stores. In the second round, the remaining apps were downloaded on mobile devices and exclusion was based on the content of each application. Several non-functioning apps and apps retracted from the app stores were excluded during this last round.

      Assessment of apps

      After selection, the remaining apps were assessed with regard to their functionality and quality. To assess app functionality, and compare it with previous research, we adapted the seven functionality categories described by the IMS Institute for Healthcare Informatics report: inform, instruct, record, display, guide, remind/alert and communicate [
      IMS Institute for Healthcare Informatics
      Patient apps for improved healthcare: from novelty to mainstream.
      ].
      The quality of the apps was assessed by testing them and rating them using a custom-built webform based on the Mobile Application Rating Scale (MARS) [
      • Stoyanov S.R.
      • Hides L.
      • Kavanagh D.J.
      • Zelenko O.
      • Tjondronegoro D.
      • Mani M.
      Mobile app rating scale: a new tool for assessing the quality of health mobile apps.
      ]. MARS is a questionnaire of 19 questions on a five-point Likert scale concerning ‘engagement’, ‘functionality’, ‘aesthetics’ and ‘information’. The MARS provides one elective domain which assesses the capability of an application to adjust health-related behaviour. This domain was not included in the assessment as not all apps intend behavioural changes in its users. Before testing the apps, the two reviewers viewed and discussed the MARS instructional video. Apps were tested on two Apple iOS and two Google Android devices, an iPhone 7 Plus, an iPad Mini 4, a Google Pixel 2, an Acer R11 Chromebook with Android. Reviewers used each application for at least 20 min before completing the MARS. Each application was evaluated by both raters independently, and after this evaluation phase discrepancies were discussed to reach a consensus score. Ratings of the apps were performed on iOS and Android platforms when the app was available on both, else the application was rated twice on the same platform. The main score for each app is the average of the ratings of the four MARS domains.

      Validity of results

      The intraclass correlation coefficient (ICC) between both reviewers was calculated as a measure of interobserver agreement. This was performed for the average score of each domain and the total average. Both the consistency of agreement ICC (CA-ICC) and the absolute-agreement ICC (AA-ICC) were calculated in a two-way mixed-effects model. This model was suitable as the assessment was performed by two predefined raters, as opposed to a random selection of raters [
      • McGraw K.O.
      • Wong S.P.
      Forming inferences about some intraclass correlation coefficients.
      ,
      • Koo T.K.
      • Li M.Y.
      A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
      ].

      Results

      On 31st October 2019, a total of 146 search term queries were submitted in the scraper, with a total of 92,726 app store search results. Of the search terms, 52 were for the English-US and 52 for the English-UK regions, and 42 for the Dutch-language region. The different number of search terms per language results from more synonyms in English. As multiple searches might return duplicate results, a filter that excludes duplicate results was applied and reduced the number of results to 5394 unique apps. After selection based on title, description, and content testing (Figure 1), the final assortment included 27 English apps and one Dutch app.
      Figure 1
      Figure 1Applications (apps) for prevention of healthcare-associated infections selection and inclusion process with the app store scraper Talos, from two app stores (iOS and Android). HAI, healthcare-associated infections; IP, infection prevention. Not installable or unusable, apps that would crash on start-up or were removed from the app stores between selection and evaluation.

      Descriptive characteristics

      Table I lists the included apps and their associated characteristics. The intended audience (e.g., healthcare workers, patients, administrators) of these apps varies. The majority of the apps are created for healthcare workers (16 apps, 57.1%). A proportion of the apps is either exclusively available on the Google Play Store (6, 21.4%) or the Apple App Store (4, 14.3%). Most of the apps, however, are available on both platforms (18, 64.3%). All found apps, except for two, are free of charge to use. The paid apps are Catheter-Patient Version (0.99 Euro) and Catheter Pro (4.99 Euro).
      Table IList of included apps
      Table thumbnail fx2

      Functionality

      The apps were categorized on functionality and cumulatively displayed in Figure 2. In the Kiviat diagram in Figure 2, the results are overlayed with the results of a study in 2015 performed by Schnall et al. This study is comparable in goal and scope. The most common functionalities were to inform (27 apps, 96.4%) and/or instruct the user (20 apps, 71.4%), these functionalities and displaying data increased in 5 years. Recording data by HAI prevention apps was reduced compared with the 2015 data, as were sharing and collecting data. Two of the 17 (12%) apps from the previous survey were available and included in this study: Guideline Central and iScrublite. Data from 2015 presented no apps that aimed at guiding users, whereas the presented set does include six apps which guide (21.4%).
      Figure 2
      Figure 2Radar graph comparing scraped apps to previous results. Progression of healthcare-associated infection app functionality over time. The apps found in this study are displayed in green, 2015 results by Schnall et al. are overlayed in red.

      Quality assessment

      In Table II each application's MARS rating is provided for the four domains, as is the final MARS rating (Supplementary Table S1). A large number of apps received a final MARS score of 4.00 or above (11 apps, 39.3%). Apps with high scores (MARS = 4.50) included EVS High Touch Test (4.59) and EVS Patient Protector (4.52). No apps received a final MARS rating below 2.00. At the bottom end of the included apps, overall ratings of 3.00 or lower (4, 14.28%) included the reference material apps All Respiratory Disease and Treatment (2.56), Diseases and Disorders Complete Guide (2.05) and Pneumonia Info (2.68). Another application with a low score was WYH (Wash Your Hands, score 2.74), an application that allows users to set alarms for themselves as a reminder to perform hand hygiene. The average total MARS rating of the included apps was 3.57 out of 5 (95% confidence interval, 3.31–3.83), whereas the average rating of the ‘functionality’ domain was higher (4.19; 95% confidence interval, 3.99–4.39).
      Table IIMobile Application Rating Scale (MARS) scores per app
      Table thumbnail fx1a
      Engagement: efforts taken to draw and retain the user's attention and use. Functionality: functions the app offers and how well are they executed. Aesthetics: efforts to present the content of the app in a clear and attractive manner. Information: accurate and credible information with verifiable sources listed. Total: average of the MARS domains. Scores on a five-point Likert scale. The green and red scores, respectively, indicate the top three and bottom three total scores.

      Agreement

      The intraclass coefficient was calculated between raters as the consistency of agreement ICC (CA-ICC) and the absolute agreement ICC (AA-ICC) between the two raters' sets of MARS results (Table III). Generally, the agreement between the two raters was moderate to good, with the notable exception of the functionality domain [
      • McGraw K.O.
      • Wong S.P.
      Forming inferences about some intraclass correlation coefficients.
      ,
      • Koo T.K.
      • Li M.Y.
      A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
      ]. The remaining domains had higher rates of agreement, especially the ‘engagement’ and ‘information’ domains.
      Table IIIConsistency of agreement ICC (CA-ICC) and the absolute agreement ICC (AA-ICC) between the two raters' sets of MARS results, with 95% confidence interval (95% CI)
      Table thumbnail fx3
      Engagement: efforts taken to draw and retain the user’ attention and use. Functionality: functions the app offers and how well are they executed. Aesthetics: efforts to present the content of the app in a clear and attractive manner. Information: accurate and credible information with verifiable sources listed. Total: average of the MARS domains. Scores on a five-point Likert scale.

      Discussion

      This study systematically and qualitatively evaluated mHealth apps designed for the prevention of HAIs. In addition, a scraper was developed to search app stores and generate a detailed list of relevant apps. We found 27 (UK and US) English apps and one Dutch app related to the prevention of HAIs.
      Our study demonstrates an increase of about 60% compared with 2015 when Schnall reported 17 apps without the use of a scraper [
      • Schnall R.
      • Iribarren S.J.
      Review and analysis of existing mobile phone applications for health care-associated infection prevention.
      ]. The majority of apps intends to inform (27 apps, 96.42%) or instruct (20, 71.43%) their targeted audience. Guiding users is reported in more apps than before, as are informing, instructing and displaying data, while the recording, sharing and collecting data function is reduced in this app field. A mere 12% of apps are still available and found after 5 years. On average, HAI apps receive high MARS scores on functionality (4.19) and relatively low scores for engagement (2.97). The general inter-rater agreement was adequate (CA-ICC 0.679, AA-ICC 0.685) with the notable exception of the low agreement in the ‘functionality’ domain (CA-ICC 0.156, AA-ICC 0.156). Multiple studies found that using standardized instructions and forms such as the MARS, produces a moderate to good agreement between raters, which is confirmed in this study [
      • Bardus M.
      • van Beurden S.B.
      • Smith J.R.
      • Abraham C.
      A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management.
      ,
      • Jamaladin H.
      • van de Belt T.H.
      • Luijpers L.C.
      • de Graaff F.R.
      • Bredie S.J.
      • Roeleveld N.
      • et al.
      Mobile apps for blood pressure monitoring: systematic search in app stores and content analysis.
      ].
      Unique to this study is the development and use of a scraper, addressing the previously described need for a detailed systematic approach in the aggregation of apps [
      • Grundy Q.H.
      • Wang Z.
      • Bero L.A.
      Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods.
      ]. The automated formatting of the data by the scraper streamlines the selection process. The main limitation of the use of a scraper is the search output of multiple large datafiles to sort and filter. The total number of unique apps (5394) was substantially larger compared with manual search methods [
      • Schnall R.
      • Iribarren S.J.
      Review and analysis of existing mobile phone applications for health care-associated infection prevention.
      ]. With the increase in results, the effort needed to exclude non-relevant apps also grows considerably. In Table I we added an overview of availability and latest update at publication, which reflects the rapid changes in app supply. The exclusion process, and other factors such as downloading, selecting and assessing potentially relevant apps, result in a less up-to-date overview at publication, which limits the implications of this study, as has unfortunately been seen with other reviews of health apps [
      • Larsen M.
      • Nicholas J.
      • Han J.
      • Lemon C.
      • Okun K.
      • Torok M.
      • et al.
      Here's something I prepared earlier: a review of the time to publication of cross-sectional reviews of smartphone health apps.
      ].
      One strength of this study is the calculation of agreement with inter-rater agreement statistics. These values determine the validity of the raters' findings. The low validity of the high functionality score demonstrates that, even with the use of standardized instructions and forms, app (store) reviews experience observer bias. This could be explained by other experiences on different platforms and devices.
      On average, HAI apps perform well in the ‘aesthetics’, ‘functionality’ and ‘information’ domains, as demonstrated in the EVS apps. The EVS apps provide accessible training on infection prevention subjects for environmental service technicians (EVS). Instructions in the app, which precede the training, are clear and concise. Additionally, during the training exercises, the EVS apps provide entertaining and frequent feedback to the user and keep score of the user's performance. This engages users to frequently use the app to improve their scores. These and other apps with average MARS scores of 4 and over, are apps for potential use at the point of care.
      Despite the efforts made by developers to engage users, for most apps the Engagement score domain is low, potentially causing less repeated use [
      • Kelders S.M.
      • van Zyl L.E.
      • Ludden G.D.S.
      The concept and components of engagement in different domains applied to eHealth: a systematic scoping review.
      ]. Engagement bridges the gap between the developer and the user, which is achieved through the presentation of the application's contents to the user. The process of engagement is complex and no consensus appears to exist on the multi-dimensional construct of behaviour, cognition and affect [
      • Kelders S.M.
      • Kok R.N.
      • Ossebaard H.C.
      • Van Gemert-Pijnen J.E.W.C.
      Persuasive system design does matter: a systematic review of adherence to web-based interventions.
      ]. However, if the app does not succeed in maintaining the attention of its users, the application will eventually be abandoned for a more engrossing competitor.
      A possible explanation for this score disparity, may be limitations brought on by the often-academic nature of the creation of these apps. Many mHealth apps are often initiated as a medical trial, and thus have different goals and limitations compared with commercially developed apps [
      • Wildeboer G.
      • Kelders S.M.
      • van Gemert-Pijnen J.E.W.C.
      The relationship between persuasive technology principles, adherence and effect of web-Based interventions for mental health: A meta-analysis.
      ]. When provided with funding, through grants or hospital administrations, grantees are bound by conditions set by the providing parties. Such conditions might leave little room for focus group testing that allows developers to identify the wishes and requirements of potential users in a co-creation process.
      A possible limitation for HAI apps is the complex interplay in the healthcare setting causing the risk of infection. We acknowledge the differences in the clinical settings, between continents, nations and even institutes. Though the reduction of HAIs is not only the task and responsibility for the designated infection control practitioners. The involved clinicians, nurses, or even patients could and should contribute for real impact against HAI. The CDC and APIC provide infographics for professionals and for the interested ‘casual reader’. This could also be true for infection control apps.
      Several lessons could be drawn from the analysis of the HAI and non-HAI apps in this study. Although a favourable expansion in HAI apps can be observed, the absolute number of clinically relevant HAI apps remains rather small. For mHealth to make a difference in the world of HAI infection prevention, more high-quality apps need to be available to the users. Additionally, studies on – and development of – HAI apps should shift their focus towards user engagement. The majority of developers appear to prevent infections by providing medical information to the user. The contents and the presentation appear to require more than appealing aesthetics. Developers should take note when developing HAI apps.
      Unfortunately, there is no fool-proof solution for the lack of engagement in apps [
      • Jamaladin H.
      • van de Belt T.H.
      • Luijpers L.C.
      • de Graaff F.R.
      • Bredie S.J.
      • Roeleveld N.
      • et al.
      Mobile apps for blood pressure monitoring: systematic search in app stores and content analysis.
      ,
      • Kelders S.M.
      • van Zyl L.E.
      • Ludden G.D.S.
      The concept and components of engagement in different domains applied to eHealth: a systematic scoping review.
      ]. If more mHealth apps are to be adopted as a measure against HAIs, more research is imperative in order to better understand how to engage the user. Developers need to communicate with their target audience and collaborate with infection prevention specialists to tease out what an HAI-application requires to function properly. Adding team-based incentives could increase user engagement by using pre-existing social networks and incorporating concepts from behavioural economics (e.g., herd behaviour) [
      • Pearson E.
      • Prapavessis H.
      • Higgins C.
      • Petrella R.
      • White L.
      • Mitchell M.
      Adding team-based financial incentives to the Carrot Rewards physical activity app increases daily step count on a population scale: a 24-week matched case control study.
      ]. Concepts such as user adherence need to be further explored so a framework can be designed that allows developers to design more appealing apps. To improve impact and uptake, other studies describe development using a holistic framework as the CeHRes roadmap, with contextual inquiry, value specification, design and operationalization all involving the stakeholders, both caregivers and patients [
      • van Gemert-Pijnen J.E.
      • Nijland N.
      • van Limburg M.
      • Ossebaard H.C.
      • Kelders S.M.
      • Eysenbach G.
      • et al.
      A holistic framework to improve the uptake and impact of eHealth technologies.
      ]. Future studies should be aware of the level of engagement and analyse what differentiates apps that engage well from their competitors.
      In conclusion, this study identifies and reviews the apps available to prevent HAIs, with regard to functionality and quality. The diverse apps mainly provide information and have good aesthetics, credible information and useful functions, but are lacking in user engagement. Engagement is essential for the uptake and impact of eHealth and further research on HAI applications should be aimed at improving engagement in a co-creation process during development.

      Author contributions

      R.B., E.H., N.C. and K.V. designed the study, R.B. and E.H. collected the data. E.H. designed and developed the scraper. E.H. and R.B. analysed the data and R.B. and E.H. wrote the original draft. All authors contributed to the final manuscript.

      Conflict of interest statement

      The authors declare no conflicts of interest related to this research.

      Funding

      The department of Medical Microbiology helped by facilitating E.H. and R.B. to work on this project. No additional funding was obtained.

      Appendix A. Supplementary data

      The following are the Supplementary data to this article:

      References

        • Magill S.S.
        • O’Leary E.
        • Janelle S.J.
        • Thompson D.L.
        • Dumyati G.
        • Nadle J.
        • et al.
        Changes in prevalence of health care-associated infections in U.S. hospitals.
        N Engl J Med. 2018; 379: 1732-1744
        • Cassini A.
        • Plachouras D.
        • Eckmanns T.
        • Abu Sin M.
        • Blank H.P.
        • Ducomble T.
        • et al.
        Burden of six healthcare-associated infections on european population health: estimating incidence-based disability-adjusted life years through a population prevalence-based modelling study.
        PLoS Med. 2016 Oct 18; 13e1002150
        • Dutch Ministry of Health Welfare and Sport
        Referentiecijfers prevalentie ziekenhuisinfecties 2015 t/m 2019.
        2019 (Available at:) ([last accessed October 2020])
        • Global Observatory for eHealth
        Global diffusion of eHealth: making universal health coverage achievable.
        (Report of the third global survey on eHealth) World Health Organisation, Geneva2016
        • Schnall R.
        • Iribarren S.J.
        Review and analysis of existing mobile phone applications for health care-associated infection prevention.
        Am J Infect Control. 2015; 43: 572-576
        • Tricco A.C.
        • Lillie E.
        • Zarin W.
        • O'Brien K.K.
        • Colquhoun H.
        • Levac D.
        • et al.
        PRISMA Extension for Scoping Reviews (PRISMAScR): Checklist and Explanation.
        Ann Intern Med. 2018; 169: 467-473
        • Grundy Q.H.
        • Wang Z.
        • Bero L.A.
        Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods.
        Am J Prev Med. 2016; 51: 1051-1059
        • Albrecht U.V.
        • Hasenfuss G.
        • von Jan U.
        Description of cardiological apps from the German App Store: semiautomated retrospective app store analysis.
        JMIR MHealth and UHealth. 2018; 6: e11753
        • Centers for Disease Control and Prevention
        HAI progress report 2017.
        2019 (Available at:) ([last accessed September 2019])
        • IMS Institute for Healthcare Informatics
        Patient apps for improved healthcare: from novelty to mainstream.
        IMS Institute for Healthcare Informatics, 2013
        • Stoyanov S.R.
        • Hides L.
        • Kavanagh D.J.
        • Zelenko O.
        • Tjondronegoro D.
        • Mani M.
        Mobile app rating scale: a new tool for assessing the quality of health mobile apps.
        JMIR Mhealth Uhealth. 2015; 3: e27
        • McGraw K.O.
        • Wong S.P.
        Forming inferences about some intraclass correlation coefficients.
        Psychol Methods. 1996; 1: 30-46
        • Koo T.K.
        • Li M.Y.
        A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
        J Chiropr Med. 2016; 15: 155-163
        • Bardus M.
        • van Beurden S.B.
        • Smith J.R.
        • Abraham C.
        A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management.
        Int J Behav Nutr Phy. 2016; 13: 35
        • Jamaladin H.
        • van de Belt T.H.
        • Luijpers L.C.
        • de Graaff F.R.
        • Bredie S.J.
        • Roeleveld N.
        • et al.
        Mobile apps for blood pressure monitoring: systematic search in app stores and content analysis.
        JMIR Mhealth Uhealth. 2018; 6: e187
        • Larsen M.
        • Nicholas J.
        • Han J.
        • Lemon C.
        • Okun K.
        • Torok M.
        • et al.
        Here's something I prepared earlier: a review of the time to publication of cross-sectional reviews of smartphone health apps.
        BMJ Open. 2020 Dec 2; 10e039817
        • Kelders S.M.
        • van Zyl L.E.
        • Ludden G.D.S.
        The concept and components of engagement in different domains applied to eHealth: a systematic scoping review.
        Front Psychol. 2020 May 27; 11: 926
        • Kelders S.M.
        • Kok R.N.
        • Ossebaard H.C.
        • Van Gemert-Pijnen J.E.W.C.
        Persuasive system design does matter: a systematic review of adherence to web-based interventions.
        J Med Internet Res. 2012; 14: e152
        • Wildeboer G.
        • Kelders S.M.
        • van Gemert-Pijnen J.E.W.C.
        The relationship between persuasive technology principles, adherence and effect of web-Based interventions for mental health: A meta-analysis.
        Int J Med Inform. 2016; 96: 71-85
        • Pearson E.
        • Prapavessis H.
        • Higgins C.
        • Petrella R.
        • White L.
        • Mitchell M.
        Adding team-based financial incentives to the Carrot Rewards physical activity app increases daily step count on a population scale: a 24-week matched case control study.
        Int J Behav Nutr Phys Act. 2020 Nov 19; 17: 139
        • van Gemert-Pijnen J.E.
        • Nijland N.
        • van Limburg M.
        • Ossebaard H.C.
        • Kelders S.M.
        • Eysenbach G.
        • et al.
        A holistic framework to improve the uptake and impact of eHealth technologies.
        J Med Internet Res. 2011; 13: e111