RESEARCH ARTICLE Imputation of adverse drug reactions: Causality assessment in hospitals Fabiana Rossi Varallo1,2☯, Cleopatra S. Planeta1☯, Maria Teresa Herdeiro3‡, Patricia de Carvalho Mastroianni1‡* 1 São Paulo State University (UNESP), School of Pharmaceutical Sciences, Araraquara, São Paulo, Brazil, 2 CAPES Foundation, Ministry of Education of Brazil, Brası́lia—DF, Brazil, 3 Departamento de Ciências Médicas—Universidade de Aveiro, Aveiro, Portugal ☯ These authors contributed equally to this work. ‡ These authors also contributed equally to this work. * patriciamastroianni@yahoo.com.br Abstract Background & objectives Different algorithms have been developed to standardize the causality assessment of adverse drug reactions (ADR). Although most share common characteristics, the results of the causality assessment are variable depending on the algorithm used. Therefore, using 10 different algorithms, the study aimed to compare inter-rater and multi-rater agreement for ADR causality assessment and identify the most consistent to hospitals. Methods Using ten causality algorithms, four judges independently assessed the first 44 cases of ADRs reported during the first year of implementation of a risk management service in a medium complexity hospital in the state of Sao Paulo (Brazil). Owing to variations in the ter- minology used for causality, the equivalent imputation terms were grouped into four catego- ries: definite, probable, possible and unlikely. Inter-rater and multi-rater agreement analysis was performed by calculating the Cohen´s and Light´s kappa coefficients, respectively. Results None of the algorithms showed 100% reproducibility in the causal imputation. Fair inter- rater and multi-rater agreement was found. Emanuele (1984) and WHO-UMC (2010) algo- rithms showed a fair rate of agreement between the judges (k = 0.36). Interpretation & conclusions Although the ADR causality assessment algorithms were poorly reproducible, our data sug- gest that WHO-UMC algorithm is the most consistent for imputation in hospitals, since it allows evaluating the quality of the report. However, to improve the ability of assessing the causality using algorithms, it is necessary to include criteria for the evaluation of drug-related problems, which may be related to confounding variables that underestimate the causal association. PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 1 / 10 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPENACCESS Citation: Varallo FR, Planeta CS, Herdeiro MT, Mastroianni PdC (2017) Imputation of adverse drug reactions: Causality assessment in hospitals. PLoS ONE 12(2): e0171470. doi:10.1371/journal. pone.0171470 Editor: Naoki Oiso, Kinki Daigaku, JAPAN Received: October 23, 2016 Accepted: January 20, 2017 Published: February 6, 2017 Copyright: © 2017 Varallo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: All relevant data are within the paper. Funding: This work was supported by the Coordination of Improvement of Higher Education Personnel (CAPES) for the scholarship (PDSE) provided [grant number 014301/2013-00]; grant 2013/10381-1, São Paulo Research Foundation (FAPESP) for partial presentation; grant 2014/ 12681-2, São Paulo Research Foundation (FAPESP), for a regular project and the Programa de Apoio ao Desenvolvimento Cientı́fico da Faculdade de Ciências Farmacêuticas da UNESP- PADC. http://crossmark.crossref.org/dialog/?doi=10.1371/journal.pone.0171470&domain=pdf&date_stamp=2017-02-06 http://crossmark.crossref.org/dialog/?doi=10.1371/journal.pone.0171470&domain=pdf&date_stamp=2017-02-06 http://crossmark.crossref.org/dialog/?doi=10.1371/journal.pone.0171470&domain=pdf&date_stamp=2017-02-06 http://crossmark.crossref.org/dialog/?doi=10.1371/journal.pone.0171470&domain=pdf&date_stamp=2017-02-06 http://crossmark.crossref.org/dialog/?doi=10.1371/journal.pone.0171470&domain=pdf&date_stamp=2017-02-06 http://crossmark.crossref.org/dialog/?doi=10.1371/journal.pone.0171470&domain=pdf&date_stamp=2017-02-06 http://creativecommons.org/licenses/by/4.0/ 1 Introduction The adverse drug reaction (ADR) causality assessment is a routine procedure in Pharmacovigi- lance[1], because it allows assessing drug safety parameters and the relationship and likelihood between drug exposure and the occurrence of ADR of health technologies in the post-market- ing period. Since the 1970s, different methods to standardize the evaluation of the causal association of ADRs have been available, ranging from small questionnaires to comprehensive algorithms[2]. The development of these tools, which are ordinary to use[3] and require minimal expertise to be employed[1,4], aims to solve methodological bias, reliability, and validity issues in the imputation of drug-induced adverse effects[5]. However, the main advantage arises from the possibility of decentralizing the causality assessment from the medical diagnosis, extending it to different health care levels: academics, the pharmaceutical industry, and health agencies[6]. By standardizing ADR causality assessment, the uncertainty of the association between a drug and an adverse event will not be reduced, but semi-quantitatively categorized[2] in differ- ent links of probability. Establishing a causal link may influence the rationale for the correlation of an event that occurs to drug consumers[7]; therefore, the results of the causality assessments using algo- rithms must be reproducible. This is important to ratify the viability of their employment in pharmacovigilance[8], as well as their capacity to detect ADR signals[9,10]. This is because the higher the agreement on a defined ADR causal link, the more robust the hypothesis about the relationship between the use of a medication and the adverse event observed, allowing the communication of the risk and, therefore, the implementation of risk minimization and patient safety plans. Because serious ADRs lead to hospitalization, it is necessary to assess the causality in the tertiary health care level. However, there are few data about the agreement on ADR causality assessment using different algorithms in patients hospitalized in internal medicine units in developing countries. It is known that most ADR evidence arises from hospitals, due to the high risks associated with treatments in the tertiary health care level[11]. Therefore, the causality assessment in high complexity institutions contributes to: i) the early recognition of adverse effects, which helps to prevent iatrogenic complications; ii) therapy optimization[2]; iii) establishing barriers to prevent recurrence; iv) reducing the time of hospitalization and unnecessary burden with hos- pitalizations that could be avoided[12]. This study aimed to compare the results of the imputation of ADRs using different algo- rithms in a Brazilian public hospital, to identify the most appropriate for establishing causal associations between medication use and the occurrence of adverse events. 2 Material and methods 2.1 Study design We assessed the causality of all of the ADRs reported by health professionals during the first year of implementation of the pharmacovigilance service (March 2012 until March 2013) in a general assistance, public, medium complexity (secondary health care level) hospital with 104 beds located in the state of São Paulo. 2.2 Selection of algorithms Twenty-nine (29) algorithms for ADR causality assessment were identified by literature review. Nineteen (19) were excluded for the following reasons: absence of equivalent terminology for Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 2 / 10 Competing Interests: The authors have declared that no competing interests exist. the level of imputation of ADRs (n = 6); inclusion of information that is not required for the causality assessment in Brazil (n = 3); tools that were developed for the assessment of specific ADRs (n = 3); and no access to the article (n = 7). The ten algorithms (Table 1) considered eligible for the study included the combination of five main criteria for the causality assessment[13], namely: i) plausible temporality; ii) prior bibliographic description of the adverse effects related to the use of the drug involved; iii) alter- native causes; iv) positive withdrawal (discontinuation of the drug with improvement of the ADR); v) positive rechallenge (reintroduction of the drug with reappearance of the ADR). Owing to the quantitative and qualitative variability in the terminologies used to express the results of the imputation of ADRs in the included algorithms, the nomenclature developed by Macedo et al. (2005)[13] was used. To improve the accuracy of the comparison, the equiva- lent terms of the likelihood level were grouped into four major categories: definite, probable, possible and unlikely. 2.3 Causality assessment Using 10 causality algorithms (Table 1), four judges: FRV (rater A), ADFS (rater B) SPS (rater C) and IO (rater D) independently assessed the first 44 cases of ADRs reported to the hospital’s risk management service during its first year of implementation. The group of judges who conducted the analysis included: a clinical pharmacist of the hos- pital (rater A) who had PhD in Pharmaceutical Sciences and 8 years of professional experience with pharmacovigilance issues; three pharmacy undergraduate students (raters B, C and D) who were in the last year of the course and had previously experience in pharmacovigilance´s scientific research for at least 1 year. The students were trained, in order to standardize the analysis of causal association. The 12-hour training included: 1) discussion of scientific papers on the subject (evaluation of ADR causality; evaluation of ADR causality with different deci- sion algorithms, application of Austin Bradford-Hill’s criteria in pharmacoepidemiological studies); 2) directed study (comparison and critical analysis) of the algorithms used; 3) simula- tion of an ADR causality assessment with a fictional case[14]. The cases of ADRs reported and selected for the study contained at least the following infor- mation: i) suspected drug (start and end date); ii) a brief description of the event (start and end Table 1. Algorithms selected for the causality assessment of adverse drug reactions in a public and general hospital in the State of Sao Paulo, Brazil (n = 10). Algorithm Reference (year) Karch and Lasagna Clin Pharmacol Ther (1977); 21: 247–254 Blanc et al. Clin Pharmacol Ther(1979); 25: 493–498 Kramer et al. JAMA(1979); 242: 623–632. Naranjo et al. Clin Pharmacol Ther (1981); 30: 239–245 Jones Fam Community Health (1982); 5: 58–67 Emanuelli Drug Inf J (1984); 18: 303–306 Mashford Drug Inf J (1984); 18: 271–273 Venulet et al. Int J Clin Pharmacol Ther (1986); 24: 559–568 WHO-UMC Available from: http://who-umc.org/Graphics/24734.pdf Gallagher et al. PLoS One (2011), 6 (12): e28096 The use of the WHO-UMC system for standardized case causality assessment. World Health Organization (WHO)—Uppsala Monitoring Centre. [Last accessed on 2013 Apr 10]. Available from: http://www.who-umc. org/Graphics/24734.pdf. doi:10.1371/journal.pone.0171470.t001 Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 3 / 10 http://who-umc.org/Graphics/24734.pdf http://www.who-umc.org/Graphics/24734.pdf http://www.who-umc.org/Graphics/24734.pdf date, data of laboratory tests when relevant); iii) polypharmacy (start and end date); iv) the patient’s medical history; v) relevant interventions. We considered ADR any noxious, unintended, or undesired effect of a drug occurring at doses used in humans for prophylaxis, diagnosis, or therapy[15]. The clinical manifestations reported were classified according to seriousness and expec- tance. Serious ADR were defined as those causing hospitalization, those that were fatal or life- threatening, or those that resulted in significant changes in patient treatment (thereby pro- longing hospitalization)[16]. Informational drug sheets approved by the National Agency of Sanitary Surveillance (ANVISA) and monographs, such as those in the DRUGDEX (MICROMEDEX1database), Uptodate1 database and LexiComp Manole (2009) were consulted to verify the expectancy of ADR. The results of imputation obtained with the ten algorithms were compared to analyze the agreement between the judges and the feasibility of the algorithms in the causality assessment in hospitals. 2.4 Statistical analysis Two descriptive statistics were used to measure the nominal agreement between two or more raters: Cohen´s kappa and Light´s kappa. Cohen´s kappa measure the degree of concordance between two judges. The analysis car- ried out by FRV (rater A) was considered gold-standard to calculate the inter-rater agreement between judges B, C and D. Light´s kappa is a multi-rater statistic which measures the degree of concordance among multiple judges without gold-standard. It is an extension of Cohen’s kappa. For both tests, we considered α = 0.05, 95%CI for all analyses. Values were interpreted according to Landis and Koch protocol (1977)[17] (Table 2). 2.5 Research ethics committee This study (E-015/10 protocol) was approved by the Research Ethics Committee of the Insti- tuto Lauro de Souza Lima. 3 Results During the period of data collection, the risk management department received 24 ADRs reports that enclose 36 different types of clinical manifestations resulting from 19 drugs (Table 3). Owing to the causality imputation was carried out case to case, each judge indepen- dently assess 44 cases, since a single report may describe more than one clinical manifestation associated with only one drug or may signalize more than one suspected drug for the occur- rence of a single clinical manifestation. Table 2. Interpretation of the Kappa (K) value. Kappa value Interpretation of the agreement <0.00 Less than chance 0.00 |– 0.21 Slight 0.21 |– 0.41 Fair 0.41 |– 0.61 Moderate 0.61|–0.81 Substantial 0.81|–|1.00 Almost perfect doi:10.1371/journal.pone.0171470.t002 Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 4 / 10 Table 3. Characteristics of the adverse drug reactions (ADR) analyzed, according to characteristic of patients, expectative, frequency, serious- ness of the event (n = 44). Characteristic of patient (age, sex and diagnosis) ADR Suspected Drug (s) Expectative Frequency1 Seriousness 42 years old, female, human immunodeficiency virus Chemical phlebitis azithromycin Expected 1.0–10.0% Non-serious 30 years old, male, viral meningitis skin rash [Ceftriaxone], [ketoprofen], [aciclovir] Expected [1.0%], [9.0%], [>3.0%] Non-serious 32 years old, male, transverse loop colostomy Shin rash [Ciprofloxacin], [clindamycin] Expected >1,8% Non-serious 51 years old, female, chronic obstructive lung disease Tachycardia1, dyspnea2 and anxiety disorders3 [Ipratropium],[Fenoterol] Expected1,2* [Tachycardia <1.0%] [dyspnea 10.0%] Serious 37 years old, male, chronic osteomyelitis Chemical phlebitis meropenem Expected 2.0% Non-serious 52 years old, female, hydronephrosis with ureteropelvic junction obstruction Tachycardia1, malaise2 and vaginal burning sensation3 azithromycin Expected1,2* <1.0% Non-serious 76 years old, female, dementia with Lewy bodies Skin rash carbamazepine Expected 7.0% Non-serious 37 years old, female, human immunodeficiency virus Malaise1, tachycardia2, hyperhidrosis3 metoclopramide Expected2* Not defined2 Non-serious 65 years old, male, pleural effusion Chemical phlebitis meropenem Expected 2.0% Non-serious 68 years old, male, stroke Shin rash1 and urticarial2 acetylsalicylic acid Expected1,2 Not defined1,2 Serious 33 years old, male, human immunodeficiency virus Red men syndrome vancomycin Expected > 10.0% Non-serious 84 years old, male, pneumonia Skin rash piperacillin and enzyme inhibitor Expected 4.0% Non-serious 84 years old, male, urinary tract infection Pruritus polymyxin B Expected Not defined Non-serious 40 years old, male, Wernicke encephalopathy Severe hypokalemia polymyxin B Expected Not defined Serious 42 years old, female, obesity, cellulitis Irritation on the local of application heparin Expected Not defined Non-serious 49 years old, male, pulmonary tuberculosis Skin rash [Ceftriaxone], [clindamycin], [metronidazole] Expected [2.0%], [not defined], [not defined] Non-serious 62 years old, female, chronic pressure ulcer Nausea and vomit dipyrone Expected Not defined Non-serious 62 years old, female, chronic pressure ulcer Acute kidney impairment vancomycin Expected 5.0% Serious 35 years old, male, human immunodeficiency virus Skin rash piperacillin and enzyme inhibitor Expected 4.0% Non-serious 50 years old, female, chronic obstructive lung disease Pain in the local of application1, tachycardia2 clarithromycin Expected �0.1% and < 10.0%1, <1.0% 2 Non-serious 42 years old, female, cellulitis Erythema levofloxacin Expected < 1.0% Non-serious 24 years old, male, human immunodeficiency virus Anemia zidovudine Expected < 1.0% Serious 66 years old, female, chronic pressure ulcer Mental confusion1, drowsiness2, constipation3, vomit4 morfine Expected1-4 6.0%1, > 9.0%2,3, >10.0%4 Serious 74 years old, male, cellulitis Myoclonus morfine Expected < 1.0% Serious 1Reference: Micromedex® database, Up to Date® database and Lexi-comp Manole (2009). *Except to anxiety disorders, vaginal burning sensation, malaise and hyperhidrosis. doi:10.1371/journal.pone.0171470.t003 Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 5 / 10 According to seriousness, seven ADR reports showed symptomatology classified as serious, since 4 of them prolonged hospital length-stay, 2 resulted in temporary disability and 1 was related to hospital admission. After causality assessment, none of the algorithms showed 100% agreement between judges on the imputation of ADRs. Fair agreement was observed for both statistic tests (Cohen´s and Light´s kappa) (Table 4). Findings suggest the poor reproducibility of the algorithms in per- forming ADR imputation with different judges. The lower inter-rater agreement was observed for the judge B, except for Emanueli (1984), Jones (1986) and WHO-UMC (2010) algorithms (Table 4). Venulet (k = 0.15), Kramer (k = 0.19), and Naranjo (k = 0.20) algorithms showed the worst multi-raters coefficient, indicating slight agreement on causal association. Moreover, the agreement was stronger in Emanueli (k = 0.36), Mashford (1984) and WHO-UMC (k = 0.36) algorithms, but not better than fair (Table 4). 4 Discussion Our data suggest that WHO-UMC algorithm is the most consistent for causal imputation of hospital ADR that affected patients admitted to an internal medicine unit of a medium com- plexity hospital. The advantage of this tool is the semi-quantitative assessment of the causal likelihood and of the quality of the report; it has been used as a gold standard in causality stud- ies[8,13]. Moreover, this tool was developed to evaluate the occurrence of adverse effects dur- ing the post-marketing period, which helps to achieve higher probability scores of causal association and a better reproducibility between the judges. Emanueli (1984) algorithm, which contains a minimalist, simplified, dichotomous structure that considers only the clinical condition of the patient with alternative cause, may overesti- mate the cases of ADR and generate false-positive signals in risk communication, which is why it is not the most recommended for causality assessment in the context of this study. For the remaining algorithms, we noted a weak agreement between the judges on ADR cau- sality. Studies have shown a great variability in the results of imputation of ADR using different algorithms[8,13,18–22]. According to Shakir and Layton (2002)[10], the tools are inconsistent and sometimes of poor quality for signal detection. Furthermore, they have significant limita- tions that reduce the accuracy and reliability of the assessment of the probability of ADR[1]. Table 4. Inter-rater and multi-rater agreement in adverse drug causality assessment, according to the statistical analysis with Cohen´s and Light ´s kappa. Algorithm Cohen’s Kappa Light’s Kappa Judge B Judge C Judge D Multiple judges Kappa; IC (95%) Kappa; IC (95%) Kappa; IC (95%) I Kappa; IC (95%) Kart e Lasagna (1977) 0.14 (0.00–0.37) 036 (0.14–0.57) 0.29 (0.05–0.51) 0.23 (0.12–0.37) Kramer et al. (1979) 0.21 (0.02–0.44) 0.53 (0.32–0.73) 0.34 (0.16–0.54) 0.19 (0.09–0.34) Blanc et al. (1979) 0.16 (0.00–0.36) 0.37 (0.15–0.57) 0.49 (0.29–0.67) 0.29 (0.19–0.42) Naranjo et al. (1981) 0.29 (0.03–0.55) 0.39 (0.13–0.65) 0.41 (0.16–0.69) 0.20 (0.08–0.39) Jones (1982) 0.54 (0.25–0.76) 0.34 (0.03–0.59) 0.09 (0.00–0.37) 0.27 (0.12–0.46) Emanuelli et al. (1984) 0.44 (0.19–0.67) 0.29 (0.04–0.54) 0.40 (0.16–0.64) 0.36 (0.21–0.53) Mashford (1984) 0.26 (0.00–0.59) 0.53 (0.19–0.79) 0.54 (0.29–0.76) 0.33 (0.21–0.50) Venulet et al. (1986) 0.16 (0.06–0.31) 0.32 (0.09–0.57) 0.37 (0.13–0.60) 0.15 (0.05–0.28) WHO-UMC (2010) 0.32 (0.09–0.58) 0.37 (0.15–0.58) 0.41 (0.17–0.64) 0.36 (0.10–0.53) Gallagher et al. (2011) 0.21 (0.01–0.42) 0.41 (0.21–0.60) 0.26 (0.04–0.50) 0.29 (0.17–0.43) Mean ± standard deviation 0.27 ± 0.13 0.39 ± 0.08 0.36 ± 0.12 0.27 ± 0.07 doi:10.1371/journal.pone.0171470.t004 Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 6 / 10 Considering Naranjo et al. (1981) algorithm, data from previous study showed slight agree- ment between the judges[21], because it was developed and validated for the assessment of ADRs that occur during randomized clinical trials[19]. Other authors suggest the use of this tool for the imputation of ADR[21] due to its rapid implementation. However, we disagree this is the only factor to consider when choosing an algorithm. In addition to this aspect, the reliability of the results and the limitations of each tool, especially in the context of medication use (clinical trial versus post-marketing surveillance), should be considered. According to the data from our study, WHO-UMC (2010) partially meets these criteria. We understand that it meets in an incomplete manner, because all of the analyzed algo- rithms do not include other factors that may be associated with adverse events, such as medi- cation errors, product quality deviations, and suspected therapeutic ineffectiveness. Most consider in the assessment only the drug safety issues and neglect (Emanueli 1884; Blanc et al., 1979; Gallagher et al., 2011) or ambiguously (Karch Lasagna, 1977), subjectively (Naranjo et al., 1981; Mashford, 1984; WHO-UMC, 2010) or complexly (Kramer et al., 1979; Venulet et al., 1986) manage other factors that may be associated with adverse events. Even Gallagher et al. (2011) algorithm which was developed after the new definition of pharmacovigilance in 2002 did not include relevant information in the assessment, allowing underestimation of a causal association. This may also be correlated with the low agreement between the judges. The arbitrary weighting given to the evaluation criteria is another limitation that may con- tribute to the inconsistency of algorithms[5,23]. This adds subjectivity inherent to the algo- rithm structure according to criteria these authors deem most important and give greater weighting in scoring. The causality assessment itself also includes some subjectivity[2,6]. Both situations described may contribute to the poor agreement between the algorithms in the imputation of the causal link of ADR. Another evidence that may decrease the accuracy of risk communication is the absence of ADR reports of good quality and underreporting[10,24,25]. Poor or missing information in the reports makes it difficult assessing causality in details, differentiating between probable and possible cases[2], and finding a definitive causal association. Consequently, the assump- tions are not robust enough to generate signals in pharmacovigilance, impairing the assess- ment of drug safety in the post-marketing period. Nowadays, there is no gold-standard algorithm for the assessment of events occurring in primary care[6,26] and studies that compared the imputation of ADR reported to national pharmacovigilance centers[13,23]. At the tertiary health care level, Kane-Gill et al. (2012)[20] found strong agreement when comparing three algorithms by active search of retrospective cases of ADR in the intensive care unit. This can be explained by the ward where the study was conducted, the methodology (one judge) and the active search method. Critical patients are constantly monitored, so the records in medical charts are more complete, which allows the collection of better information and increases the robustness of causality assessment. However, the disadvantage of the active search is the time necessary to review medical records[27], which turns the process unfeasible. Considering the limitations described, there is evidence that it is necessary to develop better quality tools that improve the diagnosis of ADR[26]. A strategy to increase the generation of pharmacovigilance signals is the monitoring of adverse drug events[28], and its evaluation cri- teria should be included in the algorithms to improve the reproducibility in the causal imputa- tion. These criteria involve the assessment of any drug-related problem. Therefore, in an attempt to minimize the described flaws and confounding variables during the assessment, the need, effectiveness, adherence and safety parameters should also be included in the algorithms in order to update the assessment with the new concepts of WHO (2002)[29] about post-marketing studies. Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 7 / 10 Considering drug use[28,30] and the intentional non-compliance with pharmacotherapy related to the diagnosis stereotypical diseases[31] are associated with undesirable effects, it is also necessary to evaluate the impact of ADRs on the patient. This would help to know the pri- orities, difficulties and factors that may motivate the use or discontinuation of therapy and therefore the occurrence of undesirable effects derived from these perceptions and practices. Finally, although the literature presents a wide range of methods for the causality assess- ment, including computational approaches, algorithms are still viable alternatives to the cau- sality assessment in hospitals, since these tools are easy to use, require little financial resources to be applied in the clinical routine and need minimal expertise to be applied[18]. Thus, it is important to update these algorithms in accordance with the new definition of pharmacovigi- lance, allowing the monitoring of adverse drug events[26], in order to minimize confounding variables associated with the causal imputation process and therefore improve the risk/benefit assessment of medications available on the market. 5 Conclusion Our data show slight agreement on the ADR causality assessment for the majority of the tested algorithms. However, WHO-UMC (2010) algorithm showed fair reproducibility and allows the analysis of the quality of the report, which is why we suggest that it is the best tool for cau- sality assessment of ADRs occurring in hospitals. Since the Naranjo algorithm was developed and validated to diagnose ADR occurring in randomized clinical trials and showed slight con- cordance between the judges, this tool is not the most consistent for the assessment of ADRs that affect non-critical patients in a secondary hospital. In addition, data demonstrate the need for the development of better quality tools, that include other criteria for the assessment of drug-related problems, such as effectiveness, safety, compliance, quality or quality deviation and medication errors. Acknowledgments The CAPES Foundation, Ministry of Education of Brazil for the scholarship (PDSE) grant n˚. 014301/2013-00. The authors would also like to thank FAPESP for the financial support in this project, under the grant #2013/10263-9, São Paulo Research Foundation (FAPESP), the Pro- grama de Apoio ao Desenvolvimento Cientı́fico da Faculdade de Ciências Farmacêuticas da UNESP-PADC and to Instituto de Biomedicina (iBiMED) FCT Ref N˚ UID/BIM/04501/2013. We are also thankful to the Hospital Estadual Américo Brasiliense, which allowed its data to be collected. Author contributions Conceptualization: FRV CSP MTH PCM. Data curation: FRV CSP MTH PCM. Formal analysis: FRV CSP MTH PCM. Funding acquisition: FRV PCM. Investigation: FRV CSP MTH PCM. Methodology: FRV CSP MTH PCM. Project administration: CSP PCM. Supervision: PCM MTH. Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 8 / 10 Validation: FRV CSP MTH PCM. Visualization: FRV PCM. Writing – original draft: FRV CSP. Writing – review & editing: PCM MTH. References 1. WHO-UMC. The use of the WHO-UMC system for standardised case causality assessment. 2010. [Last accessed on 2015 nov 16]. Available from: http://who-umc.org/Graphics/24734.pdf 2. Meyboom RH, Hekster YA, Egberts ACG, Gribnau FWJ, Edwards RI. Causal or casual? The role of causality assessment in pharmacovigilance. Drug Safety 1997; 17 (6): 374–389. PMID: 9429837 3. Theóphile H, André M, Arimone Y, Haramburu F, Miremont-Salamé G, Bégaud B. An updated method improved the assessment of adverse drug reaction in routine pharmacovigilance. Journal of Clinical Epi- demiology 2012; 65: 1069–1077. doi: 10.1016/j.jclinepi.2012.04.015 PMID: 22910538 4. Coloma PM, Avillach P, Salvo F, Schuemie MJ, Ferrajolo C, Pariente A, et al. A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases. Drug Safety 2013; 36:13–23. doi: 10.1007/s40264-012-0002-x PMID: 23315292 5. Doherty MJ. Algorithms for assessing the probability of an Adverse Drug Reaction. Respiratory Medi- cine CME 2009; 2: 63–67. 6. Agbabiaka TB, Savovic J, Ernst E. Methods for causality assessment of adverse drug reactions: a sys- tematic review. Drug Safety 2008; 31 (1): 21–37. PMID: 18095744 7. Meyboom RHB, Egbertes ACG, Gribnau FWJ, Hekster YA. Pharmacovigilance in perspective. Drug Safety 1999; 21 (6):429–447. PMID: 10612268 8. Macedo AF, Marques FB, Ribeiro CF, Teixeira F. Causality assessment of adverse drug reactions: comparison of the results obtained from published decisional algorithms and from the evaluations of an expert panel, according to different levels of imputability. Journal of Clinical Pharmacy and Therapeutics 2003; 28: 137–143. PMID: 12713611 9. OPS. Buenas Prácticas de Farmacovigilancia para las Américas. Washington, D. C.: OPS,© 2011. (Red PARF Documento Técnico No. 5). 78 pages. 10. Shakir SAW, Layton D. Causal association in pharmacovigilance and pharmacoepidemiology thoughts on the application of the Austin Bradford-Hill criteria. Drug Safety 2002; 25 (6): 467–471. PMID: 12071785 11. Mugosa S, Bukumirić Z, Kovacević A, BoskovićA, ProtićD, Todorović Z. Adverse drug reactions in hos- pitalized cardiac patients: characteristics and risk factors.Vojnosanit Pregl. 2015; 72(11):975–81. PMID: 26731971 12. Angamo MT, Chalmers L, Curtain CM, Bereznicki LR. Adverse-Drug-Reaction-Related Hospitalisations in Developed and Developing Countries: A Review of Prevalence and Contributing Factors.Drug Saf. 2016; 39(9):847–57. doi: 10.1007/s40264-016-0444-7 PMID: 27449638 13. Macedo AF, Marques FB, Ribeiro CT, Teixeira F. Causality assessment of adverse drug reactions: comparison of the results obtained from published decisional algorithms and from the evaluations of an expert panel. Pharmacoepidemiology and Drug Safety 2005; 14: 885–890. doi: 10.1002/pds.1138 PMID: 16059869 14. Farcas A, Bojita M. Adverse Drug Reactions in Clinical Practice: a Causality Assessment of a Case of Drug-Induced Pancreatitis. The Journal of Gastrointestinal and Liver Diseases 2009; 18: 353–358. PMID: 19795031 15. WHO. International drug monitoring: the role of the national centers. WHO Technical Report Series n. 498. Genebra: WHO, 1972. 16. Moore N, Lecointre D, Noblet C, Mabille M. Frequency and cost of serious adverse drug reactions in a department of general medicine. B J Clin Pharmacol 1998; 45:301–308. 17. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar; 33(1):159–74. PMID: 843571 18. Kane-Gill SL, Devlin JW. Adverse drug event reporting in the intensive care unit: a survey of current practices. Annals of Pharmacotherapy 2006; 40: 1267–1273. doi: 10.1345/aph.1H088 PMID: 16849619 Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 9 / 10 http://who-umc.org/Graphics/24734.pdf http://www.ncbi.nlm.nih.gov/pubmed/9429837 http://dx.doi.org/10.1016/j.jclinepi.2012.04.015 http://www.ncbi.nlm.nih.gov/pubmed/22910538 http://dx.doi.org/10.1007/s40264-012-0002-x http://www.ncbi.nlm.nih.gov/pubmed/23315292 http://www.ncbi.nlm.nih.gov/pubmed/18095744 http://www.ncbi.nlm.nih.gov/pubmed/10612268 http://www.ncbi.nlm.nih.gov/pubmed/12713611 http://www.ncbi.nlm.nih.gov/pubmed/12071785 http://www.ncbi.nlm.nih.gov/pubmed/26731971 http://dx.doi.org/10.1007/s40264-016-0444-7 http://www.ncbi.nlm.nih.gov/pubmed/27449638 http://dx.doi.org/10.1002/pds.1138 http://www.ncbi.nlm.nih.gov/pubmed/16059869 http://www.ncbi.nlm.nih.gov/pubmed/19795031 http://www.ncbi.nlm.nih.gov/pubmed/843571 http://dx.doi.org/10.1345/aph.1H088 http://www.ncbi.nlm.nih.gov/pubmed/16849619 19. Davies EC, Rowe PH, James S, Nickless G, Ganguli A, Danjuma M, et al. An investigation of disagree- ment in causality assessment of adverse drug reactions. Pharmaceutical medicine 2011; 25 (1): 17– 24. 20. Kane-Gill SL, Forsberg EA, Verrico MM, Handler SM. Comparison of three pharmacovigilance algo- rithms in the ICU setting: a retrospective and prospective evaluation of ADRS. Drug Safety 2012; 35 (8): 645–653. doi: 10.2165/11599730-000000000-00000 PMID: 22720659 21. Belhekar MN, Taur SF, Munshi RP. A study of agreement between the Naranjo algorithm and WHO- UMC criteria for causality assessment of adverse drug reactions. Indian J Pharmacol 2014; 46 (1): 117–120. doi: 10.4103/0253-7613.125192 PMID: 24550597 22. Théophile H, André M, Miremont-Salamé G, Arimone Y, Bégaud B. Comparison of three methods (an updated logistic probabilistic method, the Naranjo and Liverpool algorithms) for the evaluation of routine pharmacovigilance case reports using consensual expert judgments as reference. Drug Safety 2013; 36: 1033–1044. doi: 10.1007/s40264-013-0083-1 PMID: 23828659 23. Arimone Y, Bégaud B, Miremont-Salamé G, Fourrier-Réglat A, Moore N, Molimard M, et al. Agreement of expert judgment in causality assessment of adverse drug reactions. Eur J ClinPharmacol 2005; 61: 169–173. 24. Edwards R. An agenda for UK clinical pharmacology pharmacovigilance. British Journal of Clinical Pharmacology 2012; 73 (6): 979–982. doi: 10.1111/j.1365-2125.2012.04249.x PMID: 22360774 25. Pal SN, Duncombe C, Falzon D, Olsson S. WHO Strategy for Collecting Safety Data in Public Health Programmes: Complementing Spontaneous Reporting Systems. Drug Safety 2013; 36: 75–81. doi: 10. 1007/s40264-012-0014-6 PMID: 23329541 26. Khan LM, Al-Harthi SE, Osman AM, Sattar MAAA, Ali AS. Dilemmas of the causality assessment tools in the diagnosis of adverse drug reactions. Saudi Pharm J. 2016; 24 (4):485–93. doi: 10.1016/j.jsps. 2015.01.010 PMID: 27330379 27. Classen DC, Resar R, Griffin F, Federico F, Frankel T, Kimmel N, et al. ’Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011; 30(4):581–9. 28. Seeger JD. Future proofing adverse event monitoring. Drug Safety 2015; 38 (10): 847–1048. doi: 10. 1007/s40264-015-0342-4 PMID: 26323240 29. World Health Organization (WHO). The importance of pharmacovigilance. Geneva: WHO, 2002, 48 pages. 30. De Vries ST, Keers JC, Visser R, de Zeeuw D, Haaijer-Ruskamp FM, Voorham J, et al. Medications beliefs, treatment complexity, and non-adherence to different drug classes in patients with type 2 diabe- tes. J Psychosom Rs 2014; 76 (2): 134–138. 31. Kamarulzaman A, Altice FL. Challenges in managing HIV in people who use drugs. CurrOpin Infect Dis 2015; 28 (1): 10–16. Imputation of adverse drug reactions: Causality assessment in hospitals PLOS ONE | DOI:10.1371/journal.pone.0171470 February 6, 2017 10 / 10 http://dx.doi.org/10.2165/11599730-000000000-00000 http://www.ncbi.nlm.nih.gov/pubmed/22720659 http://dx.doi.org/10.4103/0253-7613.125192 http://www.ncbi.nlm.nih.gov/pubmed/24550597 http://dx.doi.org/10.1007/s40264-013-0083-1 http://www.ncbi.nlm.nih.gov/pubmed/23828659 http://dx.doi.org/10.1111/j.1365-2125.2012.04249.x http://www.ncbi.nlm.nih.gov/pubmed/22360774 http://dx.doi.org/10.1007/s40264-012-0014-6 http://dx.doi.org/10.1007/s40264-012-0014-6 http://www.ncbi.nlm.nih.gov/pubmed/23329541 http://dx.doi.org/10.1016/j.jsps.2015.01.010 http://dx.doi.org/10.1016/j.jsps.2015.01.010 http://www.ncbi.nlm.nih.gov/pubmed/27330379 http://dx.doi.org/10.1007/s40264-015-0342-4 http://dx.doi.org/10.1007/s40264-015-0342-4 http://www.ncbi.nlm.nih.gov/pubmed/26323240