Quality Assessment of Systematic Reviews of Health Economic Evaluations: Pitfalls with the Application of the PRISMA Statement. Comment on Quang et al. (Sys Rev Pharm. 2017;8(1):52-61)

Luhnen, Waffenschmidt, Schwalm, Gerber-Grote, and Hanke: Quality Assessment of Systematic Reviews of Health Economic Evaluations: Pitfalls with the Application of the PRISMA Statement. Comment on Quang et al. (Sys Rev Pharm. 2017;8(1):52-61)

Authors

Quang et al.1 critically appraised systematic reviews of health economic evaluations using the 27 items recommended by the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement.3 The authors state that PRISMA is widely known as an instrument to qualitatively appraise a systematic review of health economic evaluations. Based on the number of items fulfilled, they categorized the quality of each article on a scale with a range of “Very bad” (< 10 points) to “Excellent” (≥ 25 points). The respective threshold values for the 5 categories of quality were defined by the authors themselves.

The methods applied by Quang et al.1 seem to be questionable for the following reasons in particular:

Using the PRISMA checklist to assess the quality of systematic reviews

Although its value for a critical appraisal of the reporting of a systematic review is undoubted, “PRISMA is not intended to be a quality assessment tool and it should not be used as such” as stated in the PRISMA Explanation and Elaboration Document.4

Using the PRISMA checklist in health economics

PRISMA was primarily developed to support the reporting of systematic reviews and meta-analyses of randomized trials.3 Even though several of its criteria are also relevant for systematic reviews of health economic evaluations, others are not applicable, e.g. registration of systematic reviews of health economic evaluations is usually not possible with PROSPERO, as they do not meet the eligibility criterion of dealing with clinical outcomes. A meta-analytical approach for data synthesis is not feasible in most systematic reviews of health economic evaluations,5 as the results of health economic evaluations are, in most cases, not generalizable.6 Furthermore, a standard for the assessment of risk of bias across health economic studies is not established yet. Accordingly, Quang et al.1 found that just one or no articles at all fulfilled 5 out of the 27 items of the PRISMA checklist. A use of PRISMA for the assessment of reporting or the quality of a systematic review of health economic evaluations (without modifications) is inappropriate.

Using summary scores for the categorization of quality

Quang et al.1 derived the score for categorization of quality by counting the number of items fulfilled in each systematic review of health economic evaluations. This suggests that all the items on the PRISMA checklist are equally important for such a categorization. This can hardly be justified, particularly when taking into account that several of the items are not applicable to systematic reviews of health economic evaluations. As it is difficult to justify assigned weights and for reasons of transparency, the use of summary scores to distinguish between high and low quality is generally not recommended.7,8 The fact that the authors did not justify the arbitrarily set thresholds for the different categories of quality makes the evaluation even more questionable.

CONCLUSION

In conclusion, the assessment of quality of systematic reviews of health economic evaluations is challenging. To date, no validated tool exists which is able to consider the particularities of health economic evaluations. We strongly question the use of a summary score based on the items of the PRISMA checklist. With AMSTAR (a measurement tool to assess the methodological quality of systematic reviews) a validated tool is available for the assessment of the methodological quality of systematic reviews.9

Instead of using a summary score for quality assessment, we suggest a separate assessment and discussion of all single items.

REFERENCES

1. 

Quang VT, Phuong HL, Trung QV , authors. Quality assessment in systematic reviews: a literature review of health economic evaluation of hepatitis studies. Sys Rev Pharm. 2017;8(1):52–61

2. 

Luhnen M, Waffenschmidt S, Gerber-Grote A, et al. , authors. Health economic evaluations of sofosbuvir for treatment of chronic hepatitis C: a systematic review. Appl Health Econ Health Policy. 2016;14(5):527–43

3. 

Moher D, Liberati A, Tetzlaff J, et al. , authors. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9

4. 

Liberati A, Altman DG, Tetzlaff J, et al. , authors. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1–34

5. 

Anderson R , author. Systematic reviews of economic evaluations: utility or futility? Health Econ. 2010;19(3):350–64

6. 

Drummond M, Barbieri M, Cook J, et al. , authors. Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report. Value Health. 2009;12(4):409–18

7. 

Higgins JP, Green S , authors. Cochrane handbook for systematic reviews of interventions version 5.1.0. 2011. Available from: www.handbook.cochrane.org. [Accessed July 27, 2017].

8. 

Centre for Reviews and Dissemination , author. CRD’s guidance for undertaking reviews in health care. Available from: http://www.york.ac.uk/media/crd/Systematic_ Reviews.pdf. [Accessed July 27, 2017].

9. 

Shea BJ, Grimshaw JM, Wells GA, et al. , authors. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10