According to the definition of systematic review of systematic reviews, it is a means of summarizing current evidence across specialties of the same or very similar intervention, to provide a synthesis of treatment effects.1,2 A systematic review of systematic reviews is also known as an assessment on quality of those reviews in order to build a methodical orientation for future studies. Quality of systematic review methods was assessed using the following criteria that were adapted from different guidelines3-5 (a) Is it unlikely that important relevant studies were missed? (b) Were the in-clusion criteria used to select articles appropriate? (c) Was the assessment of studies reproducible? (d) Were the design and/or methods and/or topic of included studies broadly comparable? (e) How reproducible are the over-all results? (f) Will the results help resource allocation in health care? Each question was answered with “impossible to judge”, “no”, “partly”, or “yes”.6
To assess the qualitative or quantitative synthesis of a systematic review, a number of checklists were published, such as CONSORT,7 STARD,8 PRISMA9 and so on, which is appropriate to various research methods of only one topic. Therein, PRISMA is widely known to be a functional instrument to qualitatively appraise a systematic review of economic evaluation. PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA not only focuses on the reporting of reviews which evaluates randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions. This aims to help authors improve the reporting of systematic reviews and meta-analyses and also be useful for critical appraisal of published systematic reviews.1,10 Economic evaluations have increased as an advantageous tool for decision making in health care for the last two decades.11,12 Reviews performed almost ten years ago, however, presented noticeable gaps in the quality of methods applied to economic evaluations of health care interventions.6 Nevertheless, quality of these reviews might vary and not follow any principle of assessment. On the other hand, the trend of building a systematic review of systematic reviews has just been concentrated in clinical-effectiveness.13-15 In aspect of cost-effectiveness, it is rare to find a review of reviews, in addition, there is not any study proceed in economic evaluation of hepatitis.
Therefore, the aim of this study is to qualitatively appraise the range of systematic reviews of economic evaluation in case of hepatitis. We analyze quality of these reviews based on PRISMA guidelines including design, methods, results and discussion.
MATERIALS AND METHODS
This study was designed as a systematic review following the PRISMA guidelines to access the quality of systematic reviews studies on health economic evaluation of hepatitis. It was scoped in publications up to 2016 in international journals.
A literature search was conducted with the support of three databases including MEDLINE, SCIENCE DIRECT and COCHRANE LIBRARY. The search string used was: “review” OR (“literature”, “systematic”) AND “economic evaluation” OR (“cost effectiveness”, “cost benefit”, “cost utility”, “cost minimization”) AND “hepatitis” OR (“hep*”, “H?V”). During the search, we applied following limits; search fields: Title, Abstract, Keywords; article type: review; species: humans. In order to get an adequate number of papers as necessity, the publication year was not considered. After the last search in August 2016, 851 published papers and accepted manuscripts were identified.
Inclusion and exclusion criteria
The studies were selected by following a three-stage procedure. At the first stage, some papers were eliminated due to duplications and unavailable titles. At the next stage, we read the title initially, then continued with abstract if the title did not provide sufficient information to allow neither selection nor exclusion. All publications were included if they were recommended in their title and/or abstract as systematic reviews of hepatitis in economic evaluations. Nonetheless, exclusion was also applied in case of publications published in non-English languages and/or not related to humans. After all, the last stage was proceeded to procure eligible papers’ full-text. Rejection was once again made since we were not able to accessed full-text and some of them were not presented as articles but book sections or conference posters.
Data extraction and data analysis
The study data were retrieved by reading the entire article. The information obtained was summarized using an Endnote (Thomson Reuters®) library. Papers were reported in a table with the following fields: record number, name of author(s), publication year, article title and journal. We divided qualified articles into two groups, vaccination and non-vaccination.
Before handing in data, PRISMA checklist was edited by splitting each item into specific particles. Seven parts that should be included (Title, Abstract, Introduction, Methods, Results, Discussion and Funding) are divided into twenty-seven items using as appraisal criteria. Item is ticked as one point if it is mentioned in the publication directly or indirectly through relevant factors. Eligibility criteria for each item were set out by reviewers based on quantity and importance of particles. Two reviewers were independently reviewed full-text for all of articles. After comparison of the results, two reviewers had discussed. Data extracted from checklists was assembled and figured out into primary results. Two branches of analyzed data which were quality of each article and quality of each PRISMA checklist’s item were carried on to accentuate.
Figure 1 shows the flow chart of the selection process with the potentially relevant studies identified, studies retrieved for more detail evaluation, studies included and those excluded. As can be seen, the combined searches found 851 potential publications (448 through Medline, 310 through Science Direct and 93 through The Cochrane Library). Among the 851 references, 30 were duplicates (3.5%). Additionally, after reviewing titles and/or abstracts, 796 publications were eliminated for several reasons. Finally, there are 25 papers included in this study.
General characteristics of publications
In 25 articles we found, Beutels P (2001)16 is the first study published and the most up-to-date studies are Luhnen M et al (2016) and La Torre G et al (2016).17,18 Articles published for the last five years take up half of the total number. Each year from 2004 to 2011, only one or two papers were released focusing on economic evaluations of hepatitis. 2012 and 2015 appeared to be the most remarkable years as having the most papers published – 4 papers per year. In addition, 80% (n=20) were conducted in Europe while the other articles were from North America, South America and Asia, which had three papers, one paper and one paper respectively. Especially, almost all articles published for the last four years belong to Europe.
The additional concerned factor is whether the article is relevant to immunization or not. Only 20% (n=5) focus on vaccination strategy,16,18-21 and La Torre G et al (2016) is the most up-to-date. There is a noticeable number of publications (n=9; 36%) analyzing not only cost- but also clinical-effectiveness.
Quality assessment of publications
Each article is examined carefully using PRISMA checklist. Based on the number of items ticked, those articles could be divided into various quality categories. Up to now, however, there is no standard for classification that has been set out. Therefore, a frame of points is recommended to set them into categories An article acquires the “Excellent” if it has at least 25 points and the “Very bad” if it is under 10 points. The last three categories which are “Bad”, “Intermediate” and “Good” could be gained when holding 10 to 14, 15 to 19 and 20 to 24 ticked items respectively. As presented in Table 2, approximately three-forth of articles (n=18; 72%) are classified in “Intermediate”, following by “Bad” with 5 papers (20%). The categories “Very Bad” and “Good” share the last two papers.
It is dissatisfied that none of papers achieve the “Excellent”. Seventeen is the most popular point gained by seven papers (28%). Noticeably, the only one22 article attained the “Good” grips 22 points. On the other hand, among 27 marked items, there are four items perfectly obtained by all 25 articles, which are “Title”, “Abstract”, “Results of individual studies” and “Summary the evidence”. On the contrary, when being obtained by only one or even no paper, four items “Data items”, “Summary measures”, “Risk of bias across studies” and “Risk of bias within studies” are assessed to be unqualified. The number of articles get point in each item seems to be diffused but mainly oscillating in the range of 15 to 24.
When using PRISMA guidelines for assessing included systematic reviews, another concern about quality of individual seven sections is raised over the quality of items. Similar to item assessment, this research gives a recommendation for section assessment. A section is appraised to be good if it has over half of good items involved. According to this type of assessing, there should be an awareness that quality of sections in individual article is diverse.
In consequence, “Titles” and “Discussion” are the two sections that are accomplished excellently (n=25; 100%). “Abstract”, however, gains 60% (n=15) of articles. This section is divided into twelve particles which should be included in, most of publications, however, achieve only six to nine per twelve. In particular, five articles do not contain “Abstract”, others focus mainly on objectives, participants and results. Only one article includes limitations in “Abstract”, and approximate half of them focus on other particles.
“Introduction” is appraised to be good in 92% (n=23) of studies. Participants, Interventions, Comparisons, Outcomes and Study design (PICOS) is the most essential part that should be included in this section. Nonetheless, only four of twenty-five articles (16%) present a consummate PICOS (23-26). Jones J et al (2009) is the only one that does not contain objectives, and La Torre G et al (2016) mentions inadequate PICOS. The section Methods, which has ten (40%) papers, are unqualified. In Methods, included studies concentrate chiefly on databases sought, years published, and partly on PICOS and key words. However, “Risk of bias” “Summary measures” as well as “Additional analysis” seems to be ignored. “Results” is the lowest-point section with only six papers (24%). While some particles are greatly presented such as “Results of individual studies” and “Numbers of included studies”, others are not paid attention to. It is sorrowful that many articles omit to illustrate selection process by a flow diagram as well as add citations to the text. “Risk of bias” and “Additional analysis” are once again overlooked. Despite being a good section, some particles in Discussion should be recovered. The number of publications presenting the consideration to crucial groups and implications for future research is rather low, which are 56% (n = 14)16-19,21,26-34 and 72% (n = 18)16-19,21-24,28,29,31,33-39 respectively. Limitations should also be considered due to its lack of good publications. In addition, 80% (n = 20) of articles list their source of funding as well as conflict of interests, however, only 16 of them mention the role of funders. The current study has several limitations that should be acknowledged with the regard to interpretation of the findings. Our review was impeded by a number of assumptions and uncertainties. We restricted our assessment of cost-effectiveness to published evidence. Gray or unpublished literature is unlikely to have undergone peer review and its methodological quality cannot be guaranteed. The number of database sought out was more modest than other reviews, and language constraint was also applied, so that there might be a series of publications that was ignored. As criteria in selection and assessment were set up based on subjective authors’ points of view, quality of included studies may not be appraised appropriately. Moreover, during statistical analyzing as well as text presenting, errors could be made clumsily.
This study was designed to be a systematic review of systematic reviews using PRISMA guidelines to bring out an overview of quality of systematic reviews of economic evaluation in hepatitis. During the period of research, PRISMA was proved to be an effective instrument due to its important role in assessing included reviews. As a consequence, even though there are various systematic reviews of economic evaluation published, their qualities are not synchronous and there is a modesty in quantity of studies which was really carried out based on PRISMA. This study makes a suggestion to future researches that rather focus on PRISMA guidelines in study design and report appearance. After searching times and times, we propose that our study is the first systematic review using PRISMA for assessment, leading to face to many difficulties in article classification. A future research is in need to map out a frame for section assessment as well as classify articles into certain categories according to individuals’ quality.