Jia-Ying Wang, Nan Li, Jun-Feng Wang, Ming-Hui Wang

1Department of Rehabilitation & Acupuncture and Moxibustion, The Affiliated Wuxi People’s Hospital of Nanjing Medical University, Wuxi 214023, China; 2Medical Identification Center, Jiangsu Medicine Association, Nanjing 210000, China;3Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam Public Health Research Institute,Academic Medical Centre, University of Amsterdam, Amsterdam 1105AZ, The Netherlands; 4Medical Laboratory Technology Major, Jiangsu University, Zhenjiang 212013, China.

Background: Stroke is not only high in morbidity and mortality but also poses a great burden of disease and it is also the most reported disease in Chinese medicine systematic reviews.Therefore, the quality of such evidence couldn’t be ignored.This study aims to use a measurement tool to assess systematic reviews (AMSTAR) to assess the methodological qualities of SR/Meta-analyses of Chinese medicine on stroke.Methods: Systematic searching of seven electronic databases and PROSPERO registration platform was carried out.Two researchers separately selected studies, extracted bibliographical characteristics and scored every included study independently after training.Total score and the proportion of each item completion were explored in different subgroup comparisons.Spearman rank correlation and multivariable logistic regression were used to measure the association between bibliographical characteristics and total score or each item.Results: Total average score of AMSTAR 1.0 checklists of 234 systematic reviews/Meta-analyses was 4.47 (95% CI 4.27-4.68) and the qualities were unsatisfied especially in terms of priori setting (2.14%), grey literature inclusion (5.13%), providing a list of excluded studies (2.14%) and conflict of interest (0.00%).No improvement was found in 3 years even after the publication of AMSTAR.Chinese or nonregistered systematic reviews/Meta-analyses showed even worse methodological qualities (P ＜ 0.01).Positive correlation was found between individual items and number of pages, number of authors, research questions,languages or Meta-analyse separately (P ＜ 0.05).Conclusion: The methodological qualities of systematic reviews/Meta-analyses of Chinese medicine on stroke are poor especially Chinese studies，non-registered studies,brief studies and studies without Meta-analyse or cooperation.There is no obvious improvement over these years even after the publication of AMSTAR tool, so it is urgent to promote the use of AMSTAR or develop other efficient methods to control the quantity and monitor the quality in future.

Key words: Stroke, Meta-analysis, Medicine, Chinese traditional, Methodology quality

Background

As the concept of evidence-based medicine is introduced into the field of Chinese medicine (CM) 20 years ago [1], the number of systematic reviews/Metaanalyses (SR/MAs) of Chinese medicine has rapidly increased [2], providing new ideas for the clinical practice of Chinese medicine.However, the quality of its methodology has been a cause for concern [3-5].Low-quality evidence may mislead the application of clinicians and other decision-makers.Stroke is not only high in morbidity and mortality but also poses a great burden of disease [6-8] and it is the most reported disease in Chinese medicine SRs [2]; therefore, the quality of such evidence could not be ignored.Although several instruments have been developed for methodology quality assessment of SR [9], a measurement tool to assess systematic reviews(AMSTAR) [10] is one of the most wide-spread and reliable ones with free access and validity [9, 11-12].

Objection

The objective of this study was to assess the methodological qualities of published systematic reviews of Chinese medicine on stroke by evaluating the compliance with AMSTAR 1.0 checklist and analyze the related reasons.

Methods

The minimum standards of reporting checklist contains details of the experimental design, and statistics, and resources used in this study.

Information sources

This systematic review was performed according to criteria introduced in Cochrane handbook.Studies published in either Chinese or English were systematically searched by 2 reviewers (Wang JY and Li N) independently through the following electronic databases.The databases included PubMed (1978-2015.3), Web of Knowledge (1945-2015.3), China National Knowledge Infrastructure (1980-2015.3),Chinese Biomedical Literature Database (1981-2015.3), WanFang Database (1980-2015.3), Chinese Scientific Journal Database and VIP (1989-2015.3).The Cochrane Database of Systematic Reviews (Issue 3 of 12 ， March 2015) and PROSPERO(https://www.crd.york.ac.uk/PROSPERO/) for all relevant finished or ongoing potential SRs were also searched.The search strategy was formulated using Mesh terms in combination with free words.Details of the search strategy for the English databases are as follows: 1.systematic review; 2.review, systematic; 3.Meta analysis; 4.Meta analyses; 5.1 or 2 or 3 or 4[Title/Abstract]; 6.5 AND Chinese medicine [Text Word].

Details of the search strategy for the Chinese databasesare

The Chinese characters used to perform the search are hereafter stated in Chinese pinyin.For example: (Zhuti(title/abstract) = Xitongpingjia (pinyin) or Xitongzongshu (pinyin) or Meta.The references of literature were also checked for possible identification of relevant studies and electronically inaccessible journal articles were manually retrieved.

Inclusion criteria and exclusion criteria

Because this research was nested in another larger study which gave a full view of the citation situation and general characteristics of the published SR/MAs of traditional Chinese medicine so far, the information source and first step of study selection of this study was the same with the original study.First, we included the SR and/or MAs only, and the interventions referred to all kinds of Chinese medicine therapeutic methods including Chinese herbs, decoction, Chinese patent drugs and Chinese medicine injection, acupuncture and moxibustion.Meanwhile, we excluded repeat studies,studies without full text and non-Chinese and non-English studies as well as reviews, letters, viewpoints,and commentaries.Second, the included studies were categorized into different groups according to different diseases, and this study analyzed the part of Chinese medicine for stroke only.

Study selection and data extraction

Two researchers (Wang JY and Li N) independently finished the selection with Note Express 3 and Endnote 7 software.Ineligible studies were excluded after reading the title and abstract, and the full texts of the remaining studies were scanned for confirmation.Disagreements were resolved through discussion.These 2 reviewers also extracted data independently including bibliographical characteristics such as title, publication year, number of pages, number of authors, language,region （extracted according to the corresponding address）and so on with Microsoft Excel 2010.Any disagreements were compared and resolved by consensus.

Methodological quality assessment

As to the methodology quality assessment, it could be divided into three steps as below.First, the 2 experienced reviewers got assessment training session before the evaluation procedure about the AMSTAR 1.0 checklist to ensure they understand the meaning of every item accurately and achieve the agreement of how to assess after fully discussion.At last all items’numbers and brief explanations were input in a Microsoft Excel sheet.However, some items maybe contained 2 or more than 2 elements and they were separated into different affiliated items.For example,the item 5 of AMSTAR 1.0 checklist was “Was a list of studies (included and excluded) provided?” and the explanation was “A list of included and excluded studies should be provided.Note: acceptable if the excluded studies are referenced.If there is an electronic link to the list but the link is dead, select ‘no’.” When this item was assessed, it was divided into “5a was a list of included studies provided?” and “5b was a list of excluded studies provided?” in the EXCEL sheet and 2 reviewers scored them separately.Only if one article met the requirements of both 5a and 5b, it would be judged as “Yes”.Every “Yes” represented one point, and the total score of AMSTAR 1.0 checklist of every article was the number of the points.

Second, the 2 reviewers (Wang JY and Li N)examined the whole text of every included articles adhered to the complete methodology assessment sheet above in Excel carefully and independently.The relevant description which supported their assessment was required to highlight in the original PDF file of every included articles.After this step was finished, the results of two reviewers were compared and the kappa value was calculated.

Third, any disagreements were resolved through discussion with a third reviewer (Wang JF), and everyone should give their own reasons.Reviewers were not blinded to author or journal.

Statistical analysis

Descriptive statistics were calculated for bibliographical characteristics.Continuous variables were summarized as median and IQR, categorical variables were presented by frequency and percentage.Cohen’s kappa was calculated for each AMSTAR 1.0 item to measure the agreement between 2 reviewers.The criteria for Kappa values was defined as: less than 0, less than chance agreement; 0.01-0.20, slight agreement; 0.21-0.40, fair agreement; 0.41-0.60,moderate agreement; 0.61-0.80, substantial agreement;and 0.81-0.99, almost perfect agreement.The percentage of agreement was also given for information purpose.

For each included article, we evaluated whether each AMSTAR item was reported and counted the total number of AMSTAR items reported (0-11 items).For each item on the AMSTAR checklist, the number and percentage of articles that reported the item were calculated.X-squared test was used to evaluate whether there was significant difference in the proportion of each item reported in subgroups: 1) Chinese versus(v.s.) English; 2) corresponding author from China mainland v.s.other region; 3) registration v.s.non registration; 4) before AMSTAR 1.0 v.s.after AMSTAR 1.0; 5) MA v.s.without MA; 6) positive results v.s.nonpositive results.

Mann-Whitney U test was used to compare the total scores between these subgroups.Spearman rank correlation was used measure the association between continuous bibliographical characteristics (year,number of authors, and number of pages) and total score.After all, we used multivariable logistic regression to explore the associations between all bibliographical characteristics and each AMSTAR 1.0 item.Software R (version 3.1.2) was used to analyze the data.

Results

Results of searching and selection

Figure 1 shows a flow diagram of studies included process according to guidance of the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) statement [13].A total of 17,421 records were identified through database searching including 16,276 Chinese records and 1,145 English records.After checking for duplications within and across databases, 5,062 records’ titles and abstracts were screened and 3910 were found to be relevant potentially.Their full-text articles were assessed for eligibility and 234 SR/MAs related to stroke were included in this study.

Bibliographical characteristics

The median year of publications was 2012 (range 2000-2014, IQR = 4), and most of them (82.05%) were published after the publication of AMSTAR in 2007.The number of authors of each SR varied from 1 to 13(median = 3, IQR = 2), and the number of pages of each SR ranged from 2 to 155 (median = 4, IQR = 2).206(88.03%) SRs were published in Chinese and 28(11.97%) SRs were published in English.Seventy-three(31.20%) SRs reported the effectiveness only and the other 161 (68.80%) SRs reported both effectiveness and safety.The corresponding authors of 224 (95.73%)articles were from China mainland and 10 (4.27%) were from other regions.Almost all the reviews (229/234,97.86%) were not registered and 5 registered reviews were all Cochrane reviews.Most of the reviews(221/234, 94.44%) contained a Meta-analysis.Nearly 90% of the reviews (209/234) showed non-positive results.According to the disease classification, 64(27.35%) clearly indicated cerebral infarction, 9(3.85%) cerebral hemorrhage, 46 (19.66%) stroke sequelae and the remaining 115 (49.15%) general mentioned stroke.In the intervention fifty-one (21.79%)reported acupuncture, 111 (47.44%) used traditional Chinese medicine injections, 18 (7.69%) took Chinese patent drugs and 34 (14.53%) used decoctions.Detailed results for bibliographical characteristics were shown in Table 1 bibliographical characteristics of 234 included studies.

The inter-rater reliability

Figure 1 PRISMA flow diagram of studies included[14].PRISMA, the Preferred Reporting Items of Systematic Reviews and Meta-Analyses.

The inter-rater reliability was high, in most of the items,kappa values showed substantial agreement (item 2:0.72; item 3: 0.62; item 5: 0.75; item 10: 0.67) or almost perfect agreement (item 1: 1.00; item 7: 0.99; item 8:0.86; item 9: 0.83).However, the 2 reviewers had moderate agreement on scoring item 4 (kappa = 0.49)and 6 (kappa = 0.43).We took a further look at these 2 items, and found that the lower kappa in item 4 is due to the discrepancy in rating English articles and item 6 is due to Chinese articles (details can be found in Supplementary Materials Table 1).

Methodology qualities of the SRs/MAs on Chinese medicine for stroke

Overall compliance with the AMSTAR checklist

Overall, compliance with the 11 items was poor.The average number of items reported in all articles (n =234) was 4.47 (95% CI: 4.27-4.68) out of the maximum possible 11 items (Figure 2A).Figure 2B showed the proportion of all articles that adequately met the individual AMSTAR checklist items.Items that were better completed than others were the aggregated study characteristics (item 6, 71.79%), assessment and documentation of study quality (item 7, 72.65%),appropriately forming conclusions based on the quality of the included studies (item 8, 91.88%), and the appropriateness of the methods used to combine the findings of studies (item 9, 91.03%).Compliance was extremely poor for listing study sources of funding(item 11, 0.00%), although some of the SRs reported funding information themselves, none of the SRs reported the funding information of included primary randomized controlled trails.The percentages of reporting were also low in the inclusion of a priori design of the research question(s) and inclusion criteria(item 1, 2.14%), use of the publication status as an inclusion criterion (item 4, 5.13%) and providing a list of included and excluded studies (item 5, 2.14%).While the compliance with item “5a was a list of included studies provided?” was 100% so it mainly caused by poor compliance of “5b was a list of excluded studies provided?”.

Figure 3 showed that from 2000 to 2002 there was an increase in completeness of AMSTAR.After 2002, the average total score of AMSTAR checklist gradually decreased, even after the publication of AMSTAR in 2007 and no improvement was observed.

Association between bibliographical characteristics and methodological quality

Figure 4 showed the adherence to individual AMSTAR items in subgroups.Details of statistical results were shown in Supplementary Materials Table 2.SR/MAs published in English had significant higher proportions than SR/MAs in Chinese in item 1-7 (allP-values ＜0.01) and item 10（P＜ 0.05）.As a result, English articles had significantly higher total scores (P＜ 0.01)than Chinese articles (Supplementary Materials Figure 1).In all the years we evaluated, English articles had higher total score than Chinese articles, and no clear trend was observed (Supplementary Materials Figure 2).The results were quite similar if we divided all the SRs by regions as by language, significant differences were found in item 1-5 (allP-values ＜ 0.01) and item 6,7 (P＜ 0.05).Since all Chinese SRs were written by authors from China, we further investigated the effect of author’s country in English SRs only, and the reporting of all the items were comparable between 2 subgroups(Figure 5A and details of statistical results were shown in Supplementary Materials Table 3).

Table 1 Bibliographical characteristics of 234 included studies.

Registered reviews had significant higher proportions than non-registered reviews in item 1-5 (allP-values ＜0.01) (Figure 4C).Since all registered SRs were published in English, we also compared registration effect in English SRs only, and significant differences(P＜ 0.05) were also found in item 1, 4 and 5 and total score (Figure 5B and details of statistical results were shown in Supplementary Materials Table 3).Metaanalysis was not always performed in a SR.In those SRs including MA, the completeness was significantly better in item 6, 10 and 11 (allP-values ＜ 0.01).At last,SRs leading to positive results had better completeness in item 1-7 (allP-values ＜ 0.01) than non-positive results.However, after the publication of AMSTAR, the reporting of item 7 and 8 (P＜ 0.05) was even worse than before, for other items the publication of AMSTAR had no impact.

Multivariate regression analyses

Figure 6 illustrated the results from logistic regression analyses.The yellow color indicated a positive regression coefficient, and the red color indicated a negative regression coefficient.Dark red or dark yellow indicated a smallerP-value (**,P＜ 0.01, *,P＜ 0.05),and gray color indicated aP-value near 1.

Figure 2 Distribution of total score and overall compliance with each item of AMSTAR checklist.AMSTAR, a measurement tool to assess systematic reviews.

Figure 3 Average total score of AMSTAR in years of 2000-2014.AMSTAR, a measurement tool to assess systematic reviews.

SRs published more recently were less likely to have conducted an assessment of item 7 and item 8.As to research question, SRs which have evaluated both efficiency and safety were better in reporting item 6, 7,8, 10 and had a higher total score.There was significant positive correlation between the number of pages and item 2 and 7, and significant positive correlation was also observed in number of authors and item 2 and 3.SRs with a MA were better in reporting item 9, 10 and had a higher total score than SRs without a MA.Finally,SRs written in English had significant higher total score than Chinese.Whether the SR lead to a positive result did not influence either individual item or total score neither did the region of corresponding author.

Discussion

It has been 20 years since the first systematic review of Chinese medicine was published in 1997 [15].This study is nesting in another larger research which shows the whole picture of SRs/MAs development of Chinese medicine.After we investigated the use of evidence by citation analysis of Chinese medicine SRs/MAs [16]and analyzed the general characteristics of them through bibliographical information, we also wanted to know the methodological qualities of these SRs/MAs, so we chose to use AMSTAR 1.0 tool to appraise them.Due to the large number and different types of studies, we selected stroke as the representative disease for evaluation because it was reported most frequently among interventional Chinese medicine SRs/ MAs based on our previous study [2].

Figure 4 Comparison of proportion of compliance with each item between different languages, regions,registration statuses, before and after AMSTAR publication, with and without MA or positive and non-positive results.AMSTAR, a measurement tool to assess systematic reviews; MA, Meta-analyses.

Figure 5 Comparison of proportion of completeness in each item between different countries and registration status within English articles only

Figure 6 Results of multivariate regression analyses

SR/MA as a major research method of evidencebased medicine should provide reliable evidence for clinical practice and medical decisions making [17-19].However, this study suggests these evidences may not be so reliable due to their methodological flaws.These methodological flaws mainly concentrate on qualitative research methods rather than quantitative analysis methods.Despite the generally low AMSTAR 1.0 total scores of 234 SRs /MAs, comparison of compliance with each item shows that there are significant differences between different domains.Some of the items shows high degree of completion such as the choice of statistical methods (item 8, 91.88%) and cautious conclusion (item 9, 91.03%) while others display very low degree of completion such as item11 conflict of interest (0.00%).

In addition, this study also suggests qualities of SR/MAs in Chinese are much lower than in English.Not only the proportions of seven items’ compliance are significantly lower than that in English (allP-values ＜0.01) according to Figure 4, but also the distribution of total scores of Chinese studies were significantly lower than English studies (P＜ 0.01, Supplementary Materials Figure 1).Moreover, Figure 6 shows there is a significant correlation between language and total score (P＜ 0.05) and the average total score of Chinese SR/MAs has never exceeded English SR/MAs in any year unless English studies is not published in that year(Supplementary Materials Figure 2).Besides, because of the large number of Chinese studies, the single item’s compliance of the overall literatures is often affected.For example, the completion rate of item 4 shown in Figure 2B is only 5.13% mainly due to the lack of gray literature searching in the Chinese SR/MAs (0.00%)while the completion rate of English SR/MAs only remains 42.86% (12/28).

A more interesting phenomenon is that the qualities of SR/MAs have not improved with time or the development of AMSTAR (Figure 3 and Figure 6), and we speculate it may be related to the substantial increase in the number of low-quality Chinese articles in recent years.To further analyze this situation, we compare the published quantities of Chinese and English SR/MAs from 2000 to 2014 (Supplementary Materials Figure 3).It is not hard to find that the number of Chinese publications increases very fast and has become more than seven times of English ones (206/28) until March 2015 while number of English publications remain no more than 6 every year.These large quantities and low qualities Chinese SR/MAs are very likely to lower the overall methodology level.

However, the qualities of registered SR/MAs are much higher than non-registered ones.Figure 4C shows that the completion percentages of item 1 to 9 of registered studied are all higher than that of nonregistered studies and the each statistical difference between item1 to 5 is significant (P＜ 0.01).Since the registered research is all in English, in order to control the influence of the number of Chinese SR/MAs, we conducted a subgroup analysis among English SR/MAs separately to test the impact of registration on the methodological quality of the study.The result shows that the quality of the registered research is still superior(Figure 5B).Besides, the multivariate regression analysis finds that the more authors a SR/MA has, the better methodological quality a SR/MA shows,indicating that effective teamwork can provide a guarantee for the quality of methodologies.

There are still some shortcomings in this study.First the evaluators assess the methodological quality through the original report in the articles, which may not also be equal to the actual situation due to the various reporting quality [20].Second, due to the earlier start of this study, the first version of the AMSTAR checklist was used here instead of the second edition published latest with some adjusted items.

Conclusion

In conclusion, the methodological qualities of SR/MAs of Chinese medicine on stroke are not satisfied especially Chinese SRs/MAs and there has been no trend of gradual improvement in these years or after the publication of AMSTAR checklist.In order to make these evidences more reliable, it is urgent to promote its application or develop other efficient methods or tools such as registration and effective teamwork to control the quantity and guarantee the quality in future.