Giulia Manzini, Fabius Hapke, Ian N Hines, Doris Henne-Bruns, Michael Kremer

Giulia Manzini, Fabius Hapke, Doris Henne-Bruns, Michael Kremer, Department of General and Visceral Surgery, University of Ulm, Ulm 89081, Germany

lan N Hines, Department of Nutrition Science, College of Allied Health Sciences, East Carolina University, Greenville, NC 27834, United States

Michael Kremer, Department of General and Visceral Surgery, Hospital of Aarau, Aarau 5000,Switzerland

Abstract

Key words: Rectal cancer; Validity; Meta-analysis; CONSORT checklist; Postoperative chemotherapy; Overall survival

INTRODUCTION

With the growing aging population, the prevalence of rectal cancer is significantly increasing[1]. Seven hundred and four thousand three hundred and seventy-six new cases of rectal cancer (3.9% of all sites) worldwide were registered in 2018 with 310394 deaths (3.2% of all deaths from all sites)[2]. Chemotherapy (CTx) after curative resection for non-metastatic rectal cancer is commonly used in the US, but this is not the case in Europe[3]with its role in improving patient survival remaining controversial[4-6], partly because many studies addressing this topic include also patients with colon cancer regardless of the biological differences of the clinical behavior of these two distinct diseases[7].

In 2012, Petersenet al[3]reported through a Cochrane review the impact of postoperative adjuvant CTx used for curatively resected rectal cancer (Tany, Nany, M0)on overall survival (OS) and disease-free survival (DFS)[3]. The authors identified 21 randomized controlled trials (RCT) reporting OS from a total of 9221 rectal cancer patients with 4854 of these patients which were randomized to adjuvant CTx(treatment arm) with the remaining 4367 patients not receiving adjuvant CTx representing the control arm. The meta-analysis of these studies highlighted a significant reduction in mortality risk (17%) among patients undergoing postoperative CTx as compared to those patients with simple follow-up observation [hazard ratio(HR) = 0.83; 95% confidence interval (CI): 0.76-0.91]. Twenty trials reported DFS with a total of 8530 patients examined. Again, the meta-analysis revealed a decrease in disease recurrence (25%) among patients undergoing adjuvant CTx when compared to the observation only group (HR = 0.75; 95%CI: 0.68-0.83)[3].

奶牛隱性乳腺炎并不會(huì)表現(xiàn)出典型的臨床癥狀，但卻有特有的癥候群。隨著病情進(jìn)一步加重，奶牛的產(chǎn)奶量和泌乳量持續(xù)減少，乳汁中含有的成分發(fā)生明顯變化，乳汁中原來(lái)的弱酸性變成弱堿性，乳汁質(zhì)量普遍下降，容易腐敗變質(zhì)。隱性乳腺炎如果不及時(shí)治療，就會(huì)轉(zhuǎn)變?yōu)榈湫偷娜榉垦祝殡S明顯的臨床癥狀，導(dǎo)致養(yǎng)殖場(chǎng)奶牛淘汰率升高。本次研究該養(yǎng)殖場(chǎng)發(fā)病的23頭患病奶牛并沒(méi)有出現(xiàn)死亡現(xiàn)象。

In the era of “choosing wisely” decisions, we deemed it necessary to reevaluate treatment recommendations for special tumor entities, in particular we aimed to assess the validity of studies on which meta-analysis rely and form the basis for these recommendations. As done for gastric[8]and esophageal cancer[9], the current study examined the validity of those studies within the meta-analysis of Petersenet al[3](2012) which confirmed the benefit of post-operative CTx in rectal cancer. We do not aim to answer the clinical question about the use of adjuvant CTx after radical resection for rectal cancer, as this would imply a more extensive literature research than the critical analysis of a Cochrane review. The purpose is to critically evaluate both the results and the methodology by which the results were derived. It is imperative to offer the patient an evidence based therapy that justifies potential side effects as well as costs.

MATERIALS AND METHODS

The meta-analysis of Petersenet al[3](2012) included a total of 21 studies with the endpoint of OS. Five studies (23.8%) (CCCSGJ[10]1995, Krooket al[11]1991, Quasar[12]2007, Grageet al[13]1981, Hamaguchiet al[14]2011) found a statistically significant advantage in survival in curatively resected patients undergoing adjuvant CTx compared to those undergoing observation (HR ＜ 1 with significant 95%CI because not including the 1 - this means that the graphic representation of the CI in the forest plot of the meta-analysis does not overcome the line of No Effect). The remaining 16 studies (76.2%) did not show statistically significant results. Table 1 lists the 21 studies with number of included patients in each arm as well as the weight of the study and information about statistical significance. Weight reflects the influence (in %) of each study within the overall meta-analysis,i.e., studies with high weight affect the results more than those with low weights with respect to the meta-analysis results.Weighting is determined by type of model, either fixed or random effect model,sample size (largern= more weight), and precision of the estimate (narrower CI =more weight). The data provided in Table 1 is based on the description of the studies on pages 6-9 as well as on the tables from pages 24 to 37 of the original metaanalysis[3]. The total number of included patients was 9411. In the original publication the authors describe 9785 enrolled patients with rectal cancer and, in 9221 of them,data were available for meta-analysis.

In the first part of the results section, we assessed the validity of the three most powerful studies included in the Cochrane review by Petersenet al[3](2012) which found a statistically significant advantage in survival in curatively resected patients with rectal cancer receiving adjuvant CTx compared to patients undergoing observation following surgery. These studies are those of CCCSGJ[10]1995, Krooket al[11]1991, Quasar[12]2007. The assigned weights are 7.6%, 6.8% and 7.0%, respectively.

In the second part of the results section, we performed a second meta-analysis without these aforementioned three studies (n= 18, with a total of 7255 patients, 3459 in the control and 3796 intervention group, respectively), and finally we present the results of a third meta-analysis with all five statistically significant studies confirming the survival advantage for patients treated with postoperative adjuvant CTx excluded(n= 16, with a total of 6917 patients, 3293 in the control and 3624 in the intervention group, respectively). In this last case, only statistically non-significant studies were included in the meta-analysis.

Selection of the studies and assessment of their validity

As showed in Figure 1 and as previously described in detail in another publication[8],we selected three studies with the greatest power as weighted by the original authors which supported post-surgical CTx treatment among all included studies (n= 21)with endpoint overall survival: CCCSGJ[10]1995, Krooket al[11]1991, Quasar[12]2007.The assigned weights were 7.6%, 6.8% and 7%, respectively. We then utilized the CONSORT checklist to assess the validity of these studies[15]. Two independent review authors (GM and FH) then examined the validity of these three publications.

Meta-analysis

We performed a meta-analysis using R excluding the three analyzed studies discussed above (n= 18) and compared these results with those of the original meta-analysis comprising the entire 21 studies. Next, all single studies with a statistically significant benefit of post-operative CTx after curative resection of rectal cancer were removed and a third meta-analysis with the remaining 16 studies was performed. The metaanalysis were performed with R, version 3.2.0, with the package “meta” (http://www.r-project.org/foundation).

Table 1 Sample size and significance of the 21 studies included in the Cochrane meta-analysis

RESULTS

Assessment of study validity

Table 1 provides an overview of the 21 studies of the original meta-analysis focusing on sample size, weight and statistical significance. As previously described in the methods section, this table is based on the data provided in the description of the studies on pages 6-9 as well as the tables from page 24 to 37 of the original metaanalysis[3]. According to these, the total number of included patients is 9411. In the original publication, the authors report 9785 enrolled patients with rectal cancer and in 9221 of them, data were available for meta-analysis. Table 2 presents a summary of the three analyzed papers described in the methods. Table 3 summarizes the items present in the CONSORT checklist[15]and how the studies address each evaluated component. The results are reported for each of the three included studies. In this section, we describe the issues identified through use of the CONSORT checklist evaluation.

Regarding the CCCSGJ[10]study (1995), validity criteria were not met in 14 of 32 items (43.75%) while five were not applicable. 1004 patients from 140 centers over 2 years were randomized to one of the three arms with 98 assessed as non-eligible and not further analyzed. This causes a loss of the balance in the three groups used within the randomization process. In the author's power calculation, 310 patients were needed in each of the three groups. Included in the final study were 316, 297 and 293,respectively. As the randomization procedure is not described in detail beyond mention of use of the envelope method, it is not possible to know if the allocation concealment was maintained or not. The absence of blinding limits the possibility to correctly interpret the results of the study because difference between control and intervention group may be caused by placebo effect.

Figure 1 Four steps to the analysis of validity of a systematic review. We identified the endpoint of interest(overall survival) and selected the three most powerful studies addressing this endpoint based on the assigned weights from the authors of the systematic review as these studies contributed essentially to the positive result of the systematic review. We finally assessed the validity of these studies by using the CONSORT checklist.

In the study of Krooket al[11](1991), we identified poor validity in 15 of the 32 items of the CONSORT checklist (47%). Five items were not applicable. Specifically, the control group was different compared to the standard control group used in the other studies included in the Cochrane meta-analysis (i.e., surgery alone) with exception of the study by Bossetet al[16]2006. In the study of Krooket al[11], two groups were compared: Surgery plus adjuvant radiotherapy (n= 100)vssurgery plus adjuvant radiochemotherapy (n= 104), whereas in the study of Bossetet al[16], 505 patients that received preoperative radiotherapy or radiochemotherapy were compared with 506 patients receiving preoperative radiotherapy and postoperative CTx or preoperative radiochemotherapy and postoperative CTx. In the study of Krooket al[11]patients included were stratified by operation, extent of invasion, nodal involvement and time to study entry and then randomly assigned to the control or intervention group. It is thus not clear if this design reflects a randomization by strata. Moreover, information about the randomization process is absent as is a power calculation. It is not possible to understand if sample size is high enough, as this should be calculated based on the primary endpoint, which is not clearly defined. Several endpoints are listed (time to local recurrence or metastasis, local recurrence rates and metastasis, and survival).Not all randomized patients were analyzed (209 patients were enrolled in the study,only 205 analyzed). Likewise, no discussion of use of neoadjuvant treatment, if utilized, is described.

In the Quasar study[12](2007), validity criteria were not met in 6 of 33 items (18.2%).Four items were not applicable. A pragmatic design for trial organization was adopted, with clinical teams dividing patients as having either a clear or an unclear indication for adjuvant CTx. This means that the indication for CTx was decided by each clinician after consultation with the patient. This is a source of bias. Moreover,use of the minimization method does not allow allocation concealment to be maintained. This study is a pragmatic controlled trial rather than a randomized controlled trial. Blinding could not be possible in these studies, as the control group failed to receive any type of treatment. Additionally, patients with colon and rectal cancer were analyzed together.

In the study by Krooket al[11]only patients with high risk rectal carcinoma were included. This was defined as the histological presence of an indicator associated with poor prognosis [e.g., perirectal fat invasion (T3), adjacent organ involvement (T4), or regional lymph node metastasis (N1 or N2)]. In the Quasar trial[12], patients with low risk of recurrence were included. These two studies analyzed different subgroups of patients and the results cannot therefore be compared.

Meta-analysis

Figure 2 shows meta-analysis results when the three individually analyzed studies were removed, leaving a total of 18 studies included. Two studies (Grageet al[13]1981,Hamaguchiet al[14]2011) showed statistically significant result in favor of post-surgical CTx following curative resection of rectal cancer. The other sixteen included studies were not statistically significant. The modified meta-analysis estimate had an HR of 0.87 with a 95%CI: 0.79-0.96. The original meta-analysis showed an HR of 0.83 with 95%CI: 0.76-0.91. Removal of the three studies did not significantly change the result of the original meta-analysis.

Table 2 Summary of the three analyzed studies

A second meta-analysis was conducted (Figure 3) without the two studies[13,14]which showed a positive, statistically significant result. Together, the exclusion of the five studies that showed a statistically significant results by themselves still resulted in a statistically significant result (HR = 0.90, 95%CI: 0.83-0.97) with improved survival in patients receiving adjuvant CTx after resection of rectal cancer compared to patients with surgery alone, when a new meta analysis was performed.

DISCUSSION

The present study aimed to assess the validity of the three most powerful studies(CCCSGJ[10], Krooket al[11], Quasar[12]) included in the meta-analysis by Petersenet al[3]2012 supporting the survival advantage of adjuvant CTx which lends support to the results of enhanced survival with post-operative adjuvant CTx after surgical resection for rectal cancer. Rather than to answer the clinical question about the use of adjuvant CTx after radical resection for rectal cancer, as this would imply a more extensive literature research than the solely critical analysis of a Cochrane review, this work focus on the critical evaluation of the methodology used in the analyzed RCTs to achieve results, as bias endangers the validity of studies and needs to be evaluated. To this end, three studies which contributed the greatest to the findings of the Petersen meta-analysis were evaluated for validity using the standardized CONSORT checklist. We demonstrated that these three studies lack validity. Firstly, we would like to summarize the main and the specific problems found in the three critically revised studies. The main common problems among the three studies were the randomization procedure, the lack of a power calculation or when done this was not respected. Additionally, the absence of blinded, placebo-controlled study design limits the soundness of three of the studies and consequently the overall conclusionsof the review. In the absence of placebo control, one cannot differentiate between specific pharmacological and placebo effects. Placebo effect is defined as the“response of a subject to a substance or any procedure known to be without specific therapeutic effect for the condition being treated”[17]. Patients assigned to the control group often experience disappointment when they expect to be treated. Furthermore,lack of concealment of treatment allocation prevents the randomization process leading to conscious or subconscious bias[18].

Table 3 Assessment of validity of the analyzed studies according to the CONSORT checklist

Figure 2 Meta-analysis of n = 18 studies after the exclusion of the three analyzed studies. Each study included in the meta-analysis is represented on the forest plot by a box [put either in the region of favors (T) or favors (C) of the graphic according to the hazard ratio] and a line which indicates the 95% confidence interval.The pooled hazard ratio of the meta-analysis is represented by a diamond. In this case the estimate of overall effect is in the region which favors the treatment [favors(T)]. HR: Hazard ratio; CI: Confidence interval; W(fixed): Weight assigned to the study by using a fixed effect model; W(random): Weight assigned to the study by using a random effect model; Favors (T): Result in favor of the treatment (intervention) group (i.e., adjuvant chemotherapy after curative surgery); Favors(C): Result in favor of the control group (no adjuvant chemotherapy after surgery).

In the meta-analysis of Petersenet al[3]inclusion criteria for the studies were RCTs comparing patients receiving radical surgery for non-metastatic rectal cancer (Tany,Nany, M0) who did not receive adjuvant CTx with those receiving postoperative CTx regimen of any kind[3]. This has as consequence that the included studies are heterogeneous regarding inclusion criteria, making them difficult to compare, as different subgroups of patients were analyzed. In the study by Krooket al[11], only patients with high risk rectal carcinoma were included. This was defined as the presence in the histology of an indicator of disease progression or expansion outside the original location [invasion of perirectal fat (T3), extension to adjacent organs (T4),or metastasis to regional lymph nodes (N1 or N2)]. In the Quasar trial[12], patients with low risk of recurrence were included. Moreover, in the study of Krooket al[11], a control group treated only with surgery followed by observation does not exist. In this trial, two intervention groups are present (surgery plus radiotherapyvssurgery plus radiochemotherapy). Due to these issues, the studies by Krooket al[11]and Bossetet al[16]should not be included in the Cochrane review, as their control group differs from the control group (surgery alone) of the other 19 studies.

Regarding the single studies, some problems should be mentioned. In the CCCSGJ study[10], patients were first randomized and then assessed for eligibility, which is a methodological error because, in this way, the balance obtained through randomization is lost. Ninety-eight randomized patients were excluded because they did not meet the inclusion criteria. This prevents an intention to treat analysis from being performed.

In the study by Krooket al[11], the primary endpoint on which sample size should be calculated, is not clearly defined. Although the results seem to be clinically relevant,they probably lack sufficient power to provide usable conclusions.

The Quasar study is a pragmatic controlled trial and not a RCT. This contrasts with the declared inclusion criteria by the author of the Cochrane meta-analysis (only RCT). Additionally, in this study, a minimization method is used. Minimization[19-22]refers to a type of dynamic allocation, where the subject's treatment regimen is dictated by the evaluation of the potential imbalance of covariates that would result if the patient were assigned to the treatment or the control group[23]. This design seeks to balance patient numbers over a large group of pre-specified prognostic factors at once. Minimization determines the group a prospective subject would be assigned and consequently the allocation concealment is impossible[24]. The European Medicines Agency's Committee[25]states that “dynamic allocation is strongly discouraged”.

Figure 3 Meta-analysis of n = 16 studies after the exclusion of all studies which found a statistically significant survival advantage in the experimental group. Each study included in the meta-analysis is represented on the forest plot by a box [put either in the region of favors (T) or favors (C) of the graphic according to the hazard ratio] and a line which indicates the 95% confidence interval. The pooled hazard ratio of the meta-analysis is represented by a diamond. In this case the estimate of overall effect is in the region which favors the treatment [favors (T)]. HR: Hazard ratio; CI: Confidence interval; W(fixed): Weight assigned to the study by using a fixed effect model; W(random): Weight assigned to the study by using a random effect model; Favors (T): Result in favor of the treatment (intervention) group(i.e., adjuvant chemotherapy after curative surgery); Favors(C): Result in favor of the control group (no adjuvant chemotherapy after surgery).

Regarding the two meta-analysis performed after exclusion of the three analyzed studies, as well as all five studies showing a survival benefit of adjuvant CTx, both confirmed the advantage in survival in patients receiving adjuvant CTx after curative surgery. In the last case, none of the single included studies (n= 16) could demonstrate this benefit. This is similar to previous evaluations by our group where postsurgical CTx improved survival in patients with gastric cancer[8]. Similar metaanalyses where overall significance exceeded individual study results are present in the literature[26]and are critically examined[27]. Meta-analyses provide for larger sample sizes by combining many individual studies, thereby enhancing the power to detect differences[28,29]. The findings here and in other studies again demonstrates the critical need of including specific studies which meet high level criteria for inclusion.On the other hand, meta-analysis will not always compensate for the limitations of individual studies. Bias can still exist within the single studies included despite sound meta-analysis techniques. Meta-analysis increase precision, but accuracy of overall findings hinges upon the individual studies included with their potential faults (e.g.bias), potentially significantly affecting the overall outcome[30]. Study inclusion criteria is only one of several questions that can seriously affect quantitative results and qualitative conclusions derived from the analysis: Measuring publication bias,quantifying risk of bias for specific domains, appropriate statistical techniques for pooled data, and the use of unpublished literature are further important themes[30]. In particular, as the inclusion criteria for trial selection in the Petersen meta-analysis were so wide (Tany, Nany, M0), it is difficult to evaluate the clinical relevance. The actual German S3 guideline for treatment of rectal cancer[31]generally recommends neoadjuvant radiochemotherapy or short radiotherapy for UICC stage II and III middle and low rectal cancer and for T4 tumors with contact to the mesorectal fascia and near the sphincter. T1/2 tumors and limited or uncertain lymph node metastasis and T3a/b tumors found in the middle rectum and small perirectal infiltration assessed by standard MRI (T3a ＜ 1 mm; T3b 1-5 mm) without involvement of surrounding lymph nodes or extramural vessel invasion can be treated by primary surgery as recently reported[31]. Rectal carcinomas found in the upper third are generally resected without any secondary treatment. Neoadjuvant therapy, in those instances, should be utilized only in risk situations (T4, positive circumferential resection margin, clearly lymph node +)[31]. Depending on the final pathological TNM stage after surgery, adjuvant therapy should be determined by tumor boards. Cases involving T1/2 N0 rectal cancer do not warrant no adjuvant therapy based on these results. An adjuvant radiochemotherapy is indicated in presence of risk factors for recurrence (for example R1-resection, intraoperative tumor rupture, poor quality of total mesorectal excision, pT4, pT3c/d, pN2, pT3 in lower rectum)[31].

A limitation of our study is that the focus was put on the statistical and methodological aspect considering three studies of a Cochrane review rather than to answer the clinical question regarding the survival benefit of adjuvant CTx for rectal cancer operated for cure. The critical analysis of larger amount of studies and metaanalysis is mandatory in order to assess the clinical relevance of adjuvant CTx.

However, our purpose was to inform the reader of the importance of critical interpretation of the results of RCTs and meta-analysis as well as the selection process of the studies to be included in such reports, as this cannot overcome bias present in the single included studies. Most important of all, the safety of the patient with a clear benefit of the suggested treatment should remain the main goal of medical decision making.

Implications for practice

Following the results of the Cochrane review, post-surgical CTx was shown to improve survival in patients after curative resection of rectal cancer. Importantly, it is noted that a portion of the reviewed studies contain limitations which impair a definitive assessment and recommendation development and that final guidance on this topic should be delayed until additional properly designed studies are conducted.The three analyzed studies which were of highest weight in the Cochrane review had insufficient validity, based on key study features, to be included in a meta-analysis.Systematic review and/or meta-analysis quality is directly dependent on the quality of the studies included, as studies with bias risk can consequentially deteriorate the validity of the entire analysis[29]. As results of meta-analysis are often taken into big consideration in the medical community, the literature needs to be reevaluated in order to avoid unnecessary side effects for the patients as well as unnecessary costs for the health care systems.

ARTICLE HIGHLIGHTS

Research background

The use of chemotherapy (CTx) after curative surgery for non-metastatic rectal cancer and its role in improving patient survival remains controversial.

Research motivation

In 2012, Petersenet al[3]reported in a Cochrane review the effect of postoperative adjuvant CTx following curatively resected rectal cancer (Tany, Nany, M0) on overall survival. The authors identified 21 randomized controlled trials (RCT) reporting overall survival as primary endpoint.The meta-analysis of these RCTs showed a significant reduction in the risk of death (17%) among patients undergoing postoperative CTx as compared to those undergoing observation (hazard ratio = 0.83; 95% confidence interval: 0.76-0.91).

Research objectives

We aimed to analyze the quality of the data supporting the advantage of adjuvant CTx after surgery for rectal cancer.

Research methods

Using the CONSORT Checklist, the current analysis evaluated the validity of the three most powerful studies reviewed and analyzed within the Cochrane review by Petersenet al[3]2012 which support the survival benefit of adjuvant CTx.

Research results

The detailed analysis of the three most powerful studies highlighted inconsistencies including inappropriate answers in up to 47% of the items of the CONSORT checklist. Inadequate or unclear randomization without allocation concealment, missing blinded set-up, absence of intention-to-treat analysis and omission of sample size calculation were the most common findings.

Research conclusions

We suggest a more critical appraisal regarding the validity of single RCTs, as these studies are included in meta-analysis that are the basis for guidelines.

Research perspectives

As CTx has several side effects for the patient and generates costs for the health system, it should be used only if its benefit is real.