KT Update

TABLE OF CONTENTS

IPDMA: Individual Patient/Participant Data Meta-Analysis

Marcel Dijkers, PhD, FACRM
Icahn School of Medicine at Mount Sinai

Department of Rehabilitation Medicine

 [ Download PDF version 200kb ]

In a previous article in KT Update (Dijkers, 2014), the development of meta-analysis (MA) was sketched from (1) meta-analysis of individual study findings (e.g., means and standard deviations, correlations) reported in the literature, to (2) meta-analysis of individual subject data harvested from the published literature, to (3) merging of datasets from multiple studies that share subject population (but not the same individuals) and have similar variables, to (4) creation of common data elements (CDEs) to standardize the variables in related studies, to (5) trialists’ collaborations, which offer a mechanism to prospectively combine related studies into a single dataset. Of the above, 2, 3, 4, and 5 all require or offer opportunities for individual patient/participant (data) MA, or IPDMA. The major differences between IPDMA and a secondary analysis of a single-site or multi-site dataset is the need to acquire data or datasets, carefully develop a common coding scheme for all variables, and in the analysis address any issues resulting from the origin of the data (e.g., clustering). How much of this painstaking work needs to be done depends largely on the source of one’s data – differences between 2, 3, 4 or 5 above, essentially.

IPDMA is most closely associated with 3 above: synthesizing information resulting from a re-analysis of the data produced by two or more studies. Alternative names that have been proposed for this process are “mega-analysis” and “integrative data analysis.” These have not caught on; even though one might ask whether for a one-stage IPDMA (explained below), the term “meta” may be appropriate. 

The steps of IPDMA

The steps in this process generally involve the following (Abo-Zaid et al., 2013; Broeze et al., 2010; Burdett & Stewart, 2002; Cooper & Patall, 2009):

1. Develop a question or set of questions that can be answered with existing data.
2. Develop a protocol, with as much detail as is possible, on the hypotheses to guide the analyses that will be performed to answer the questions the nature of the relevant datasets, and the statistical methods to be employed.
3. Systematically search in multiple bibliographic databases and by other means for all published and unpublished research that did result in an applicable dataset.
4. Contact the principal investigator (PI) of each study, and request a copy of the dataset, de-identified or anonymized if needed to protect subjects’ confidential information.
5. Check each dataset carefully for the quality of the data (e.g., duplicate cases, empty fields) and clean it as much as possible, in collaboration with the original PI and her/his staff.
6. Recode all variables in all datasets to make them compatible (e.g., the variable is consistently called “sex” and coded 1=female, 2=male, and 9=unknown).

The next steps differ for a one-stage vs. a two-stage statistical analysis. In a one-stage IPDMA approach, the merged data are analyzed together, using a statistical procedure that takes into account the clustered nature of the cases. In a two-stage approach, the individual datasets (or their subsets—for example, male vs. female subjects) are analyzed in step 1, and the resulting means, correlations, or other descriptive parameters are used in step 2, which combines them using traditional aggregate data (AD) MA methods.

Ergo, for a one-stage IPDMA, the steps are:

7. Merge all of the datasets.
8. Analyze all combined cases, and/or subgroups of interest, to answer the research questions.
9. Report the findings, making sure that complete information on the original studies and the new analyses is provided.

In a two-stage IPDMA, the steps are:

7. Analyze all datasets in parallel, producing means and other values of interest for all cases in each, or for subgroups of interest.
8. Consolidate these parameters as appropriate (with or without bringing in parallel information from studies for which individual patient/participant data are not available) and analyze them as one would in any aggregate data meta-analysis (ADMA).
9. Report the findings, providing comprehensive information on the original studies and the MA analyses performed.

Combinations of these two methods are possible, and not unusual (Abo-Zaid, Sauerbrei, & Riley, 2012). For instance, it is uncommon to be able to acquire all datasets in existence, and performing a two-stage analysis that includes newly calculated information for the studies that are present, and literature-reported parallel information for the missing studies allows one to assess the degree to which “missingness” is likely to bias one’s findings.

The advantages of IPDMA compared to ADMA

Compared with “traditional” ADMA (i.e., meta-analysis that relies on published data summaries only), IPDMA has a number of advantages (Abo-Zaid et al., 2012; Broeze et al., 2010; Cooper & Patall, 2009):

1. The data in the various datasets can be checked extensively. For instance, investigators performing IPDMA of randomized control trials (RCTs) can determine whether randomization was performed properly (Burdett & Stewart, 2002), and if so, whether the treatment and comparator groups are actually balanced in terms of the most important outcome predictor variables. (If not, adjustments can be made in the analysis).
2. By duplicating the original analyses, one can determine whether the published literature contains valid evidence, and if not, publish a correction.
3. If the original authors reported selectively on those subgroups or outcome variables that happened to be statistically significant, or if they made a unit-of-analysis error, or failed to use intent-to-treat analysis, the public record can similarly be corrected.
4. Subgroup analyses that the original studies could not execute (because there were too few cases in the groups to offer sufficient statistical power) can be performed.
5. More complex analytical methods that did not exist at the time the original studies were reported or that the original authors failed to use (e.g., latent growth curve analysis) can be used to provide more precise or more certain answers to old questions, or answers to completely new ones.
6. Standardized analyses, whether traditional or more advanced (e.g., time-to-event analysis), can be conducted across all datasets.
7. One can analyze the effects of within-study and between-study moderators of effect sizes even when subgroups in the joined samples are still considered “small.”

Some methodologists add the following advantage, although the IPDMA in this case starts to meld into a multi-site longitudinal follow-up dataset:

8. By adding to the dataset outcome information not yet available at the time the original investigators published their findings, the long-term outcomes of treatments, diagnostic decisions, or natural recovery can be analyzed.

The disadvantages of IPDMA compared to ADMA

Everything has a price, and this great power of IPDMA comes with some disadvantages compared to ADMA (Abo-Zaid et al., 2012; Cooper & Patall, 2009):

1. IPDMAs take much more effort and time (and thus money) to complete.
2. Incomparable outcome measures used in studies constitute a major problem, at least for one-stage IPDMA.
3. Traditional ADMA can be performed even if individual patient/participant data are not available, by “just” summarizing the published parameters. Even if IPDMA has a much higher power than ADMA, this advantage may be lost if the ADMA can use many more published and unpublished studies.
4. The proper statistical analysis to be used in IPDMAs, especially one-stage IPDMAs, is still a topic of intense discussion, more so than that of group-data level MA.
5. Compared to ADMA, IPDMA brings tricky authorship issues. Are the investigators of the primary studies to be offered the opportunity to become co-investigators on the IPDMA? Offered the chance to “veto” interpretations of the combined data they disagree with?

Effort. A number of reports have indicated how time-intensive IPDMAs are – possibly five times as high as ADMAs (Steinberg et al., 1997). Finding contact information for PIs, and getting them to create and supply a useable deidentified dataset (with documentation sufficient documentation to allow reanalysis) takes much time. However, the increasingly common requirement by funding sources that data be shared, often by uploading them to a proper data-sharing site, will change the difficulty of obtaining data in the next decade. Sharing is now required by the National Institutes of Health and will be required in the near future by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR). It also is required by many journals (e.g. Nature, BMJ); and professional organizations (e.g. the American Psychological Association, APA).

Checking datasets and making the data compatible can take even more time; the experience of the IMPACT TBI researchers (International Mission for Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury) offers some guidance here (Marmarou et al., 2007). The creation of CDEs, and funding agency expectations that these will be used for all key constructs in a particular scientific domain of research, in the future may also reduce the effort needed to merge datasets.

Different measures for the same construct. Discrepant outcome measures used in various studies (e.g., Beck Depression Inventory-II, or BDI-II, vs. Patient Health Questionnaire 9, PHQ-9) to quantify depression symptomatology is not a serious issue for ADMA, but it creates major problems for a one-stage IPDMA: the apples cannot be mixed with the oranges without making some possibly untenable assumptions in turning both into the same fruit. For instance, for both BDI-II and PHQ-9, one could calculate what percentage of the maximum score a subject had, but a sophisticated analysis of a sample for which both measures are available might find a low correlation between the raw scores of the two or the percentage-of-maximum scores. A Rasch analysis might even show a poor correlation of person trait estimates resulting from separate scaling efforts, and the impossibility of creating a “crosswalk.” If the IPDMA uses dichotomous or dichotomized outcomes, such as death, hospital readmission, or a score above the clinical cut-off for depression, these problems do not exist, or can be resolved more easily.

Access to data. Even if the majority of trialists (the opinions of health care researchers who use methods and designs other than clinical trials have not been investigated to any significant degree) agree that research data ought to be publicly available, and state that they personally will make their data available upon request (Rathi et al., 2012; Rathi et al., 2014), the reality is different: various investigators planning or reporting IPDMAs have reported unwillingness of PIs to share their data, with detrimental effects for analyses (Abo-Zaid et al., 2012; Ebrahim et al., 2012; Villain, Dechartres, Boyer, & Ravaud, 2015; Wicherts, Borsboom, Kats, & Molenaar, 2006), up to complete abandonment in at least one instance (Jaspers & Degraeuwe, 2014). In some instances the concerns behind a failure to share are justified—for example, if the dataset has been lost, or, as possible with some case studies and other qualitative research, the data cannot be anonymized so as to maintain patient confidentiality. Another possible reason could be the cost inherent in preparing a deidentified version. (All of these reasons can be put forward, but the complete lack of reply often encountered by candidate IPD meta-analysts is against the collegiality declarations of most professional groups.) Traditional ADMA is not hampered by such problems, unless the research report is unpublished, or the published report is lacking in crucial information that the meta-analyst can obtain only from the primary investigator. As suggested above, ADMA can be used to estimate how much bias is introduced to the IPDMA, because datasets do not go missing “at random”; both analytic strategies, of course, are still subject to publication bias. “Availability bias” is the term that has been coined to describe the problem of selective lack of access to datasets that is unique to IPDMA.

Statistical expertise. Even when the first accounts by statisticians on the proper methods of performing a one-stage IPDMA are at least 20 years old, this is still an area of active research, with reports on methods for handling specific issues being published on a regular basis (Abo-Zaid et al., 2013; Debray, Moons, Ahmed, Koffijberg, & Riley, 2013; Fisher, Copas, Tierney, & Parmar, 2011). The fact that some IPDMA teams do not include a specialized statistician has led to quite a few inappropriate analyses, with the most common issue being a lack of appropriate accounting for the clustering of subjects in one-stage IPDMAs. This is an issue that is not uncommon in primary research involving multiple sites, where the analysis often also omits a proper adjustment. However, there is more of an issue in one-stage IPDMAs: subjects from various studies differ in geographic location, nature of the health care system that served them, specific inclusion/exclusion criteria used in finding them, details of the treatment they received (if intervention research participants), and details in the methods and timing of baseline and outcome data collection methods, for example. Consequently, they cannot be considered a random sample of the population of potential participants, and the analysis should take this into account. (In two-stage IPDMA, clustering is handled “automatically,” whether a fixed or random model is used in the analysis.)

Prospects for IPDMA

While no one has completed a comprehensive inventory of IPDMAs, several partial counts suggest that the number of these studies has increased significantly in each decade, after the first ones were published in the 1970s. While intervention research data are still the most common target of researchers using this technique (especially in cancer care and cardiology, and in areas such as surgery where small RCTs or case series are common), there are now also IPDMAs of diagnostic test assessment, prognosis studies, causal investigations, and other observational research. In 2015, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) group (Stewart et al., 2015) published guidance as to how this type of secondary research is to be reported. IPDMA, because of its many advantages over ADMA, now is considered the “gold standard” for MA (Simmonds, Stewart, & Stewart, 2015). With more examples in the literature, and more datasets potentially available each year, we can expect continuing growth of the number of publications.

In the area of disability and rehabilitation studies, the number of published IPDMAs is still small, and stroke intervention research seems to be overrepresented. However, there are a few interesting studies in other populations (Karyotaki et al., 2015; Kivimaki et al., 2015), and in the last two years a number of IPDMA protocols have been published (e.g., Buffart et al., 2013; Jonkman et al., 2014). Anyone intrigued by the options this “new” IPDMA method proffers would do well to start exploring some of these to see whether the methodology indeed offers the pay-off its enthusiasts claim.

Postscript

In a few instances where the findings of ADMA and IPDMA have been compared, the size and even the direction of the effect sizes have been found to be different. According to Cooper and Patall (2009), the explanation lies in the “ecological fallacy” well known to social science students, and in a related phenomenon called Simpson’s paradox. An hypothetical example of the ecological fallacy might be: at the census tract level, tracts with a high percentage of households over the national median income are also the tracts with more police reports of breaking and entering per 1,000 people. Does this mean the rich do more stealing? Or would we see, if we had individual-level or household-level data, that the poor do more breaking and entering, but the tract-level data are distorted because:

• The poor go steal where the rich live.
• The poor do not report break-ins of their residence to the police.

Cooper and Patall offer a more complete description of the ecological fallacy and its “inverse,” the Simpson paradox, and discuss how they apply to MA. Berlin et al. (2002) discuss how this applies to the difficulty of generalizing from group analysis findings to individual patient/participant benefits of treatment.