Author Archive

ISCTM 2012 Scientific Meeting – Metric-Based Measurement to Assess Cognitive Outcome in AD Trials Summary

22 Feb 2012

The Working Group titled “Metric-Based Measurement to Assess Cognitive Outcome in AD Trials” (Working Group 7) convened recently at the 8th Annual ISCTM meeting in Washington, DC to continue work towards defining a new cognitive endpoint for Alzheimer’s disease clinical trials as an alternative to the ADAS-Cog.

The overall goals of the Working Group are to:

2012 Autumn: Options and Methods to Improve Cognitive Assessment in Clinical Trials of AD and Its Precursors Workshop

Co-chairs: Holly Posner, MD; Phil Harvey, PhD

Objectives:

Understand the challenges in assessment of cognitive enhancement studies in AD and its precursors.  Critically evaluate whether sophisticated statistical techniques are adequate to overcome psychometric challenges in existing instruments. Propose alternatives in the event that it appears as if current data sets cannot be salvaged. Evaluate new approaches to assessing outcome using new performance-based  strategies.  Participants will include Terry Goldberg, PhD, (LIJ Hofstra School of Medicine),  Dan Mungas, PhD, (UCI Davis), Yaakov Stern, PhD, (Columbia University), Veronica Logovinsky (Eisai), and Allitia DiBernardo, JnJ.

Background: The Alzheimer’s Disease Assessment Scale (ADAS-cog) is a composite of several neuropsychological tests and is the de facto cognitive outcome standard in AD trials.  This scale was rationally derived on the basis of knowledge and measurement techniques that are now close to 30 years old and many of the items lack sensitivity to a broad range of impairments in the AD spectrum.  Thus, the instrument typically generates data that is insensitive to impairment in milder cases and hence is handicapped in its ability to detect changes.

Multiple pharmaceutical companies have re-analyzed ADAS-cog and other AD/ MCI/ Prodromal AD endpoints in an effort to improve the sensitivity of the outcome measure to detect improvement from a pharmaceutical or biologic intervention. Some companies have used purely statistical methods and others have also used psychometric analysis, like IRT. Most have also attempted to combine multiple outcomes measures, such as MMSE, CDR, and other scales along with ADAS-Cog data. A sampling of these efforts will be presented.

There is an understandable sense of urgency regarding solutions to the low sensitivity issue and an understandable reluctance to wait for alternative instrumentation to be developed. Since very large and expensive drug development decisions can rest on the quality of outcomes assessments, key stakeholders require a thorough understanding of whether statistical techniques can overcome the intrinsic limitations of existing assessment methods.  At the same time, if statistics cannot resurrect data collected with older methods, then new strategies must be developed or alternative regulatory approaches will be needed.  Critical evaluation of the competing strategies of “charge forward now” vs. “develop something that works” is required. Further there are recent developments in co-primary assessments in other conditions, such as schizophrenia, that have been applied with considerable success in mildly impaired cases.

Potential deliverable: we plan on producing a white paper, starting out by laying out the issues:

·         Questionable sensitivity

·         Possible inadequate coverage

·         Urgent need to assess less impaired patients

·         The  tension of the need to run trials vs. the need to have an outcome measure that work.

This would include a review of the publicly available information on salvaging  ADAS-cog data with statistics.  It would also outline previous efforts at consensus-based outcomes development, including evaluation of the time-line for this.  Finally, the white paper could comment on the regulatory perspective and whether some thinking, like we have seen in schizophrenia, on performance based measures of everyday functioning could be a substitute for current cognitive measures supplemented by clinical judgments and informant reports.

2012 Autumn: Neurocognition and Social Cognition as Endpoints for Clinical Trials

2012 Autumn Conference – Concurrent Workshop Sessions

Objectives: Understand the challenges in assessment of cognitive enhancement studies in AD and its precursors.  Critically evaluate whether sophisticated statistical techniques are adequate to overcome psychometric challenges in existing instruments. Propose alternatives in the event that it appears as if current data sets cannot be salvaged. Evaluate new approaches to assessing outcome using new performance-based  strategies.

Background: The Alzheimer’s Disease Assessment Scale (ADAS-cog) is a composite of several neuropsychological tests and is the de facto cognitive outcome standard in AD trials.  This scale was rationally derived on the basis of knowledge and measurement techniques that are now close to 30 years old and many of the items lack sensitivity to a broad range of impairments in the AD spectrum.  Thus, the instrument typically generates data that is insensitive to impairment in milder cases and hence is handicapped in its ability to detect changes.

Multiple pharmaceutical companies have re-analyzed ADAS-cog and other AD/ MCI/ Prodromal AD endpoints in an effort to improve the sensitivity of the outcome measure to detect improvement from a pharmaceutical or biologic intervention. Some companies have used purely statistical methods and others have also used psychometric analysis, like IRT. Most have also attempted to combine multiple outcomes measures, such as MMSE, CDR, and other scales along with ADAS-Cog data. A sampling of these efforts will be presented.

There is an understandable sense of urgency regarding solutions to the low sensitivity issue and an understandable reluctance to wait for alternative instrumentation to be developed. Since very large and expensive drug development decisions can rest on the quality of outcomes assessments, key stakeholders require a thorough understanding of whether statistical techniques can overcome the intrinsic limitations of existing assessment methods.  At the same time, if statistics cannot resurrect data collected with older methods, then new strategies must be developed or alternative regulatory approaches will be needed.  Critical evaluation of the competing strategies of “charge forward now” vs. “develop something that works” is required. Further there are recent developments in co-primary assessments in other conditions, such as schizophrenia, that have been applied with considerable success in mildly impaired cases.

Potential deliverable: we plan on producing a white paper, starting out by laying out the issues:

·         Questionable sensitivity
·         Possible inadequate coverage
·         Urgent need to assess less impaired patients
·         The  tension of the need to run trials vs. the need to have an outcome measure that works.

This would include a review of the publicly available information on salvaging  ADAS-cog data with statistics.  It would also outline previous efforts at consensus-based outcomes development, including evaluation of the time-line for this.  Finally, the white paper could comment on the regulatory perspective and whether some thinking, like we have seen in schizophrenia, on performance based measures of everyday functioning could be a substitute for current cognitive measures supplemented by clinical judgments and informant reports.

WS8: Approaches to Improve Signal Detection in Studies Comparing Drug and Placebo Workshop
Chair: Amir Kalali, MD

2012 Autumn: Statistical, Clinical, Payor and Regulatory Perspectives on Dimensions That Define the Spectrum of Efficacy and Effectiveness

Co-chairs:  Larry Alphs, MD, PhD; Nina R. Schooler, PhD; Ginger Haynes, PhD

Brief Description

Demands for personalized medicine, comparative effectiveness, and cost control are increasingly driving sponsors to consider both efficacy (explanatory) and effectiveness (pragmatic) trials to be part of a complete drug development program.  Yet, there is much confusion in the field about the differences between these trials including their definition, requirements, design and interpretation.  Further, within each approach, a wide range of clinical trial designs (each with its own strengths and weaknesses) can be used to address these considerations.  Effectiveness trials are particularly poorly understood. Few designs completely address the various types of effectiveness questions that can be raised.  Indeed, essential elements of trials claiming to be ‘effectiveness’ are often not clearly reported and few broadly established standards for qualifying trials as either ‘effectiveness’ or ‘efficacy’ exist.  Not infrequently meta-analyses use results from an explanatory approach and draw conclusions to address questions that require a pragmatic design.  Commonly agreed upon regulatory, provider and publication quality standards for these two approaches are particularly needful.  The lack of clear standards within the field has led to a plethora of generally inadequate local standards that vary by country and by publication. Thus, even when studies are available, a clear, coherent body of work is not being developed and the overall quality of clinical trial programs for effectiveness questions is diminished.  This session will attempt to clarify issues inherent in conducting effectiveness trials and provide initial solutions to address them.

 Goals 

The goals of this session are to:

1. Increase the understanding of the difference in efficacy (explanatory) and effectiveness (pragmatic) trials, clarifying questions of clinical, regulatory and provider stakeholders that require pragmatic designs
2. Clarify the criteria that should be addressed when designing effectiveness trials and identify standards to assess whether such trials have been well-designed and well-conducted
3. Provide initial solutions to address these needs

Stratified Medicine and Targeted Clinical Trials for Alzheimer’s Disease Drug Development in Light of Recent Phase III Trial Results: Example ApoE Genotype

Co-Chairs: Lon S. Schneider, MD; Terry Goldberg, PhD; Larry Ereshefsky, PharmD, BCPP 

The purpose of this panel is to provide audience members with a strong scientific and economic rationale for considering stratified approaches to clinical trials and in particular using APOE as a stratification variable in Alzheimer’s disease related trials. In a presentation entitled, “Stratified medicine for AD drug development,” Dr. Mark R. Trusheim will provide an overview of and an argument for use of stratified medicine techniques in clinical trial design, He will discuss a broad array of trials, but will focus on bapineuzemab (a monoclonal antibody to beta amyloid plaques). Using models that in combination address power, biology, and economics and can be manipulated to test a variety of strategies, he will demonstrate why it is scientifically sound and economically imperative to consider this approach. 

In a presentation entitled, “The impact of ApoE4 carriage on clinical trials,” Dr. Terry E. Goldberg will review basic APOE neurobiology, including its roles in lipid transport and amyloid clearance. He will present current novel work on isoform specific effects at the message, protein and biomarker levels. He will also review treatment impact of APOE4 individuals (who carry the risk allele) carriers in the context of clinical trials. Perhaps most strikingly he will review new data on the large protective effects of APOE2 on biomarkers and cognition and the implicatins that these have for clinical trial design and drug development. 

Dr. Eric Reiman will present his experiences in “Using registries and genotyping for trials: What are the implications and pragmatics.” He will address the use of registries and genotyping the Alzheimers Prevention Initiative. He will review his own experiences in accessing such cohorts and will discuss the scientific motivations for stratification by APOE and early intervention in PS1 mutation carriers. More specifically he will discuss proposed or possible trials using APOE genotype as an inclusion requirement for a range of drugs/prevention approaches. This will largely involve the exploitation of a registry of APOE e4 carriers established around Phoenix, aspects of planning or undertaking such a trial, its theoretical reasons, and outcomes including imaging as a potential surrogate or co-primary. 

In a talk entitled, ‘Simulating stratified medicine trials in AD,” Dr. Lon S. Schneider will present results of subanalyses of AD trials based on a post hoc stratification such as APOE 4 carriage have provided interesting and sometimes contradictory results. Although some results might be due to play-of-chance in underpowered subanalyses or to the metrics of the outcomes scales, other results may be due to actual interaction of the drug with the subgroup.  As APOE 4 is the strongest risk factor for AD, and to a large degree is associated with age of onset of AD in late-life it has received particular attention for stratified medicine.  After reviewing the few trials that published outcomes based on APOE 4 carriage, Schneider and colleagues will present simulations derived from a large ADCS and ADNI data base that empirically test the efficiency of developing a drug based on several trials scenarios of APOE 4 carriage. Questions around differential drop outs, heterogeneity of outcomes, and disease severity will be examined.

Discussion:  Next steps what can be done now?

2012 Autumn: Clinical Trials in Autism Spectrum Disorders: Methods, Measurements, Designs and Outcomes

Co-chairs: Douglas Feltner, MD; Joseph Horrigan, MD

This session will provide an overview of the progress toward new treatments for  Autism Spectrum  disorders (ASD). The characteristics of the disorders will be reviewed, along with impending changes in DSM-V and their impact on subject selection and outcome measurement. Results from recent clinical trials will be presented, along with insights into the challenges of designing and running clinical trials for these disorders.  Methods of clinical measurement for use in early and late phase development will be discussed. Established regulatory precedents will be reviewed along with areas of regulatory uncertainty.

ISCTM 2011 Autumn Meeting – Meta-analysis: Methods and Applications to Policy Session Summary

Meta-analysis: Methods and Applications to Policy Frequentist Methods of Meta-analysis
Dr. Michael Borenstein, Biostat, Inc.

Dr. Borenstein gave the foundational presentation of the session. He began by introducing the concept of meta-analysis, a statistical approach to summarize and synthesize multiple studies in a single figure and numerical estimate. In meta-analysis, the effect size, often a risk ratio associated with one hypothesis of interest, is presented for each study and a pooled estimate over all studies is computed. These effect sizes are presented as single values and the related 95% confidence interval estimates. The relative weight of each study’s data in the pooled estimate is dependent on how similar the studies are to each other by design.  

The benefits of meta-analysis are that it provides context, statistical power, and the ability to examine subgroups. Meta-analysis generates a visual summary of the literature with regards to a particular outcome, allowing the reader to easily identify the consistency or inconsistency of the scientific finding. As meta-analysis incorporates data from multiple studies there is greater power to examine the question of interest than in any one study. This visual presentation and related modeling may help identify important subgroups where the primary hypothesis holds or does not hold. These benefits were illustrated using examples from the literature on the impact of naltrexone in opioid use, impact of statin dose on death and myocardial infarction, impact of rosiglitazone on death and myocardial infarction, efficacy and tolerability of first versus second generation antipsychotics, and psychological approaches to treating specific phobias. 

The next portion of the talk illustrated some of the mechanics of meta-analysis including heterogeneous versus homogeneous studies, weighting schemes, and data types. An important decision prior to beginning a meta-analysis is whether the outcome of interest can be considered a fixed or random effect as this will determine how each study will be weighed in the overall estimate. If one can reasonably assume that each included study is modelally identical, such as when a drug company has conducted each of these studies using the same drug under similar protocols in a single patient population, then the fixed effect model is appropriate. If this assumption is not tenable, then a random effects model should be used. 

In concluding, Dr. Borenstein presented and addressed some of the published criticisms of meta-analysis – that it is not appropriate to summarize results in a single number, that meta-analysis compares apples and oranges, and that meta-analysis is often performed poorly. 

Bayesian Methods and Meta-Analysis
Dr. John Seaman, Baylor University

Dr. Seaman provided an overview of the foundation of Bayesian statistics and highlighted how the Bayesian approach to inference has advantages in meta-analysis. The Bayesian principles were illustrated using non-restorative sleep (NRS) examples derived from published biomedical literature. Bayes’ Theorem is the cornerstone of Bayesian inference; it can be simply expressed as the idea that the best model to estimate something is proportional to your prior knowledge about that attribute (prior model) multiplied by what you observe in a particular study (data model). This best model incorporating both sets of information is referred to as the posterior model and allows one to make probabilistic statements about the variable you wish to estimate. An example of such a statement would be, “There is a 0.954 probability that the true rate of male patients experiencing NRS on benzodiazepine-like (BL) drugs exceeds 50%.” To get to the best model/posterior models, assumptions have to be made about the prior and likelihood models. 

The prior model represents all the information that is known about an attribute (for example, rate of NRS in male patients administered BL drugs) prior to executing a new trial. This model may be determined on the basis of previous studies, opinions of subject matter experts, or both. The shape of this curve will be narrower if there is greater certainty about the prior information, or wider (flatter) if there is less certainty. The likelihood model is determined by the characteristics of the observed data in the current trial. Multiplying the prior and likelihood model, and a little algebra/integration, leads to the posterior model for the rate of males experiencing NRS on BL drugs. This model allows one to make statements about the true rate of NRS in male patients administered BL drugs. For example, “The probability is 0.45 that the true rate of NRS in male patients administered BL drugs is between 50% and 75%.” 

Mathematical adjustments to the posterior model lead to the posterior predictive model. The posterior predictive model allows one to make statements about the rate of NRS that will be observed in the next sample of male patients administered BL drugs. For example, “There is a 0.189 probability that the rate of NRS in the next sample of male patients on BL drugs will be 75%.” 

Bayesian inference is designed to easily adapt to the evolution of knowledge that is central to scientific research. Today’s knowledge, captured as the posterior model, can serve as the prior model for the next clinical trial. This accumulation of prior knowledge to update the results in a current trial achieves the goals of meta-analysis. As Stangl and Berry said, “The Bayesian paradigm is synonymous with meta-analysis. For both, the goal is to incorporate all information to predict with as much accuracy as possible some future event, and the uncertainty associated with it, and to present this prediction in a manner that leads to coherent decisions.”  

Dr. Seaman then presented an example of Bayesian meta-analysis, extending the example of estimating rates of NRS in benzodiazepine-like drugs. In this example from (Parmigiani, 2002), there were multiple studies that determined the relative risk (RR) of NRS in BL drugs versus placebo and 13 met the inclusion criteria for meta-analysis.  All studies were published, so sample sizes, observed rates, and RR for each study were available for analysis. For each study, a posterior model was built for the RR; from this model the posterior predictive model for RR of NRS associated with BL drugs and placebo was derived. The posterior model was used to build 98% posterior probability interval estimates of RR of NRS for each study. These are interpreted for a single study as follows: “the probability is 98% that the RR of NRS with BL drugs versus placebo in Study 1 is between 0.6 and 1.1.” Heirarchical Bayesian models combined these individual posterior models, incorporating individual study variability, into a common overall posterior model of the true rate RR of NRS with BL versus placebo. This common overall posterior model was queried to determine the probability that the true RR of BL drugs versus placebo was >1 which would indicate a treatment effect. In this example, the probability that the true RR of BL drugs versus placebo being >1 was 0.01, so the meta-analysis concluded that it was unlikely that BL drugs caused NRS. The overall posterior predictive model of RR for a new trial showed that the probability of the observed RR in that trial exceeding 1 was 0.14. 

Finally, advantages and challenges of the Bayesian approach were summarized. In Bayesian analyses (including meta-analyses), the conclusions are probability statements about the attribute of interest which are easy to understand. The modeling approach precludes having to make an assumption about fixed or random effects and allows one to account for unobserved aspects of variability across published studies. Bayesian meta-analysis does not require the assumption of specific data models (e.g. normality) and allows one to predict outcomes of future trials facilitating sample size estimation and prior specification for future analyses. However, prior model selection and elicitation can be challenging, sensitivity analyses are essential as conclusions may vary according to prior model assumptions, there is no significance cut-off of 0.05, and the methods are less familiar to the consumer. A number of technical references including recommended reading on Bayesian statistics for non-statisticians were also presented. 

Proof in Medicine: The Role of Research Synthesis
Dr. Joel Greenhouse, Carnegie Mellon University

Dr. Greenhouse presented the use of cross-design synthesis methods via work funded by NIMH and AHRQ to assess association between suicidality and antidepressant use in children and adolescents. The goal of medical research is to determine which interventions are most effective (or harmful) for which patients under what circumstances. To reach this goal, it is essential to generate evidence about interventions, both benefits and harms, and to assess and weigh evidence about interventions. The complete evidence about a treatment or intervention typically comes from multiple studies of varied design, from explanatory (randomized placebo-controlled trials) to observational (epidemiological or claims database studies). Ultimately judgments about the evidence must be made, with respect to the reliability and generalizability of each component as well as the overall conclusions regarding a potential association. 

Cross-design synthesis refers to the use of evidence from multiple data sources, experimental and observational, to address similar questions of interest. The FDA meta-analysis of 24 pediatric trials (23 industry-sponsored and 1 NIMH-sponsored) was reviewed and limitations noted. In this application of cross-design synthesis, a new meta-analysis was performed using a subset of the FDA trial database, the sixteen randomized placebo-controlled clinical trials of patients with major depressive disorder, using a different outcome measure, definitive suicidal behavior. Both frequentist and Bayesian methods of meta-analysis were applied to these studies. The 95% CI for RR of definitive suicidal behavior using frequentist meta-analysis was (1.01, 3.96) while the 95% credible interval from the Bayesian meta-analysis was (0.96, 4.12). The posterior probability of the RR exceeding 1 from the Bayesian meta-analysis was 0.96. To increase the generalizability of the evidence set, an analysis of the LifeLink Health Plans Claims database was performed. This database represents over 95 managed care plans in the United States and over 42 million covered lives. The cohort for analysis was restricted to members aged 5-17 between January 1, 1999 and June 30, 2008 with a depression claim based on ICD 9 codes. The index episode was identified as the first claim coded for depression in the time interval for which there was not a previous antidepressant claim in the past 3 months or depression diagnosis in the past 6 months. Data for 12 months prior to the index episode and 6 months following was examined. There were 52, 293 members who met these criteria (LifeLink cohort) and were eligible for analysis. Members in the cohort were classified as having been treated with SSRI, No Psychotherapy (N = 6,872), Other Antidepressant (N = 17,608), No Treatment (N = 6,677), SSRI + Psychotherapy (N = 10,949), or Psychotherapy, No Antidepressant (N = 10,187) according to the presence of relevant claims in the pre-index period. The outcome of definitive suicidal behavior was defined as having a CDC e-code for self-inflicted injury during the 6 month period post index episode. 

The LifeLink cohort was split into two groups, one that was more reflective of the RCTs examined in the FDA database, Restricted Cohort, and those that were not. To be included in the Restricted Cohort, members could not meet any of the following criteria – high risk for suicidal behavior, current schizophrenia diagnosis, current or lifetime history of drug or alcohol dependence, current bipolar I or II diagnosis, currently pregnant or sexually active/no acceptable contraceptive use, or attempted suicide within 7 days of index episode. The Restricted Cohort was comprised of 39,396. The RR of definitive suicidal behavior was computed for the Restricted Cohort, incorporating propensity scoring, age, gender, geographical region, and Medicaid status to adjust for potential confounding differences across treatment groups. Using no treatment as a reference group, the adjusted RR(95% CI) of SSRIs without psychotherapy was 2.3 (1.1–5.1) and the adjusted RR (95% CI) of any antidepressant in the Restricted Cohort was 4.5 (2.3–8.9). Raw RR was also computed for the LifeLink Cohort. The observational RR from the Restricted Cohort was graphed with the meta-analytic results from RCTs with RR of the whole LifeLink Cohort illustrated as a reference line, allowing a grouped snapshot of clinical trial and observational study findings relevant to the research question. Dr. Greenhouse summarized the talk by pointing out the benefits of this method for synthesizing various sources of medical research and generating new hypothesis. He highlighted that cross-design synthesis is inherently interdisciplinary as medical expertise is essential to proper design and conclusions and there is need for sensitivity analyses to understand the dependencies of conclusions upon underlying definitions, judgments and model assumptions. 

Impact of Meta-Analysis for Policy Decisions about Reimbursement-Payer Perspective
Dr. Rhonda Robinson-Beale, Optum Health

Dr. Robinson-Beale presented the current procedures Optum Behavioral Health uses to synthesize the multiple types of published evidence for in their regular review of non-pharmacologic interventions (technologies) for behavioral health to determine plan coverage. They have an established hierarchy of evidence that is used in  scoring relevant publications according to a scientific merit rating scale which grades the study’s research design, measurement of the dependent and variables, participant ascertainment, and generalizability. Each technology is classified as Unproven, Emerging, or Proven based on the magnitude, quality, and consistency of published evidence related to the technology. Inherent in this classification is an assessment of the ability of the technology to be implemented with fidelity. This is especially important in psychotherapeutic interventions. Fidelity is assessed according to the rigor of training, licensing, and re-training applied. Dr. Robinson-Beale concluded with recommendations to researchers as they are preparing non-pharmacologic behavioral interventions for technology assessment – clearly delineate the population who will use the technology, include naturalistic population-based studies, analyze ethnic and racial subpopulations, perform comparative effectiveness analyses with standard established treatments, include cost of implementation, and provide detail on how fidelity was established where appropriate. She encouraged those who fund research of these treatments to establish requirements for integration and standardization of research protocols, including a central database for all studies and standard research parameters, procedures, and methods of assessment. She also recommended that the central database be structured to allow meaningful integration of datasets and the establishment of registries and this database be accessible to payers to promote low cost and informative descriptive analysis and matched controlled comparisons.

2012 Autumn: Precision Medicine in CNS Clinical Trials Workshop

2012 Autumn: Medication Development for Stimulant Dependence Workshop