Editorial

10.1586/17512433.1.4.471 © 2008 Expert Reviews Ltd ISSN 1751-2433 471www.expert-reviews.com

Measuring the mind: assessing cognitive change in clinical drug trialsExpert Rev. Clin. Pharmacol. 1(4), 471–473 (2008)

John HarrisonAuthor for correspondenceCogState Ltd, PO Box 3223, Warminster, Wiltshire, BA12 8XA, UKTel.: +44 198 521 5286jharrison@cogstate.com

Paul MaruffCogState Ltd, 21 Victoria Street, Melbourne, Victoria 3000, AustraliaTel.: +61 939 664 1300pmaruff@cogstate.com

“This remarkable shift toward interest in measuring cognition has brought with it a number of challenges for us to deal with.

Some encouraging progress has been made, although it is evident that in many respects we can, and must, do better.”

Forgive the pun, but it seems as thougheveryone is thinking about cognition.Until recently, the main concern in CNSdrug development was the extent towhich new compounds might have dele-terious effects on cognition. The oneexception to this was the hoped forimprovement in cognition expected inclinical drug trials of putative therapiesfor Alzheimer’s disease. Currently, thelandscape is very different. The originalconcerns regarding drug safety continueto require that we carefully monitor thecognitive toxicity potential of drugsacross a range of indications (e.g., epi-lepsy, schizophrenia and Parkinson’s dis-ease). What has changed is the interest inmeasuring the potential for cognitiveimprovement. This revolution is mostobvious in the two therapeutic areas ofneurology and psychiatry. In addition tomeasuring the cognitive efficacy of drugsfor use in Alzheimer’s disease, drug-development companies are now rou-tinely seeking evidence of cognitive effi-cacy in Parkinson’s disease, Lewy bodydementia, schizophrenia and attention-deficit hyperactivity disorder. Evidence ofcognitive deficit in these disorders hasbeen available for some time, but it isonly recently that procognitive effects oftreatment have been investigated.

This shift of interest in neurologicaland psychiatric conditions was, perhaps,only a matter of time. After all, these aredisorders of the brain – the seat of cogni-tion – and so it makes sense to supposethat enhancing brain function throughpharmaceutical interventions would havethe potential to enhance cognition.

However, in addition to neurological andpsychiatric conditions, the past couple ofyears have witnessed developing interestin measuring cognitive effects of interven-tions in therapeutic areas not traditionallyconsidered cognitive targets. For example,in addition to psychiatric and neuro-logical conditions, such as Alzheimer’sdisease and schizophrenia, our own expe-rience of supplying the CogState meas-ures of cognitive change has yielded expe-rience in therapeutic indications asdiverse as head injury, endocrine disor-ders, sleep, immune, cardiovascular andrespiratory disorders.

This remarkable shift toward interest inmeasuring cognition has brought with it anumber of challenges for us to deal with.Some encouraging progress has beenmade, although it is evident that in manyrespects we can, and must, do better.

Deficiencies in instrumentationA significant issue exists with respect tothe quality of the instrumentation weuse to measure cognition in clinical drugtrials. Thus far, the temptation has beento borrow tests from the disciplines ofexperimental psychology and neuropsy-chology. Test selection has been sensibly

“…there has also been considerable interest in measuring the impact of

cognitive change in indications that have not traditionally

treated cognition as a therapeutic target.”

472 Expert Rev. Clin. Pharmacol. 1(4), (2008)

Editorial Harrison & Maruff

predicated on evidence that the disease under investigation reli-ably causes cognitive change measured by the adopted tasks.Clearly, this is a helpful starting point, as the effect of drugsdesigned to ameliorate the cognitive deficits of the conditioncan be sensibly measured according to the extent that theyreturn the patient toward normality. For example, the patternof cognitive deficit typically seen in patients with schizophreniaindicates a lower than expected performance in most areas ofcognition [1]. However, the deficits are often most marked inthe domains of executive function and working memory. Con-sequently, these domains make tempting targets for measuringthe precognitive effects of drugs designed to ameliorate the cog-nitive impairment associated with schizophrenia (CIAS). How-ever, the tests we borrow are rarely, if ever, designed to makerepeated assessments of cognition in the same individuals,which, for us, raises a very important issue. Our view is thatthere is a clear distinction between the measurement of cogni-tive impairment and the measurement of cognitive change. Theformer is of key importance to us when wishing to characterizecognition after a brain insult, such as one that might occur afterischemic or hemorrhagic stroke. The emphasis here is on deter-mining the level of impairment seen in a particular patient; tomake this judgement we compare the patient’s performancewith normative data for individuals of a similar age, sex, IQ andethnicity. Our benchmark is, thus, based on a single, cross-sec-tional observation in which we compare the individual withexpected models of normal performance. This is in markedcontrast to our benchmark for measuring cognitive change,where our principal concern is to characterize the patient’s cog-nitive performance relative to a baseline state. Clinically, weoften use traditional cognitive tests to determine whether anintervention has been successful. Our testing regime typicallyextends to measuring cognition before and after interventionand, in these limited circumstances, we can just about justifyemploying the same test. Sometimes, all that is required for aclinical drug trial is for us to make an evaluation of function atthe beginning and end of our study. However, more often, weneed or wish to evaluate cognition far more frequently. Forexample, in exploratory studies we seek to evaluate cognition asoften as three- or four-times within a single day. Repeatedmeasurement is often a required characteristic of confirmatorystudies, especially when we are employing adaptive designs orwishing to analyze the slope of cognitive change. In these cir-cumstances, employing traditional cognitive measures can behazardous. A key problem is that frequent exposure to a cogni-tive measure can often lead to improvements based on thepatient’s growing familiarity with the test. The absence of paral-lel versions often effectively precludes the possibility of trainingstudy participants before a baseline measure is established. Thisis regrettable in that the most acute effects on test experienceand familiarity are a feature of the data collected at baseline.Our experience suggests that a significant proportion of indi-viduals show marked learning effects as a result of a single expo-sure to cognitive tests but, thereafter, the effects of familiarity

are markedly reduced and, in some cases, entirely expunged [2].A major virtue of training is that it helps to reduce improve-ments owing to the study participant’s familiarity with theexperience of being tested, so-called ‘method’ variance.

Thus, the opportunity to train study participants has signifi-cant utility, and employing genuinely equivalent parallel ver-sions of the test can facilitate this. Unfortunately, most of thetests used by clinical psychologists rarely have more than oneversion; therefore, the administration of the test before baselineassessment runs the risk of study participants remembering keydetails with respect to the test materials and their responses.Imagine we are administering a test of word recall and that theword list presented on each occasion is the same as last time. Inthese circumstances, it seems likely that the study participantmight learn the test stimuli ‘by heart’ so they will, probably,show improvements in performance on later testing by virtue ofsimply having learned the test stimuli. Multiple parallel versionscan help control the effect of task familiarity.

In summary, good cognitive tests are stable, reliable instru-ments that yield robust estimates of the study participant’s per-formance. Best practice, therefore, requires us to train study par-ticipants to remove or reduce these effects and, where necessary,employ equivalent parallel forms of our tests.

Closing commentsThe measurement of the cognitive change seen in clinical drugtrials is in a fascinating period of change. Clinical trials of puta-tive drugs for Alzheimer’s disease have tended to feature thewell-known Alzheimer’s Disease Assessment Scale – CognitiveSubscale (ADAS-cog) [3] measure, largely owing to a regulatoryrequirement that this measure be employed. This no longerappears to be the case, with regulators in Europe declaring thatthere is no reference technique [4]. Recent presentations fromUS FDA officials also suggest that they have no expectationthat the ADAS-cog will be employed as a measure of drug effi-cacy. The notion of CIAS has prompted experts and key opin-ion leaders to propose batteries of tests drawn from clinical psy-chology. Various test vendors have proposed the use ofcomputerized test batteries, the content of which mirror thecognitive domains indexed by the Measurement and TreatmentResearch to Improve Cognition in Schizophrenia (MATRICS)Consensus Cognitive Battery. Cognition testing in other neuro-logical and psychiatric indications has also become commonplace, especially Parkinson’s disease dementia, Lewy bodydementia, depression and anxiety. However, there has also beenconsiderable interest in measuring the impact of cognitive changein indications that have not traditionally treated cognition as atherapeutic target.

“…good cognitive tests are stable, reliable instruments that yield robust estimates of the

study participant’s performance.”

www.expert-reviews.com 473

Measuring the mind: assessing cognitive change in clinical drug trials Editorial

A typical response to the need for cognition assessment hasbeen to borrow from the tests traditionally employed as meas-ures of cognitive impairment [5]. It seems that success with aparticular test, or battery of tests, then sets a precedent forfuture studies. However, we would suggest that there is nothingspecial or mystical about measures that have been successfullyemployed. Our own preference would be to guide sponsors bysuggesting the areas of cognition that might, most profitably, beinvestigated. In addition, rather than mandate specific tests, wewould be inclined to suggest that tests be selected according tothe best practice guidelines set out in this editorial and otherreviews [6].

Financial & competing interests disclosureJ Harrison and P Maruff are salaried employees of CogState Ltd andretain option rights over CogState stock. P Maruff has received honorariaand consultancy fees from companies concerned with drug development.J Harrison has received consultancy fees and honoraria from drugdevelopment companies and has additionally received consultancy feesfrom various cognitive testing vendors. The authors have no other relevantaffiliations or financial involvement with any organization or entity witha financial interest in or financial conflict with the subject matter ormaterials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

References1 Heinrichs RW, Zakzanis KK.

Neurocognitive deficit in schizophrenia: a quantitative review of the evidence. Neuropsychology 12, 426–445 (1998).

2 Falleti MG, Maruff P, Collie A, Darby D. Practice effects associated with the repeated assessment of cognitive function using the CogState battery at 10-minute, one week and one month test–retest intervals. J. Clin. Exp. Neuropsychol. 28(7), 1095–1112 (2006).

3 Rosen WG, Mohs RC, Davies KL. A new rating scale for Alzheimer’s disease. Am. J. Psychiat. 141(11), 1356–1364 (1984).

4 Committee for Medicinal Products for Human Use. Guidelines on Medicinal Products for the Treatment of Alzheimer’s Disease and other Dementias. EMEA, London, UK (2007).

5 Harrison JE, Minassian SL, Jenkins L, Black RS, Koller M, Grundman M. A neuropsychological test battery for use in Alzheimer disease clinical trials. Arch. Neurol. 64(9), 1323–1329 (2007).

6 Ferris SH, Lucca U, Mohs R et al. Objective psychometric tests in clinical trials of dementia drugs. Position paper from the International Working Group on Harmonization of Dementia Drug Guidelines. Alzheimer Dis. Assoc. Disord. 11(Suppl. 3), 34–38 (1997).