casp5 assessment of fold recognition target...

CASP5 Assessment of Fold Recognition Target PredictionsLisa N. Kinch,2* James O. Wrabl,2 S. Sri Krishna,1 Indraneel Majumdar,1 Ruslan I. Sadreyev,2 Yuan Qi,1

Jimin Pei,1 Hua Cheng,1 and Nick V. Grishin1,2

1Department of Biochemistry, University of Texas Southwestern Medical Center, Dallas, Texas2 Howard Hughes Medical Institute, University of Texas Southwestern Medical Center, Dallas, Texas

ABSTRACT We present an overview of the fifthround of Critical Assessment of Protein StructurePrediction (CASP5) fold recognition category. Pre-diction models were evaluated by using six differentstructural measures and four different alignmentmeasures, and these scores were compared to thoseassigned manually over a diverse subset of targetdomains. Scores were combined to compare overallperformance of participating groups and to esti-mate rank significance. The methods used by a fewgroups outperformed all other methods in terms ofthe evaluated criteria and could be considered state-of-the-art in structure prediction. We discuss a fewexamples of difficult fold recognition targets tohighlight the progress of ab initio-type methods ondifficult structure analogs and the difficulties ofpredicting multidomain targets and selecting predic-tion models. We also compared the results of manualgroups to those of automatic servers evaluated inparallel by CAFASP, showing that the top perform-ing automated server structure predictions ap-proached those of the best manual predictors.Proteins 2003;53:395–409. © 2003 Wiley-Liss, Inc.

Key words: protein fold prediction; structure com-parison; alignment quality; threading;domain structure; CASP5

INTRODUCTION

We present a detailed report of the CASP5 fold recogni-tion assessment, which included evaluating predictionmodels, quantifying these evaluations in a meaningfulway, and using these measurements to produce a rankingof groups that was subject to various tests of significance.Automation played a crucial role in completion of theanalysis. Our report outlines the resulting evaluationprocedure, the logic behind its development, and theresults of its application to CASP5 fold recognition targetpredictions. On the basis of these results, we highlight thepitfalls and progress of both the top performing predictiongroups and the fold prediction community as a whole.

In addition to defining CASP5 target domain classifica-tions (see Target Classification article in this issue1), theassessment of the CASP5 fold recognition category encom-passed evaluations of both the structural quality of modelpredictions with respect to defined experimental targets(see Table I) and the alignment quality derived from suchpredictions. To help with this task, the Livermore Predic-tion Center provided automated evaluations of submitted

model predictions using methods developed throughoutthe decade-long course of CASP.2–6 Such measures provedto be an essential component of the assessment, consider-ing the growing number of experimental targets (46 foldrecognition domains) and the increasing number of groupsrepresented in CASP5 (149 manual groups and 59 auto-matic servers predicted fold recognition domains). For thefold recognition category alone, 20220 predictions had tobe evaluated, making manual judgment of all predictionsimpossible within the required timescale of the assess-ment.

Given the drawbacks of relying on a single measure toestimate the quality of all fold recognition predictions (e.g.,see Fig. 3), we decided to incorporate scores from recog-nized structural comparison methods including Dali,7,8

CE,9 and a relatively new structural comparison method,Mammoth.10 In addition, we incorporated scores from acontact distance method we developed for the purpose ofCASP evaluation (see Methods and Results). We consid-ered scores from these four methods, along with the twoLivermore automated evaluation measures (GDT_TS andSOV_O scores3–6) to reflect the overall structural qualityof the predictions.

In previous CASP assessments of the fold recognitioncategory, alignment quality had been an important compo-nent of evaluation.11–14 To expand on this design, we choseto evaluate the quality of prediction alignments using theoutput from structural superpositions of DALI, CE, andMammoth, in addition to the output from a sequence-independent structure alignment (LGA5) provided by thegroup at Livermore. The quality of alignments produced bythese four structural superposition methods was scoredindependently on the basis of the fraction of correctlyaligned residues. Thus, our overall evaluation includedscores generated by six different structural measures andfour different alignment measures for every fold recogni-tion target prediction.

To compare the overall prediction quality of differentgroups, scores generated with all measures and for all foldrecognition domains had to be combined to produce asingle value reflective of group performance, despite thefact that groups predicted varying numbers of targets.

*Correspondence to: Lisa N. Kinch, Howard Hughes Medical Insti-tute, University of Texas Southwestern Medical Center, 5323 HarryHines Blvd., Dallas, TX 75390-9050. E-mail: [email protected].

Received 28 February 2003; Accepted 17 June 2003

PROTEINS: Structure, Function, and Genetics 53:395–409 (2003)

© 2003 WILEY-LISS, INC.

Given that each measure provided different scores andthat each target varied in difficulty, any procedure used tocombine scores had to include rescaling. To address thisproblem, we chose to assign scores (z scores) based on acomparison with the average prediction scores for indi-vidual target domains. Once scaled according to eachtarget, group scores were combined by using various

strategies and averaged over all measures to producevalues reflective of the overall performance of each group.

This general procedure allowed us to compare and rankeach group participating in CASP5 fold recognition predic-tion and to assign statistical significance to the results.Although group rankings were based entirely on theseautomatic scores, we believed that including some aspectof manual judgment of predictions would add value to theassessment. In addition to manually inspecting a numberof models for each target domain, we chose a subset of10-fold recognition domains that ranged in difficulty toscore manually (scoring all prediction models with correctoverall fold; see Methods and Results) and compared theseresults to our automatic scoring method. To gain insightinto which groups performed better on different types oftargets, we evaluated subsets of fold recognition homo-logues and analogs separately. To assess which groupsperformed better at different tasks, we evaluated struc-tural quality measures and alignment quality measuresseparately. These various subsets of scores helped us tomore closely evaluate the most successful approaches tofold recognition.

METHODS AND RESULTSFold Recognition Target Domains

Traditionally, the fold recognition category has includeddomains that fell in between the more clearly definedcomparative modeling targets (displayed sequence similar-ity to known folds) and the new fold targets (displayed nostructural similarity to known folds), with some overlap.Because of the increasing power of sequence similaritydetection methods, the overlap between comparative mod-eling and fold recognition domains in CASP5 has nowbecome extensive and difficult to define. Previous CASPdomain classifications have been rather subjective andlimited to available sequence (defined by PSI-BLAST15,16

in CASP4) and structural (defined by ProSup17 in CASP4)information for similar folds.11,13 We chose to expand onthese traditional domain class boundary definitions bycombining a broader measure of sequence similarity withan objective estimation of target difficulty based on thequality of submitted predictions.

Target sequence similarity to the closest known fold canbe established by a variety of methods. As measured byPSI-BLAST procedures, similarity estimates depend onthe program version, sequence length, database size, andinput cutoffs.15,16 A measure that does not depend on thestructure of sequence space or on arbitrary cutoff valueswould provide a more reliable estimate of similarity.Starting with a structure-based alignment to the closestPDB template, we estimated sequence similarity with thefollowing score (Sseq):

Sseq �S12 � S�random�

S�max� � S�random��

S12 � ¥i,j20 fi

�1�fj�2�sij

S11 � S22

2 � ¥i,j20 fi

�1�fj�2�sij

where S12 represents the score between the aligned target(1) and template (2) sequence measured by the BLO-SUM62 similarity matrix ignoring gap regions, S11 and

TABLE I. Fold Recognition Domains

Targeta Domain bordersb Classc

T0130 6-105 CM/FR(H)T0132 B6-B152 CM/FR(H)T0133 A16-A308 CM/FR(H)T0134_1 A878-A1006 FR(H)T0134_2 A1007-A1112 FR(H)T0135 C3-C108 FR(A)T0136_1 E4-E259 CM/FR(H)T0136_2 E260-E523 CM/FR(H)T0138 A1-A135 FR(H)T0141 A1-A187 CM/FR(H)T0146_1 A1-A24, A114-A196 FR(A)/NFT0146_2 A25-A113 FR(A)/NFT0146_3 A244-A299 FR(A)/NFT0147 A2-A245 FR(A)T0148_1 A2-A9, A101-A163 FR(A)T0148_2 A10-A100 FR(A)T0149_1 A2-A202 CM/FR(H)T0152 G12-G209 CM/FR(H)T0156 A2-A157 FR(H)T0157 A2-A138 FR(H)T0159_1 X1-X91, X234-X309 CM/FR(H)T0159_2 X92-X233 CM/FR(H)T0162_1 A7-A62 FR(A)T0162_2 A63-A113 FR(A)T0165 A1-A318 CM/FR(H)T0168_1 A1-A68, A210-A327 CM/FR(H)T0168_2 A69-A209 CM/FR(H)T0169 A1-A156 CM/FR(H)T0170 1-69 FR(A)/NFT0172_1 A2-A115, A217-A294 CM/FR(H)T0172_2 A116-A216 FR(A)/NFT0173 A3-A299 FR(A)/NFT0174_1 B8-B28, B199-B374 FR(H)T0174_2 B39-B198 FR(H)T0184_2 B166-B236 CM/FR(H)T0185_1 A1-A101 CM/FR(H)T0185_3 A299-A446 CM/FR(H)T0186_2 A45-A256, A293-A330 CM/FR(H)T0186_3 A257-A292 FR(A)/NFT0187_1 A4-A22, A250-A417 FR(A)/NFT0187_2 A23-A249 FR(A)T0189 A1-A319 CM/FR(H)T0191_1 A1-A104, A248-A282 FR(A)T0192 A2-A153, B154-B171 CM/FR(H)T0193_1 A1-A78 FR(H)T0195 A1-A299 CM/FR(H)

aCASP5 target codes. Domain numbers are appended to the codes formultidomain proteins.bDomain boundaries according to Experimental model residue num-bers.cTarget class. CM: Comparative Modeling; FR: Fold Recognition; (H):Homologous; (A): Analogous; NF: New Fold.

396 L.N. KINCH ET AL.

S22 represent the same scores between the target (1)sequence and itself and the template (2) sequence anditself ignoring gap regions, sij represents the same scorebetween amino acids i and j, and fi

(1) and fj(2) represent the

frequency of amino acid i in the target sequence (1) and thefrequency of amino acid j in the template sequence (2),respectively. This score provided an estimate of sequencesimilarity that did not depend on the structure of sequencespace. Furthermore, this score correlated with sequenceidentity between all CASP5 domain target/template pairs(r � 0.974) but was more sensitive to target/template pairdifferences at low-sequence identity.

Although similarity measures indicate the general pre-dictability of targets, the performance of the predictorcommunity as a whole provides the most direct estimate oftarget difficulty. As expected, the target similarities esti-mated by Sseq scores generally correlated with targetdifficulties estimated by average GDT_TS scores [diagonalline, Fig. 1(A)]. A model-based clustering18 of this datasuggested the existence of four different groups and re-flected the discrete nature of protein space. We chose touse this clustering to define the boundary between the foldrecognition assessment [red targets, some blue targets,Fig. 1(A)] and the comparative modeling assessment[purple and green targets, Fig. 1(A)]. The relatively lowaverage GDT_TS scores and similarity scores of the redand blue clusters suggested differences in overall fold thatwarranted evaluation using fold recognition assessmentcriteria. Because most of the domain sequences from thered cluster [Fig. 1(A)] found known folds with sequence-based methods, this boundary definition resulted in anextensive overlap between the two assessment categories[21 CM/FR(H) domains].

Unfortunately, these clusters became less meaningfulfor defining the boundary between new fold and foldrecognition targets. The structure templates required toestimate similarity (Sseq score) were somewhat arbitraryfor new folds and difficult fold recognition domains. Thus,to establish this boundary, we used traditional structuralcriteria (see classification article for discussion1). Thecluster reflecting the most difficult domains (blue, Fig. 1)contained all but two (T0170 T0172_2) of the resulting newfolds (NF and FR(A)/NF domains). Table I summarizes alldomains considered in the CASP 5 fold recognition cat-egory and defines the respective domain borders accordingto residue numbers of experimental models.

Domains included in the fold recognition category couldbe split into homologues (30 domains) and analogs (16domains) of known folds. We briefly summarize the classi-fication illustrated in Table I: comparative modeling/foldrecognition domains that detected known folds usingsequence-based methods were considered to be homo-logues [CM/FR(H)]. This overlapping category included 22domains, which represented almost half of the total num-ber of identified fold recognition targets (46 domains). Foldrecognition domains that did not detect known folds withsequence-based methods but possess substantial struc-tural similarities to known folds were also considered to behomologues [FR(H)]. For the eight domains of this class,

various justifications allowed an inference of evolutionaryrelatedness (see target classification article1). Fold recogni-tion analog domains [FR(A)] contained general structuralsimilarity to known folds, although evolutionary related-ness could not be established. Finally, new fold/fold recog-nition (NF/FR) domains contained very distant structuralsimilarities to known folds. For these eight domains,topologies generally resembled those of known folds, butthe overall size or packing arrangement of secondarystructural element differed significantly.

Numerical Evaluation of Predictions

To evaluate the overall quality of fold recognition predic-tions for a given target, each prediction must be comparedwith the experimental model by considering both struc-tural similarities and alignment quality. Ideally, an over-all score could be assigned to each prediction based onthese criteria. Unfortunately, no single standard measureexists to generate such scores, despite an expanding anddiverse number of available protein structure comparisonmethods (for a review, see Ref. 19). To aid in our assess-ment, the group at Livermore provided various evaluationscores (GDT_TS, SOV_O, and LGA_Q, e.g.) includingthree structural superpositions (2 sequence dependentand 1 sequence independent) for each prediction ranked bytarget in order of descending GDT_TS scores.20 Given thepossible limitations of using a single measure to estimatethe overall quality of structural predictions and the di-verse range of fold recognition targets, we decided to applya combination of available structural comparison mea-sures to our assessment of fold recognition domains. Justas different predictions captured various details of foldrecognition targets, different structural comparison meth-ods potentially captured various aspects of such predic-tions and provided a certain robustness to method-specificerrors.

For our evaluation, we chose to use the global distancetest total score (GDT_TS) and the segment overlap mea-sure observed score (SOV_O), which have been developedover the time course of CASP evaluation.2–4,6,21 TheGDT_TS score represents the average of the percentage ofresidues that can be superimposed within a given distanceover four optimal sequence-dependent superpositions (1,2, 4, and 8 Å).2,4,6 This GDT analysis has been generallyaccepted by the prediction community and has providedthe basis for previous CASP assessments. Although theGDT_TS score reflects the overall tertiary structural qual-ity of a model prediction, the SOV_O score provides anassessment of prediction quality that depends on a seg-ment-based evaluation of secondary structure.3,21 Thesetwo scores provided alternative automated evaluations ofCASP predictions that addressed both global (GDT_TS)and local (SOV_O) aspects of model quality.

To further increase the range of model evaluation, wechose to include scores from three additional structuralevaluation methods developed in other laboratories whoseprograms were available for local use (Dali,7,8 http://www.ebi.ac.uk/dali/; CE,9 http://cl.sdsc.edu/ce.html; andMammoth,10 http://icb.mssm.edu/services/mammoth/mam-

CASP5 397

moth). Of these structural alignment programs, both Daliand CE compare intramolecular C� geometries (distancesand angles, respectively) of the target structure with thoseof the model structure. Although the two programs usedifferent procedures to generate optimal structural align-ments, each defines the quality of the results in terms of a zscore. Results of the two methods tend to diverge signifi-cantly for dissimilar structures (SCOP superfamily/familypairs under about 10% sequence identity)22 and may thus

detect different aspects of structural predictions for thevarious fold recognition targets with marginal predictions.

The third structural evaluation method (Mammoth) wasdeveloped to evaluate structural similarities at the foldlevel, which seems ideal for scoring domains that fallwithin the fold recognition category. Mammoth differsfrom the previous two methods by producing structuralalignments that do not depend on contact maps butinstead depend on unit-vector root-mean-square dis-

Fig. 1. Target difficulty and domain classification. A: Average GDT_TS scores for all target domainsrescaled according to z score (GDT_TSavg) are plotted against BLOSUM (Sseq) similarity scores generatedfrom a structure-based sequence alignment of the target with the closest template and rescaled by z score. Thelinear regression fit (diagonal gray line) represents predicted average GDT_TS scores for difficulty levelsrepresented by Sseq. Each data point is labeled according to domain and colored according to clusteranalysis.18 For clarity, we omitted the “T01” from the start of all target identification numbers so that T0129became 29, for example. B: The residuals of average GDT_TS scores with respect to predicted averageGDT_TS scores from data shown in (A) are plotted against the same similarity score described in (A). Datafrom single-domain targets are colored yellow, and data from multiple domains are colored light blue. A blackline and a gray line indicate the averages of all single-domain and multiple-domain residuals, respectively.


tances. The program also associates a statistical signifi-cance value to the similarity score.10 Such a method allowsfor possible registration shifts that occur frequently instructural analogs and concentrates on structural similari-ties at a more general level than does Dali or CE.

As an additional structural evaluation method, we gen-erated a sequence-dependent score that relied on similari-ties between intramolecular C� contact distances of thetarget and model structures. The contact distance scorewas calculated with the following equation:

Scontact � ��i�1

N �j�i�1

N

e��dij�di�j��2/�where dij represented the distance between the C� atomsof residues i and j in the target structure, di�j� representedcorresponding residues i� and j� in the model structure, Nrepresented the total number of amino acid residues in thetarget structure, and represented an adjustable parame-ter that was taken to be the value (0.5 Å2) at which thecorrelation between GDT_TS and contact score was maxi-mized over all models for all targets. LiveBench uses asimilar measure to continuously evaluate structure predic-tion servers.23

To evaluate the alignment quality of fold recognitionpredictions, we used sequence alignments produced by thesequence-independent structural superposition methods[Dali, CE, Mammoth, and the local-global alignment (LGA)provided by Livermore]. Alignment quality (Q) scores foreach of the different superpositions was defined as thefraction of correctly aligned residues. Raw scores weregenerated by adding one point for each correctly alignedresidue from the structural superposition and dividing thesum by the length of the target sequence. No fractionalpoints are awarded for shifts. These scores were ratherstringent and could be viewed as structure-independentmeasures of alignment quality.

Combining Scores

With a goal of comparing the overall prediction qualityof different groups, scores generated by using the sixstructure and four alignment measures needed to becombined to produce a single value. The fact that eachtarget exhibited a different level of difficulty complicatedthis task, making combined scores meaningful only afterproper scaling across targets had been applied. Over thecourse of CASP, different assessors have approached thisscaling problem in different ways. For example, in theCASP4 fold recognition assessment, scores were normal-ized by assigning an overall numerical value (1–4) to eachprediction based on manually assigned thresholds thatwere different for each target.11 We decided to apply a lesssubjective methodology that rescaled prediction scoresaccording to the average prediction quality of that target(using z scores). For each participating group, these zscores could be either summed or averaged over all targets(or a subset of targets) and then averaged over all mea-sures (or a subset of measures).

As a first approach to combine target scores, we chose toapply the following equation (z score summation) in whichthe overall score for a group j is defined as:

score j�sum� � �i�1

N �score i � �score i�predictions

i,predictions�

where N represents the number of predicted FR domains,�score i�predictions represents the average score of all groupsfor a given target i with a given measure, and i,predictions

represents the standard deviation. This summation pro-duced a single group score for each given measure. Bysumming the scores of each target domain, groups whopredicted more targets had an opportunity to achieve ahigher overall score than those that predicted fewer tar-gets (although penalties for worse than average predic-tions counted as negative z scores). Thus, as a secondapproach to combining group scores, we decided to averagez scores over all predicted targets.

score j�mean� � �i�1

N �score i � �score i�predictions,2

N � i,predictions,2�

For this method, we recalculated the mean (�scorei�predictions,2) and sigma (i,predictions,2) disregarding scoresbelow �2� of the entire sample, and we divided by thenumber of predicted domains (N) to achieve an average. Inthe process of averaging, each negative z score was re-placed with zero to eliminate the penalty for worse thanaverage predictions. This method of combining scores wasbased on the same principles outlined in current andprevious assessments of comparative modeling targets.24,25

After combining the target scores, each group was as-signed a single summation score [score j(sum)] and a singleaverage score [score j(mean)] for each measure (6 struc-tural measures and 4 alignment measures). Again, werescaled both the summation scores and the average scoresfor each measure to obtain z scores.

Once rescaled, z scores for each individual measurecould either be compared with the z scores produced by theother measures directly (Fig. 2) or could be combined bysimple averaging to generate overall scores (Tables II–IVscores). Linear correlation coefficients (Pearson r values)from pairwise plots of z scores for individual measures(first models) are reported in Figure 2. The z scores of eachindividual measure compared to those averaged over allmeasures suggested that as a single measure, the GDT_TSscore (r � 0.973) most closely represented that of allcombined measures, with Mammoth following closely be-hind (r � 0.964).

Although correlations between all measures suggestedthat the GDT_TS score was the single most representativemeasure for assessing fold recognition targets, the GDT_TSscore alone fell short in evaluations of difficult targets. Forone such target [T0147, Fig. 3(A)], GDT_TS scores placedat rank 4 a structural prediction [Fig. 3(B)] that appearedfragmented, included some overlapping coordinates, andcontained irregular helices and strands. Alternatively,

CASP5 399

GDT_TS scores ranked at 27 a structural prediction [Fig.3(C)] with a general fold topology identical to the target.Manual inspection of these two structures disagreed withthese ranks. For this target, our combined score strategycaused the fragmented prediction to drop to rank 70 (dueto unfavorable structural measure scores), whereas thetopologically correct prediction moved to a higher rank(10). Although such extreme cases of GDT_TS shortcom-ings occur infrequently, this example highlights the dan-ger of relying on a simple ranking scheme for evaluation.Using the input from more than one evaluation methodprovides one way to increase the significance of scores andthus the meaning of ranks, because the shortcomings of

individual methods are essentially averaged over theinputs of other methods.

Our evaluation process assigned a single overall score toeach prediction group based on the input of 10 differentautomated scores. How did this process compare to amanual evaluation? The large number of fold recognitionpredictions to be evaluated (20220 models) prevented acomplete comparison of manual scores with those gener-ated automatically. Therefore, we chose a subset of 10 foldrecognition domains (T0130, T0132, T0133, T0135, T0138,T0141, T0147, T0156, T0157, and T0173) for manualevaluation to address this question. The chosen domainsrepresented a broad range of difficulty levels (average

TABLE II. Top 20 Predictors Ranked by Combined Scores, All Domains

Rankfirst(sum)a

Rankbest

(sum)a

Rankfirst

(mean)b,c

Rankbest

(mean)b Groupc,d Predictorc,d,ePredictions

scored

z scorefirst

(sum)a,d

Bootstrapfirst

(sum)a,d

z scorebest

(sum)a,d

Bootstrapbest

(sum)a,d

1 1 2 1 2 Baker 46 3.21 0.83 3.58 0.982 3 3 6 453 Ginalski 46 2.76 0.70 1.87 0.263 8 6 13 6 Rychlewski 46 2.37 0.62 1.60 0.104 6 10 12 29 Robetta (S) 46 2.02 0.33 1.68 0.125 13 7 14 517 Bujnicki 43 1.96 0.29 1.33 0.046 2 12 3 10 Skolnick 46 1.88 0.17 2.69 0.977 25 14 32 96 Bates 46 1.66 0.12 0.93 0.078 18 17 23 427 Fischer 46 1.64 0.18 1.18 0.079 11 4 5 20 Bujnicki 33 1.61 0.13 1.38 0.17

10 23 13 21 110 Honig 36 1.50 0.09 1.05 0.0911 16 15 16 28 Shi 39 1.49 0.05 1.20 0.0512 5 22 9 12 Xu 46 1.46 0.08 1.79 0.2313 14 19 22 450 Labesse 42 1.43 0.05 1.25 0.0714 9 27 19 373 Brooks 46 1.35 0.07 1.48 0.1415 31 18 28 153 Takeda-Shitaka 39 1.34 0.10 0.83 0.0416 17 31 29 112 Friesner 46 1.24 0.05 1.19 0.0716 45 32 72 67 Jones 46 1.24 0.07 0.54 0.0518 20 29 31 1 Karplus 46 1.21 0.07 1.14 0.0918 7 23 10 40 Pmodel (S) 46 1.21 0.06 1.65 0.1420 3 21 8 45 Pmodel3 (S) 46 1.15 0.04 1.87 0.38

aSum refers to combining target scores by summation as described in the text.bMean refers to combining target scores by averaging z scores (�2, no negative scores) as described in the text.cGroup 425 (Venclovas) and group 448 (Murzin) do not appear in the top 20 using the summation method (predicted 7 domains and 22 domains,respectively). Venclovas ranked 1 and Murzin ranked 5 using averaging method of combining scores.dGroups/scores in bold display significant ( 95% confidence) difference from most of the remaining top 20 groups using Student’s t-tests.eAutomatic servers are indicated with (S) following the predictor name.

Fig. 2. Measure correlations. z scores using first models of all fold recognition targets for each measurewere plotted against those of every other measure and the average scores over all measures (Avg). ThePearson r value for each comparison is reported for pairwise measure comparisons. Individual measures areabbreviated as GDT_TS for GDT_TS structural measure, Contact for contact distance measure, DALI for Dalistructural measure, CE for CE structural measure, MM for Mammoth structural measure, SOV_O for segmentoverlap measure, Qlga for LGA superposition alignment score, Qdali for Dali alignment, Qce for CE alignment,and Qmm for mammoth alignment.


GDT_TS scores from 14.0 to 49.4) and included bothhomologues and analogs of known folds.

To construct manual scores, one point was assigned toeach correctly predicted secondary structural element, and

one point was added for every secondary structural ele-ment that produced correctly aligned residues. Half pointswere subtracted for erroneous distances, gaps, or non-protein-like architectures. Based on the individual tar-

TABLE III. Top 20 Predictors Ranked by Combined Scores, Homologues, and Analogs

Homologues Analogs

Rankfirst(sum)a

Rankbest

(sum)a Groupb Predictorb,cPredictions

scored

z scorefirst

(sum)a,b

Rankfirst

(sum)a

Rankbest


scored

z scorefirst

(sum)a,b

1 3 453 Ginalski 30 2.56 1 1 2 Baker 16 5.092 5 6 Rychlewski 30 2.27 2 6 349 Shortle 12 2.733 8 517 Bujnicki 30 2.00 3 4 1 Karplus 16 2.234 1 2 Baker 30 1.95 4 21 453 Ginalski 16 2.145 12 96 Bates 30 1.88 5 3 29 Robetta (S) 16 2.086 10 20 Bujnicki 27 1.88 6 31 6 Rychlewski 16 1.697 9 427 Fischer 30 1.74 7 2 10 Skolnick 16 1.658 16 29 Robetta (S) 30 1.65 8 7 68 Jones 8 1.479 2 10 Skolnick 30 1.63 9 17 51 Samudrala 16 1.26

10 18 110 Honig 27 1.50 10 51 67 Jones 16 1.1711 15 28 Shi 30 1.49 10 14 112 Friesner 16 1.1712 6 12 Xu 30 1.47 12 59 132 I-sites/Bystroff (S) 16 1.0913 22 153 Takeda-Shitaka 28 1.46 13 43 517 Bujnicki 13 1.0314 13 450 Labesse 28 1.40 14 15 373 Brooks 16 0.9415 7 40 Pmodel (S) 30 1.33 15 34 28 Shi 9 0.9116 27 368 Saldanha 28 1.30 16 45 110 Honig 9 0.8917 34 265 Sasson-Iris 30 1.27 17 29 450 Labesse 14 0.8818 11 373 Brooks 30 1.26 18 64 423 Taylor 16 0.7919 28 84 Elofsson 30 1.22 19 8 16 Levitt 14 0.7820 26 448 Murzin 17 1.21 19 9 12 Xu 16 0.78

aSum refers to combining target scores by summation as described in the text.bGroups/scores in bold display significant ( 95% confidence) difference from most of the remaining top twenty groups using Student’s t-tests.cAutomatic servers are indicated with (S) following the predictor name.

TABLE IV. Top 20 Predictors Ranked by Structure Measures and Alignment Measures

Structure Alignment

Rankfirst(sum)a

Rankbest


scored

z scorefirst

(sum)a,b

Rankfirst

(sum)a

Rankbest


scored

z scorefirst

(sum)a,b

1 3 453 Ginalski 25 3.11 1 1 2 Baker 25 3.542 1 2 Baker 25 2.74 2 5 453 Ginalski 25 3.003 5 6 Rychlewski 25 2.57 3 3 29 Robetta (S) 25 2.844 2 10 Skolnick 25 1.97 4 7 6 Rychlewski 25 2.765 11 110 Honig 20 1.87 5 2 10 Skolnick 25 2.696 9 29 Robetta (S) 25 1.78 6 14 427 Fischer 25 1.907 22 517 Bujnicki 25 1.70 7 15 28 Shi 23 1.888 32 96 Bates 25 1.54 8 35 96 Bates 25 1.679 21 427 Fischer 25 1.51 9 24 517 Bujnicki 25 1.64

10 15 28 Shi 23 1.45 10 9 1 Karplus 25 1.6311 7 450 Labesse 22 1.44 11 27 110 Honig 20 1.6112 19 20 Bujnicki 17 1.32 12 8 373 Brooks 25 1.6013 26 448 Murzin 15 1.30 13 32 448 Murzin 15 1.3814 47 67 Jones 25 1.27 14 26 20 Bujnicki 17 1.3215 9 41 Akiyama 25 1.25 15 6 12 Xu 25 1.3016 37 153 Takeda-Shitaka 20 1.22 16 55 67 Jones 25 1.2017 39 435 Fujita 21 1.20 17 53 265 Sasson-Iris 25 1.1518 57 105 Sternberg 25 1.15 17 17 450 Labesse 22 1.1519 49 368 Saldanha 23 1.14 19 47 242 Bujnicki 22 1.0520 13 373 Brooks 25 1.12 20 21 68 Jones 10 1.03

aSum refers to combining target scores by summation as described in the text.bGroups/scores in bold display significant ( 95% confidence) difference from most of the remaining top 20 groups using Student’s t-tests.cAutomatic servers are indicated with (S) following the predictor name.

CASP5 401

gets, various bonus points were also possible for functionalor conserved residue placement or for unusual structuralfeature predictions. All models for each target were evalu-ated in order of decreasing GDT_TS score, and evaluationsstopped when predictions no longer contained a majority ofthe target fold. Thus, the number of evaluations rangedfrom 74 different models for T0173 (difficult target) to 200different models for T0141 (easy target). When the uneva-luated models were assigned a score of zero, the resulting

manually assigned scores correlated with the averagedscores generated for the same subset of target domainsusing our automated procedure (r � 0.960, first models,Fig. 4).

Predictors were allowed to submit up to five models foreach target. Although this evaluation focused on modelslabeled “1” (first models), the organizers of CASP did notoutline specific criteria for addressing the additional mod-els. In the evaluation of the fold recognition category for

Fig. 3. Example of GDT_TS measure shortcoming. A: Target 147 E. coli YcdX experimental model. B: Example of GDT_TS score 32.8 (rank 4)prediction. C: Example of GDT_TS score 27.57 (rank 27) prediction. Objects are colored by using a rainbow ramp of secondary structure progressionfrom N-terminal (blue) to C-terminal (red). Figures are prepared with MolScript.34

Fig. 4. Comparison of manual evaluation scores to automated evaluation scores. Manual evaluationscores were generated as described in the text. These manual scores (y axis) were summed over all evaluatedtargets (10 domains) and rescaled according to z score. The automatic scores (x axis) were generated byaveraging z scores generated by the six structural and four alignment measures over the same targets (10domains). The data points represent scores for each first model prediction for all evaluated targets, and thegray trend line fit by linear regression shows the correlation between scores generated manually andautomatically (r � 0.96).


CASP4, only first models were considered. For our assess-ment, we chose to evaluate both “first models” and “bestmodels” to ascertain the best-quality models produced bygiven methods. To give the predictors every opportunity toobtain a higher overall score, we defined the best model asthe model with the highest score for a given measure. Thisbest score may not necessarily represent the same modelnumber for all of the measures (on average, �7.5 bestmodels were identical out of 10 measures). Overall rank-ings appeared to be quite sensitive to whether first or bestmodels were used as input. For example, the automaticserver Pmodel3 ranked 3 when using best models of all foldrecognition domains but dropped to rank 20 when consider-ing first models (see Table II).

Evaluation of Significance

Table II summarizes the top 20 prediction groups rankedby “first” model combined measure scores for all definedfold recognition domains (46 domains). By using thesummation method, the highest scores achieved by group 2(Baker, 3.21), group 453 (Ginalski, 2.72), and group 6(Rychlewski, 2.37) appeared to stand apart from theremaining scores (�2.02). To test if the prediction qualityof these groups could be reliably distinguished from thoseof the other groups, we sought to evaluate the statisticalsignificance of the results using various measures. Previ-ously, the statistical significance between paired samplesof CASP4 comparative model domains were estimated byusing the parametric Student’s t-test and the nonparamet-ric Wilcoxon signed rank test.26 Figure 5 illustrates thecomparison of the top 20 CASP5 fold recognition groupsusing the paired Student’s t-test on combined measurescores for all fold recognition targets (first models). Indeed,these three groups tended to perform significantly differ-

ently from the remaining groups (gray highlights 95%confidence of difference). Below these top groups, thestatistical significance of the differences in ranking wasmarginal. Results were similar for evaluations using theWilcoxon signed rank test (data not shown).

Successful application of Student’s t-test to target predic-tions assumes a normal distribution of model qualitydifferences (although this is not an assumption of theWilcoxon test) and a large number of common models.11,27

Although most of the top scoring groups predicted most orall of the fold recognition targets, these assumptions didnot necessarily hold for all groups participating in CASP.Therefore, we applied an additional evaluation of ranksignificance to the fold recognition results through abootstrap selection of z scores.27 In the bootstrap proce-dure, the score for each group over all targets was calcu-lated from a random selection of N z scores, where Nequals the number of targets predicted by the group andthe selection set was composed of the actual first or bestmodel z scores. The selection was performed N times withreturns. Thus, some z scores may have been selected morethan once, whereas others may not have been representedat all. The scores obtained from this random selectionprocess were used to rerank the groups. This procedurewas repeated 200 times, and the number of times eachgroup obtained the same bootstrap rank as their actualcalculated rank was tabulated and reported as a fraction ofthe total number of repeats (Table II). By using thisprocedure, a larger bootstrap value at a particular rankindicated a higher significance of that rank. The results ofthis bootstrap procedure also suggested that the ranks ofthe top three groups were significantly better than those ofthe other groups.

Fig. 5. Statistical significance by Student’s t-test. The number of domains common to both groups are listed to the left of the diagonal, whereas thevalues of t are listed to the right of the diagonal. Statistically significant t values that correspond to a probability of deviation 95% are highlighted in gray.

CASP5 403

Prediction Ranks: Categories and Highlights

When summing the scores of first model predictions ofall fold recognition domains, our assessment and evalua-tion of ranking significance suggested that the methods ofthree groups (Baker, Ginalski, and Rychlewski) outper-formed the methods of all other groups. However, rankingschanged when we used the best model predictions for each

target or when we scored the predictions by averaging.Notably, two groups (group 425 Venclovas and group 448Murzin) who did not score well by the summation ap-proach emerge in the top 20 when the mean approach wasused. Each of these groups predicted fewer target domainsthan the remaining groups (Venclovas with 7 domains andMurzin with 22 domains, out of 46 possible domains).Other than these exceptions, the rankings produced byeach method followed the same general trends as most ofthe top performing groups predicted all target domains.

Table II includes overall scores, ranks, and bootstrappercentages for all best model fold recognition domainpredictions of the top 20 groups. Considering best model

Fig. 6. First models and best models. Target T0170 sequence-independent structural superpositions (Livermore) with (A) group 2 Bakerprediction best model labeled “4,” (B) group 2 Baker prediction first model,(C) group 10 Skolnick prediction best model labeled “3,” and (D) group 10Skolnick prediction first model. In all superpositions, the experimentalmodel structure is colored in thick purple, and the prediction is colored inred. If the distance between aligned residues is below 4 Å, then theresidue backbone traces are colored yellow (not aligned correctly) orgreen (aligned correctly).

Fig. 7. Score distribution of manual predictors and automatic servers. Ahistogram depicting the percentage of the total number of predictions scored(y axis) for manual predictors (lavender) and automatic servers (maroon)falling within the indicated z score bin ranges (x axis). z scores represent theaverage scores of all measures calculated for best model predictions.

Fig. 8. Partial fold recognition. Target T0173 M. tuberculosis mycothiol deacetylase (A) experimental model, (B) group 349 Shortle first predictionmodel, (C) group 2 Baker first prediction model, and (D) group 12 Xu first prediction model. The secondary structural elements belonging to theN-terminal Rossmann-like fold are colored blue (�-helices), yellow (�-strands), and pale green (coil). Secondary structural elements C-terminal of theRossmann-like fold are colored white (not predicted), purple (correct prediction), and green (prediction in incorrect orientation).


predictions, the maximum score was still achieved bygroup 2 (Baker, 3.58), followed closely by that of group 10(Skolnick, 2.69). Figure 6 highlights difficulties of thesetwo top scoring groups (Baker and Skolnick) with pickingmodels. Group 10’s (Skolnick) prediction model 3 [Fig.6(A), green] and group 2’s (Baker) prediction model 4 [Fig.6(C), green] superimposed nicely with experimental targetT0170, whereas their first model predictions superim-posed poorly [Figs. 6(B) and (D), respectively]. It is interest-ing that each of these pairs of predictions corresponded tomirror images, where the group assigned the incorrecthanded structure as the first model.

The top 20 groups summarized in Table II includedthree automatic servers. The first model predictions ofgroup 29 (Robetta, 2.02) ranked high (finish ranked 4 in33% of bootstraps), whereas the best model predictions ofgroup 45 (Pmodel3, 1.87) and group 40 (Pmodel, 1.65)ranked 3 (38% of bootstraps) and 7 (14% of bootstraps),respectively. Although the average of the overall scoredistribution (first models, combined measures) for auto-matic servers (�0.28) was lower than that for manualgroups (0.11), the performance of the top automatic serverfirst model predictions (group 29 Robetta) approachedthose of the best manual predictors, and the performancesof the top automatic servers best model predictions (group45 Pmodel3, group 29 Robetta, and group 40 Pmodel)approached those of the best manual predictors. Figure 7illustrates the score distribution of manual predictors andautomatic servers on best model predictions.

The fold recognition category included a variety ofdomains encompassing a wide range of difficulty levels.Many of the top performing fold recognition groups alsoperformed well in the comparative modeling and the newfold categories. We could emphasize this concept by split-ting the fold recognition domains into homologues (30domains) and analogs (16 domains) of known folds. TableIII summarizes the resulting scores. Top achieving groupsfor fold recognition homologues (group 453 Ginalski, group6 Rychlewski, and group 517 Bujnicki, Table III) per-formed well in the comparative modeling assessment,25

and top achieving groups for fold recognition analogs(group 2 Baker, group 349 Shortle, and group 1 Karplus,Table III) performed well in the new fold assessment.28

For fold recognition homologues, the score of one addi-tional group (group 96 Bates) approached that of some ofthe best performers. For fold recognition analogs, group 2(Baker) significantly outscored (5.09) all other methods,whereas the automatic server (group 29 Robetta) devel-oped by the Baker laboratory ranked an impressive 3 (bestmodels) and 5 (first models).

Because of improved sequence homology detection meth-ods and increasing numbers of available sequences indatabases, a significant overlap existed between the CASP5comparative modeling (defined as any domain with detect-able sequence similarity to a known fold) and fold recogni-tion (defined by a natural boundary between domainclusters in Fig. 1). To minimize this overlap, we defined asubcategory of FR domains that included structures dis-playing obvious structural similarity in the absence of

significant sequence similarity (25 domains). The corre-sponding predictions were evaluated by separating struc-tural and alignment quality measures (Table IV). Al-though the same groups tended to perform significantlybetter than others (bold, Table IV) using structural mea-sures (group 453 Ginalski, group 2 Baker, and group 6Rychlewski) or alignment measures (group 2 Baker andgroup 453 Ginalski), one additional group (group 110Honig) could be distinguished by their overall structuralquality of predictions in this subcategory.

Highlights and Pitfalls of CASP5 Predictions

To exemplify the progress made in CASP5 and toidentify potential focuses of future efforts in structureprediction, we highlight the predictions of several interest-ing fold recognition targets. Mycothiol deacetylase from M.tuberculosis (T0173) represents one of the most difficultexamples. This single-domain protein, classified as FR(A)/NF, contains a novel Rossmann-like �/� fold with anoverall topology similar to that of SAM-dependent methyl-transferase but with a distinct sheet curvature. Many topscoring groups correctly predicted various aspects of theRossmann-like N-terminus. However, no group correctlyrepresented the overall fold topology. In fact, the closestprediction (group 349 Shortle, first model) to the overallfold topology in terms of coverage correctly extended theRossmann-like fold by only one helix [purple, Fig. 8(A)]and incorrectly extended the sheet with a parallel �-strand(green, Fig. 8). This prediction ranked 7 among all firstmodels, however, because it missed the distinct sheetcurvature present in the experimental structure.

The first model prediction of group 2 (Baker) achievedthe top score for this difficult target with all measures [Fig.8(B)]. The Baker model correctly assigned four of five of theRossmann-like �-strands with accurate sheet curvature,relatively good alignment, and correct stacking of �-heli-ces. Model 1 submitted by group 12 (Xu) representedanother noteworthy prediction for this target (rank 2 forcombined measures). This prediction included all of theRossmann-like �-strands in a curved sheet [Fig. 8(C)].However, the lengths and stacking of the �-helices differedslightly, as indicated by a lower structural measure rank-ing for this individual target (rank 3 among first modelsand rank 4 among best models).

In CASP5, the correct prediction of multiple-domainproteins in general remained a challenging task. Some foldrecognition targets contained rather difficult domain orga-nizations, including swaps of secondary structural ele-ments between two domains (T0148) or between twomonomers (T0192 and T0193) and discontinuous bound-aries in terms of primary sequence (T0146, T0148, T0159,T0168, T0172, T0174, T0186, T0187, and T0191). Toevaluate the performance of the prediction community onsuch proteins, we compared the average GDT_TS scoresattained on single-domain targets with those attained onmultiple-domain targets in terms of target difficulty.Briefly, a trend line through the target difficulty data [Fig.1(A)] represents the predicted average performance foreach difficulty level. The relative performance of each

CASP5 405

target with respect to this trend line was calculated byresiduals (i.e., the difference between the calculated aver-age performance and the actual performance), with posi-tive values representing better than expected results andnegative values representing worse than expected results[Fig. 1(B)]. We then averaged single-domain target residu-als (0.194) and multiple-domain target residuals (�0.151)separately [Fig. 1(B), trend lines]. These results indicate abetter performance of the CASP5 community as a whole onpredicting single-domain targets.

To demonstrate the difficulties of complex domain orga-nizations, we discuss one interesting target (T0148) thatcontains a tandem repeat of ferredoxin-like fold domains[each classified as FR(A)] with swapped N-terminal�-strands [Figs. 9(A)–(C)]. Although no group correctlypredicted the strand swap, several top performing groupspredicted the ferredoxin-like fold duplication [group 10Skolnick, Fig. 9(D); group 2 Baker, group 6 Rychlewski,group 450 Labesse, group 28 Shi, group 427 Fischer, andgroup 110 Honig, not shown]. The server Robetta (group

29) produced a remarkable first model prediction for thistarget. Although the model contained incomplete sheets, itcorrectly assigned the discontinuous boundaries of the twodomains [Fig. 9(E)], and outperformed the manual predic-tion from the same group [group 2 Baker, Fig. 9(F)]. Thisresult highlights a potential use of such fragment-basedprediction methods in domain parsing.

Several predictions more closely resembled the targetstructure than any available template. The first modelprediction of the FR(H) target T0134_2 by group 453(Ginalski) represented one such accomplishment (Fig. 10).Ginalski combined two existing PDB template structures(1qts and 1e42) to produce a model structure with aGDT_TS score (79.01) slightly better than that of theclosest template (1qts, 78.19). Similarly, group 28 Shiachieved a slightly higher GDT_TS score (63.96) than theclosest available PDB template 1hjr (63.89) with a secondmodel prediction of the target T0157. In view of thenumerous insertions/deletions and the relatively low-sequence identities between each of these targets and their

Fig. 9. Difficult domain organization. Secondary structural elements making up the tandem repeat offerredoxin-like folds found in Target 148 are colored blue (first domain) and yellow (second domain), with thestrand swap highlighted in red and purple. Secondary structural �-strands (arrows) and �-helices (cylinders) ofthe target structure are represented as (A) cartoon and (B) linear diagrams. Secondary structural elements of(C) the T0148 experimental model structure, (D) group 10 Skolnick first model fragment 1 and 2 predictions, (E)group 29 Robetta first model prediction, and (F) group 2 Baker first model prediction. Secondary structuralelements of the target structure are colored according to the structural diagrams. Secondary structuralelements of the prediction models are colored according to the corresponding residues in the target structure,with incorrectly placed (nonswapped) elements highlighted in green.


templates (12.7% for T0134_2 and 1qts; 15.4% for T0157and 1hjr), both of these high scoring predictions achievedremarkable alignment quality. For example, the Ginalskimodel for T0134_2 correctly aligned all but one of thesecondary structural elements [black, Fig. 10(C)]. Consid-ering this alignment shift, the higher GDT_TS scorebecomes more significant in a structural sense. In fact, thetarget domain superimposed with the Ginalski model(RMSD 1.6 over 105 residues) better than it superimposedwith either template used to generate the model (1qts:RMSD 1.8 over 100 residues and 1e42: RMSD 1.8 over 104residues).

The CASP5 fold recognition targets included an exampleof a singleton sequence (T0174) that did not find anysimilar sequences in public databases using PSI-BLASTprocedures. Such an isolated sequence represents a particu-lar challenge for fold recognition methods. The T0174experimental structure belongs to the two-domain SCOPsuperfamily of GHMP kinases. Remarkably, many groups(11 automatic servers and 10 manual predictors) correctlyidentified the GHMP kinase fold as a template for thisprotein (reported as parent 1kvk, 1kkh, 1fwk, 1fi4, 1k47,and 1h73). Of these groups, two automatic servers (group45 Pmodel3 and its input server group 39 Pcons3) andseveral manual predictors produced first models withreasonable overall folds (group 373 Brooks, group 437Ho-Kai-Ming, group 96 Bates, and group 10 Skolnick;group 453 Ginalski and group 6 Rychlewski reported N/Aas parent template but identified the correct fold). Consid-ering best models, more automatic servers found thecorrect fold and obtained satisfactory scores (group 362PSPT, group 40 Pmodel, group 38 Pcons2, group 14FUGUE2, group 226 FUGUE3, and group 221 INBGU)than did manual predictors (group 47 Gibrat, group 464Catherinot, and group 8 Royyuru).

Although many predictors identified a correct tem-plate for the singleton target T0174, the resultingstructural models diverged significantly from the targetstructure. GDT_TS scores for the best domain predic-tions (group 45 Pmodel3, 26.015 for T0174_1; and group437 Ho-Kai-Ming, 40.161 for T0174_2) did not approach

those calculated for a manual structural alignment withthe closest template (1kvk, 55.492 for T0174_1 and68.614 T0174_2). For most of the predictions withcorrect parent templates, these poor scores reflectedboth incomplete alignment coverage and poor alignmentquality. For example, the coordinates of one automaticserver prediction (group 189 SAM-T02-server best model)only extended over one domain of the target structure,whereas another group (group 537 Asogawa) misalignedthe entire target sequence with that of the parenttemplate (1kvk). Although the structure of this mis-aligned prediction looked similar to that of the top

Fig. 10. Prediction beats available templates. Structural models are depicted for (A) the closest availabletemplate (1qts) to target domain T0134_2, (B) the experimental domain T0134_2, and (C) the best prediction(group 453 Ginalski first model) for T0134_2. Helices and strands common to all structures are colored blueand yellow, respectively. Compared with the corresponding experimental model structure (B), secondarystructural elements with incorrectly aligned residues are colored red.

Fig. 11. Physically unrealistic models. Ball-and-stick representationsof backbone coordinates from one best model prediction for Target T0174illustrate (A) two turns of a helix-like secondary structure from residue 350to residue 360 and (B) two overlapping strand-like secondary structuresconnected by a turn from residues 176 to 186. Atoms are coloredaccording to type: red (oxygen), blue (nitrogen), and gray (carbon). Bondsare drawn between atoms �1.6 Å apart (pale green), and the backbonetrace is illustrated in purple.

CASP5 407

predictions, the poor alignment was reflected in a lowoverall alignment quality score ( z score of �0.34).

In addition to poor alignment quality, many top predic-tions for Target T0174 displayed poor local structuralquality. For example, one ab initio method produced aphysically unrealistic model that generated one of the topoverall scores for this target (average z score for bothdomains 2.035) (Fig. 11). Figure 11 illustrates examples oftwo secondary structure regions in this model wherebackbone coordinates essentially overlap (fall within abond distance of 1.6 Å). Despite the presence of these poorstructural regions, the model correctly represented the�-strand with respect to the template structure (1kvk) forthe second domain (T0174_2). Such a result underscores ageneralized need for applying some form of model refine-ment to structures produced by a number of fold recogni-tion and ab initio-type methods (see also new fold assess-ment article28).

DISCUSSION

This report concentrates on evaluating the top scoringgroups in the CASP5 fold recognition category to establishthe current state of the art in protein structure predictionmethods. Despite the obvious advantages of such anassessment, the evaluation process falls short in identify-ing promising new methods that may ultimately drivefuture progress in the field. One of the original objectives ofCASP was to identify the best methods in the threedifferent categories: comparative modeling, fold recogni-tion, and ab initio protein structure prediction. Today, thisgoal is masked by a growing trend to combine variousexisting prediction tools in unique ways. Both manualgroups and automatic servers that exploit such combina-tions surpass the independent “threading” methods thatonce dominated the fold recognition field.

In many ways, the CASP assessment process itself hasdriven the tendency of predictors to merge existing predic-tion techniques. Pressures to predict each and every targetforce groups to use a combination of the best availabletechniques for each prediction category. Fortunately, suchpressures did not distract from developing and improvingmethods in CASP5. The group 29 automatic server Ro-betta provides an excellent example of this point. Robettacombined results of group 38 Pcons2, an automatic serverthat performs well on comparative modeling targets, withresults of the ab initio method Rosetta developed by theBaker group. This combination produced an outstandingprediction for target T0186, a three-domain protein classi-fied as CM and CM/FR(H) for two domains and FR/NF fora third, inserted domain. Observations in previous CASPassessments that different groups performed better ondifferent targets also reinforced the notion that sharingtechniques could lead to better performance. The 3D JURYmethod developed and used by group 6 Rychlewski and bygroup 453 Ginalski took full advantage of this idea.29 Byusing a detailed scoring scheme, 3D-Jury used inputmodels produced by a large set of publicly available foldrecognition servers and picked those that had the mostabundant high scores as predictions.

The results of the fold recognition assessment suggestthat two general types of methods performed well inprotein structure prediction: template-based methods andfragment assembly-based methods. For fold recognitionhomologues, groups using methods that combined modelsbuilt from existing templates with some type of modelrefinement performed well. The procedure used by group453 Ginalski, one of the top scoring groups in both foldrecognition and comparative modeling, provides an excel-lent example of such a method. Ginalski started withtemplates provided by comparative modeling and foldrecognition servers, improved template alignments manu-ally by using various criteria, built homology models fromthese alignments, and improved the resulting models witha combination of available programs and manual inspec-tion. As discussed, this procedure produced a predictioncloser to the experimental target than the closest template(Fig. 10).

The second general type of method has been categorizedas ab initio protein structure prediction. This method typegenerally concentrates on local structural properties andperforms well on fold recognition analogs and new folds.The Rosetta program developed by group 2 (Baker) and theTOUCHSTONE folding algorithm developed by group 10(Skolnick) are both examples of such methods. Rosettaidentified small fragments from a library generated fromexisting structures,30,31 whereas the ab initio foldingsegment of TOUCHSTONE identified conserved contactsfrom multiple weakly significant template fragments foundby threading.32

With the overall performance of fold recognition groupsbeing easily divided into two main classifications (methodsthat work on homologues and methods that work onanalogs) and with the increasing depletion of fold recogni-tion domains into the comparative modeling group, per-haps a change in CASP assessment categories is required.Under the current definition, significant overlaps existbetween assessment categories, especially between thoseof fold recognition and comparative modeling. Althoughthis overlap provides a nice check for assessment methods,it detracts from the very different aspects of model compari-sons that need to be made for each category. For example,the local structural details of side-chains and loops cannotbe assessed in most of the target domains currentlydefined as comparative models due to significant struc-tural divergence. Although such a localized comparison isessential for the methods development of comparativemodeling groups, the task becomes overwhelmed by thehuge number of target domains assigned to the categorythat must be addressed on a more global level. Therefore,we suggest that targets be categorized on the basis of themethods required for assessment, rather than the out-dated methods used for predictions. This type of classifica-tion scheme fits more naturally within the outline ofCASP, because assessors have no knowledge of methodswhen establishing ranks.

In addition to the promising predictions of the topscoring groups outlined in this report, the predictioncommunity as a whole achieved higher overall average


scores (GDT_TS) in CASP5 than in CASP4 (see CASP5progress article33). However, the question remains aboutwhether this better performance actually reflects signifi-cant advances in prediction technology. Groups participat-ing in CASP5 have larger databases of sequence andstructure information available, in addition to a number ofpublicly accessible fold recognition servers. One of themost promising results of the CASP5 assessment of foldrecognition concerns the performance of automatic serv-ers. Predictions by the best performing servers approachthose of the top scoring manual groups. In fact, group 29Robetta performed among the top groups (with a statisti-cally significant rank 3) for the subset of “real” foldrecognition domains using alignment measures on firstmodels, whereas group 45 Pmodel3 and group 40 Pmodelperformed consistently among the top groups using bestmodels.

This assessment identified the top scoring groups for allfold recognition domains and for various subsets of do-mains, including homologues and analogs, using variousstructural and sequence alignment criteria. General trendsin average scores attained by all groups on various targetswere described. A huge amount of data produced in thecourse of our analysis of fold recognition predictions existsthat cannot be contained within this report. Furtheranalysis of this data could provide additional insights intogeneral trends of protein structure prediction and intospecific strategies of individual prediction groups. Inter-ested readers can access the data generated for thisassessment in the files at http://predictioncenter.llnl.gov/casp5/fr/.

ACKNOWLEDGMENTS

We thank the CASP 5 organizers, John Moult and TimHubbard, for asking us to be a part of the CASP experi-ence. We greatly appreciate input and discussions frommany CASP participants: Alexey Murzin, Rob Russell,Patrick Aloy, Anna Tramontano, Veronica Morea, Krzysz-tof Fidelis, Ceslovas Venclovas, and Adam Zemla. Addi-tional thanks goes to the crystallographers and NMRspectroscopists who provided structure data for CASP 5and to the predictors who provided a tremendous amountof work (and fun) for us.

REFERENCES

1. Kinch LN, Qi Y, Hubbard T, Grishin NV. CASP5 target classifica-tion. Proteins 2003;Suppl. 6:340–351.

2. Hubbard TJ. RMS/coverage graphs: a qualitative method forcomparing three-dimensional protein structure predictions. Pro-teins 1999;Suppl 3:15–21.

3. Zemla A, Venclovas C, Fidelis K, Rost B. A modified definition ofSov, a segment-based measure for protein secondary structureprediction assessment. Proteins 1999;34 2:220–223.

4. Zemla A, Venclovas, Moult J, Fidelis K. Processing and evaluationof predictions in CASP4. Proteins 2001;Suppl 5:13–21.

5. Zemla A. LGA program: a method for finding 3-D similarities inprotein structures. Nucleic Acids Res 2003;31:3370–3374.

6. Zemla A, Venclovas C, Moult J, Fidelis K. Processing and analysisof CASP3 protein structure predictions. Proteins 1999;Suppl3:22–29.

7. Holm L, Park J. DaliLite workbench for protein structure compari-son. Bioinformatics 2000;16:566–567.

8. Holm L, Sander C. Protein structure comparison by alignment ofdistance matrices. J Mol Biol 1993;233:123–138.

9. Shindyalov IN, Bourne PE. Protein structure alignment by incre-mental combinatorial extension (CE) of the optimal path. ProteinEng 1998;11:739–747.

10. Ortiz AR, Strauss CE, Olmea O. MAMMOTH (matching molecu-lar models obtained from theory): an automated method for modelcomparison. Protein Sci 2002;11 11:2606–2621.

11. Sippl MJ, Lackner P, Domingues FS, Prlic A, Malik R, AndreevaA, Wiederstein M. Assessment of the CASP4 fold recognitioncategory. Proteins 2001;Suppl:55–67.

12. Levitt M. Competitive assessment of protein fold recognition andalignment accuracy. Proteins 1997;Suppl 1:92–104.

13. Murzin AG. Structure classification-based assessment of CASP3predictions for the fold recognition targets. Proteins 1999;Suppl3:88–103.

14. Lemer CM, Rooman MJ, Wodak SJ. Protein structure predictionby threading methods: evaluation of current techniques. Proteins1995;23:337–355.

15. Altschul SF, Madden TL, Schaffer AA, Zhang J, Zhang Z, MillerW, Lipman DJ. Gapped BLAST and PSI-BLAST: a new generationof protein database search programs. Nucleic Acids Res 1997;25:3389–3402.

16. Altschul SF, Koonin EV. Iterated profile searches with PSI-BLAST—a tool for discovery in protein databases. Trends Bio-chem Sci 1998;23:444–447.

17. Lackner P, Koppensteiner WA, Sippl MJ, Domingues FS. ProSup:a refined tool for protein structure alignment. Protein Eng 2000;13:745–752.

18. Grishin VN, Grishin NV. Euclidian space and grouping of biologi-cal objects. Bioinformatics 2002;18:1523–1534.

19. Koehl P. Protein structure similarities. Curr Opin Struct Biol2001;11:348–353.

20. Zemla A. Proteins 2003;Suppl 6.21. Rost B, Sander C, Schneider R. Redefining the goals of protein

secondary structure prediction. J Mol Biol 1994;235:13–26.22. Sauder JM, Arthur JW, Dunbrack RL Jr. Large-scale comparison

of protein sequence alignment algorithms with structure align-ments. Proteins 2000;40:6–22.

23. Bujnicki JM, Elofsson A, Fischer D, Rychlewski L. LiveBench-2:large-scale automated evaluation of protein structure predictionservers. Proteins 2001;Suppl 5:184–191.

24. Tramontano A, Leplae R, Morea V. Analysis and assessment ofcomparative modeling predictions in CASP4. Proteins 2001;Suppl5:22–38.

25. Tramontano A, Morea V. Assessment of homology based predic-tions in CASP5. Proteins 2003;Suppl 6:352–368.

26. Marti-Renom MA, Madhusudhan MS, Fiser A, Rost B, Sali A.Reliability of assessment of protein structure prediction methods.Structure (Camb) 2002;10 3:435–440.

27. Kendall MG, Stuart A, Ord JK. Kendall’s advanced theory ofstatistics. New York: Oxford University Press; 1987. v. p. 77–80,365–368.

28. Russell RB, Aloy P. Predictions without templates: new folds,secondary structure and contacts in CASP5. Proteins 2003;Suppl.

29. Ginalski K, Elofsson A, Fischer D, Rychlewski L. 3D-Jury: asimple approach to improve protein structure predictions. Bioinfor-matics 2003;19:1015–1018.

30. Simons KT, Kooperberg C, Huang E, Baker D. Assembly of proteintertiary structures from fragments with similar local sequencesusing simulated annealing and Bayesian scoring functions. J MolBiol 1997;268:209–225.

31. Simons KT, Bonneau R, Ruczinski II, Baker D. Ab initio proteinstructure prediction of CASP III targets using ROSETTA. Pro-teins 1999;37:171–176.

32. Kihara D, Lu H, Kolinski A, Skolnick J. TOUCHSTONE: an abinitio protein structure prediction method that uses threading-based tertiary restraints. Proc Natl Acad Sci USA 2001;98:10125–10130.

33. CASP5 Progress. Proteins 1993.34. Esnouf RM. An extensively modified version of MolScript that

includes greatly enhanced coloring capabilities. J Mol GraphModel 1997;15:132–134, 112–113.

CASP5 409

casp5 assessment of fold recognition target...

Documents