document resume ed 429 129 attaining excellence: a ... · document resume. ed 429 129 tm 029 698....

DOCUMENT RESUME

ED 429 129 TM 029 698

AUTHOR McKinney, KayTITLE Attaining Excellence: A Handbook on the Standards for the

Conduct and Evaluation of Research Carried Out by the Officeof Educational Research and Improvement.

INSTITUTION National Educational Research Policy and Priorities Board(ED/OERI), Washington, DC.

REPORT NO PPB-1999-6302PUB DATE 1999-04-00NOTE 40p.PUB TYPE Guides - Non-Classroom (055) Reports Descriptive (141)EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS *Educational Research; Elementary Secondary Education;

*Evaluation Methods; Higher Education; *Peer Evaluation;Program Evaluation; *Research Methodology; *Standards

IDENTIFIERS *Office of Educational Research and Improvement

ABSTRACT

The National Educational Research Policy and PrioritiesBoard has developed standards for the conduct and evaluation of research,development, and dissemination carried out under the auspices of the Officeof Educational Research and Improvement (OERI) of the U.S. Department ofEducation. This handbook explains the scope and significance of thestandards, which require OERI to subject applications for funding to peerreview, to evaluate all existing projects to see how well they areperforming, and to select promising and exemplary programs to share witheducators across the United States. The three appendixes reproduce the fulltext of the standards as they were published in three phases (1995, 1997, and1998) in the "Federal Register." (SLD)

********************************************************************************

Reproductions supplied by EDRS are the best that can be madefrom the original document.

********************************************************************************

CT)

NATIONAL EDUCATIONAL RESEARCH

POLICY & PRIORITIES BOARD

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

la4irs document has been reproduced asreceived from the person or.organizationoriginating IL

0 Minor changes have been made toimprove reproduction quality.

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position or policy.

Attaining Excellence:A Handbook on theStandards for theConduct and Evaluationof Research Carried Outby the Office ofEducational Researchand Improvement

National Educational Research Policy and Priorities BoardU.S. Department of Education

2bal CUPY AVM WI

NATIONAL EDUCATIONAL RESEARCH

POLICY & PRIORITIES BOARD

Attaining Excellence:A Handbook on theStandards for theConduct and Evaluationof Research Carried Outby the Office ofEducational Researchand Improvement

National Educational Research Policy and Priorities BoardU.S. Department of Education

3

U.S. Department of EducationRichard W. RileySecretary

Office of Educational Research and ImprovementC. Kent McGuireAssistant Secretary

National Educational Research Policy and Priorities BoardKenji HakutaChair

Eve M. BitherExecutive Director

April 1999

This publication was prepared under contract to the National Educational Re-search Policy and Priorities Board. Any opinions, findings, or conclusions arethose of the author and do not necessarily reflect the views of the U.S. Depart-ment of Education or any agency of the United States Government.

Comments about this publication may be directed to

National Educational Research Policy and Priorities BoardU.S. Department of Education80 F Street NW, Suite 100Washington, DC 20208-7564Telephone: (202) 208-0692Fax: (202) 219-1528E-mail: [email protected]

Contact:Mary Grace Lucier(202) 219-2253

NERPPB Web site: http://www.ed.gov/offices/OERI/NERPPB/

Aftalning Excellence

National Educational Research Policyand Priorities BoardThe National Educational Research Policy and Priorities Board (NERPPB) isauthorized by the Educational Research, Development, Dissemination, and Im-provement Act of 1994. The Board was established to work collaboratively withthe Assistant Secretary for the Office of Educational Research and Improvement(OERI) to forge a national consensus with respect to a long-term agenda foreducational research, development, dissemination, and the activities of the Office.The Board regularly reviews, evaluates, and publicly comments upon the imple-mentation of its policies by the U.S. Department of Education and the Congress.

The Secretary of the U.S. Department of Education appoints members of theBoard. They represent the research community, school-based professional educa-tors, and individuals who are knowledgeable about the educational needs of theUnited States.

Appointed Members

Kenji Hakuta, ChairProfessorSchool of EducationStanford UniversityStanford, CA

Patricia Ann BaltzTeacherCamino Grove Elementary SchoolArcadia, CA

James E. BottomsSenior Vice PresidentSouthern Regional Education BoardAtlanta, GA

1 Attaining Excellence iii

5

National Educational Research Policy and Priorities Board

Jomills H. Braddock, IIProfessor and ChairDepartment of SociologyUniversity of MiamiMiami, FL

John T. BruerPresidentJames S. McDonnell FoundationSt. Louis, MO

Ann B. ClarkPrincipalVance High School at Governors' VillageCharlotte, NC

Edmund W. GordonProfessor EmeritusYale UniversityNew Haven, CT

Paul D. GorenDirector, Child and Youth DevelopmentProgram on Human and Community DevelopmentThe John D. and Catherine T. MacArthur FoundationChicago, IL

Sharon L. KaganSenior AssociateBush Center in Child Development and Social PolicyYale UniversityNew Haven, CT

Glenda T. LappanProfessorDepartment of MathematicsMichigan State UniversityEast Lansing, MI

6

iv Attaining Excellence

National Educational Research Policy and Priorities Board

Robert W. MarleyAdjunct ProfessorDepartment of Health and Physical EducationWichita State UniversityWichita, KS

Joyce MuhlesteinProject DirectorUtah Parent Information Resource CenterSalt Lake City, UT

Alba A. OrtizProfessorCollege of EducationUniversity of Texas at AustinAustin, TX

Claire L. PeltonDirector of Academic Services/Western OfficeThe College BoardSan Jose, CA

E. Lea SchelkeTeacherTrenton High SchoolTrenton, MI

Ex Officio Members

Assistant Secretary, Office of Educational Research and Improvement,U.S. Department of Education

Director of Research, U.S. Department of DefenseDirector of Research, U.S. Department of LaborDirector, National Science FoundationDirector, National Institutes of HealthChair, National Endowment for the ArtsChair, National Endowment for the HumanitiesLibrarian of CongressDirector, Office of Indian Education Programs, U.S. Department of the Interior

7Attaining Excellence

ForewordOne of the chief responsibilities the Congress gave the National EducationalResearch Policy and Priorities Board (NERPPB) in its authorizing legislation wasto review and approve standards for the conduct and evaluation of all research,development, and dissemination carried out under the auspices of the Office ofEducational Research and Improvement (OERI) of the U.S. Department of Educa-tion. These standards were designed to assure that such activities meet the highestcriterion of professional excellence.

The development of the standards was a lengthy process. The Assistant Secretaryfor the Office of Educational Research and Improvement examined the proceduresutilized by the National Institutes of Health, the National Science Foundation, andother federal agencies, and solicited recommendations from research organiza-tions and from the public. The standards were published in three phases overseveral years, as the law required.

The Board has chosen to complete its mandate with this brief handbook whichexplains the scope and significance of the standards for the research communityand the public, and reproduces in the appendices the full text of the standards aspublished in the Federal Register. We have firm expectations that the standardswill be another powerful engine for eliciting the knowledge upon which educa-tional reform may continue to build.

Kenji HakutaChairNational Educational Research Policy and Priorities Board

8

Attaining Excellence vii

AcknowledgmentsMany individuals contributed to this publication. Special thanks to KayMcKinney, the writer, who summarized the purpose and content of the standards,and to OERI staff members Margo Anderson and Patricia Knight, Who providedcritical technical comments. Additional commentary was provided by MarilouHyson, Donna Hinkle, and Jim Fox. Thanks also to Donna Timm of the NationalLibrary of Education; Bob LeGrand and Diane Magarity of the Media and Infor-mation Services staff at OERI; and to NERPPB staff members Thelma Leenhoutsand Mary Grace Lucier.

9

rAttaining Excellence ix

Table of Contents

National Educational Research Policy and Priorities Board iii

Foreword vii

Acknowledgments ix

Attaining Excellence 1

Appendix A: Part IIDepartment of Education, 34 CFR Part 700,Standards for the Conduct and Evaluation of ActivitiesCarried Out by the Office of Educational Research andImprovement (0ERI)Evaluation of Applications for Grantsand Cooperative Agreements and Proposals for Contracts;Final Rule 7

Appendix B: Part IIDepartment of Education, 34 CFR Part 701,Office of Educational Research and Improvement (OERI)Conduct and Activities Evaluation Standards; Designationof Exemplary and Promising Programs; Final Rule 17

Appendix C: Part VIIDepartment of Education, 34 CFR Part 702,Standards for Conduct and Evaluation of ActivitiesCarried Out by the Office of Educational Research andImprovement (0ERI)Evaluation of the Performance ofRecipients of Grants, Cooperative Agreements, and Contracts;Final Rule 27

1 0Attaining Excellence xi

Attaining ExcellenceBefore trying a new medicine, most Americans want assurance that it has metcertain safety and efficacy standards and earned the approval of the U.S. Food andDrug Administration. But what assurances are there about the reliability of educa-tion programs intended to improve the learning and achievement of the nation'schildren? How can consumers judge their quality, and who will vouch for theireffectiveness?

If the program in question is an activity funded by the U.S. Department ofEducation's Office of Educational Research and Improvement (OERI), educatorsand taxpayers can be assured that it has been subjected to a vigorous peer reviewprocess and that it meets the highest standards of professional excellence.

OERI is the branch of the U.S. Department of Education primarily responsible forfunding research and demonstration projects, collecting education statistics, andsharing information about education programs that work.

OERI has always used a system of peer review, the use of recognized experts toevaluate proposed and funded research and program activities. Now, however,OERI has for the first time defined and published the professional qualificationsindividuals must possess in order to become peer reviewers and a consistent set ofcriteria reviewers must use to determine the success or potential for success of anyprogram funded and disseminated by OERI.

The new standards govern how OERI conducts business. All OERI activitiesmust now meet a specific set of published standards that reflect the highest levelsof professional excellence. The standards not only offer OERI a yardstick bywhich to measure quality, but also ensure that everyone is measured by the sameyardstick. The standards also reinforce OERI's confidence that the projects itfunds will contribute to improving the education of all students.

The standards can help many OERI constituents. For researchers, they offerguidance about OERI's goals and criteria for awarding grants and contracts. Forteachers and administrators, they offer reassurance that the programs OERI fundsand the information it disseminates have met certain high requirements. Forparents, other taxpayers, policymakers, and members of Congress, they offer ameans of holding OERI accountable for its use of federal dollars.

11r----

Attaining ExcellenceL_

[Attaining Excellence

The new standards were developed by the Assistant Secretary, his staff, and bymembers of the National Educational Research Policy and Priorities Board, whichcollaborates with the Assistant Secretary in the development of research priorities.The 15-member Board includes educational researchers, practitioners, parents,and other individuals who have expertise and broad perspectives in the educationfield.

This handbook provides a general overview of the standards and highlights someof their key characteristics. Precise legal explanations of the standards are avail-able from the Federal Register, as indicated in the appendix to this document.

When Does OERI Use Peer Review?

The new standards require OERI to use peer review for three major programactivities: to review applications for funding; to evaluate all existing projects todetermine how well they are performing; and to select promising and exemplaryprograms to share with educators across the country.

Review of Applications

To help ensure that OERI funds only high quality projects, all applications forOERI grants and cooperative agreements and any contract proposal for more than$100,000 must undergo a peer review. Generally, applications and proposals willbe reviewed by three to five expert reviewers.

Evaluation of Funded Projects

In a major move to continuously improve the quality of funded projects, OERInow requires that these projects be evaluated at least twice after they are funded:at least once while the project is under way and again at its conclusion. Theinterim reviews will allow OERI to fine tune project activities and, if necessary, tocorrect any problems. In the past, projects were evaluated but not according to apublished set of standards. All OERI research, development and disseminationactivities, ranging from the smallest purchase orders and commissioned papers tothe largest research projects and national research centers, will now be subjectedto this peer review.

Identifring Exemplary and Promising Programs

Another of OERI's functions is to examine education programs from across thecountry to determine whether they should be designated as "exemplary" or"promising" and disseminated to the field. The new standards require that each


Attaining Excellence

program submitted for consideration be assessed by an expert panel of peerreviewers and judged by a common set of standards. The panels will designate aprogram as "exemplary" if empirical data demonstrate that it is effective in im-proving student achievement, or "promising" if there is sufficient evidence thatthe program shows promise for improving student achievement. Programs may besubmitted to OERI for review at any time by individuals and organizations, orthey may be proposed by the Secretary of Education.

Who May Serve as Peer Reviewers?

It depends on what program is being reviewed. Generally, all peer reviewers mustbe leaders in the field and must have proven expertise in the subject matter theyare reviewing. They also must meet stringent conflict-of-interest regulations. TheSecretary of Education selects each peer reviewer and may solicit nominations forpeer reviewers from professional associations, nationally recognized experts, andother sources. Some of the more specific qualifications include:

Individuals reviewing applications for funding must have knowledgeabout education policy or practice, and theory or methodology in thesubject they are reviewing.

Reviewers evaluating how well existing grants, cooperative agreements,and contracts are performing must have knowledge and expertise in thesubject area being reviewed. To evaluate the performance of the recipients,their qualifications may include expert knowledge about theory, methods,and education policies and practices as well as practical experience relatedto the topic they are evaluating and in managing complex organizations.

Experts selected to help OERI designate "exemplary" or "promising"programs must have in-depth knowledge of the subject area or content ofthe program or group of programs they will be evaluating. To ensure thatthe peer reviewers represent both practice and research, each panel mustinclude at least one current teacher, principal, or other school-based orcommunity-based professional, and at least one individual experienced inevaluating educational programs.

What Criteria Will Peer Reviewers Use?

The answer depends on which of the three activities is being reviewed. While thecriteria are different for each of the three, all were designed with one goal inmind: to ensure that everything OERI does is oriented towards its customers,


3


credible to researchers, and useful to practitioners. Here is a summary of the threesets of criteria.

Criteria for Selecting Projects for Funding

Grants and cooperative agreements are selected for funding according to a set ofevaluation criteria specifically designed to identify the best. Peer reviewersconsider one or more of the following:

> Is the project nationally significant?

> What is the quality of the project's design?

> What are the qualifications and potential contributions of the projectpersonnel?

> Is the proposed budget adequate to complete the project?

> Is the management plan adequate?

Answering these questions will help reviewers determine whether to recommendto OERI that the project be funded. For each competition, criteria are furtherdefined to target the specific purposes and expected outcomes for funded activi-ties.

Contract proposals will be judged by specific criteria, which may include but arenot limited to:

> Technical excellence;

> Management capability;

> Personnel qualifications;

> Prior experience;

> Past performance; and

> Schedule compliance.

1 4

4 Attaining Excellence


Criteria for Evaluating Funded Projects

To help OERI determine how well each of its funded projects is performing, peerreviewers will examine the projects to see if each is meeting a standard set of fourcriteria. The same criteria will be used for both interim and final reviews. Thecriteria will be especially useful during interim reviews because they will allowOERI to assess a project's progress and determine if technical assistance is neces-sary to ensure that the project achieves its objectives.

Peer reviewers will thoroughly review each project and look for evidence that theproject:

Is being well implemented and managed;

Is of high quality;

Is producing products and services that are useful; and

Will have an impact on the field of education.

OERI will ask reviewers to consider additional criteria for specific types ofprojects.

Criteria for Selecting Exemplary and Promising Programs

If they are to use them with confidence, educators must have assurance that theprograms OERI deems "exemplary" or "promising" truly meet these high stan-dards. OEM requires its peer review panels to use four criteria when examiningprograms that apply or are nominated for this honor. The standards help to assurethat these programs are high quality, research-based programs that have providedevidence that they have improved teaching, learning, or both, or have demon-strated other worthy educational performance outcomes. To determine if a pro-gram should be recommended as "exemplary" or "promising," peer reviewersmust evaluate:

Evidence of a program's success;

Quality of the program;

Educational significance; and

The program's replicability.15


Aftaining ExcellenceJ

OERI and the National Educational Research Policy and Priorities Board haveinvested much thought and energy in developing these new standards. Theysought input from practitioners, policymakers, researchers, the public, and otherfederal agencies in an effort to improve the quality of the agency's programs andprojects. The new standards will enhance OERI's ability to provide the broad fieldof education and the nation access to programs and activities that meet the highestlevel of professional excellence.

For further information, consult:

OERI's Home page: www.ed.gov/offices/OERI/Federal Register Home page: www.nara.gov/fedreg/

16

6 Attaining Excellence

Appendix A

Part IIDepartment of Education34 CFR Part 700

Standards for the Conduct andEvaluation of Activities Carried Out bythe Office of Educational Research andImprovement (0ER1)Evaluation ofApplications for Grants andCooperative Agreements and Proposalsfor Contracts; Final Rule

17I Attaining Excellence 7

AAMM-

1111=-=PIM

IM Ap p.. 'MI1MMINIL

NMIMY 11111

AM IIM IMPMI MI IMPMI IMI =M. b.-. IM I -AM=I

AMM

41II

IMIMI. imlNINI we1m 10im. Ammo,

.AinNgIN

IMI IMMNM ...AMA. MiM.IMANM 1-1I IIMI

Pr...7a 'MIPMMI

PM PIN 11.IM IM MIMA A. pp PMIMIL. AMP IMI

PMAMP=I

Om. IMMIMMMIr.la, =1/el=1Mr MO 1M

MI MI MPIM MA MIMI MI MAML MI IMBA im. ARAM

11II

AMP

AMM1

/MP MA 1M

1MI=AMEMPJ 34 CFR Part 700

Standards for the Conduct andEvaluation of Activities Carried Out by

ThursdaySeptember 14, 1995

Part II

Department ofEducation

the Office Educational Research and- --- -- Improvement (0ERI)Evaluation of, _ - _ Applications for Grants and Cooperative- - -- - -- - -- - -- - - Agreements and Proposals for Contracts;-- Rule-..=II =NI IML

MI MI= =MI MNIML AMP116- ...11

NAII..,ANIMP!7.

AIMIMINAM

I11

MA

MMMI.-.

MP=MIIMIMIIMI

INMI

MIMA

AIMM

MI.1-.il

...1111/

18

47808 Federal Register / Vol. 60, No. 178 / Thursday, September 14, 1995 / Rules and Regulations

DEPARTMENT OF EDUCATION

34 CFR Part 700

RIN 1850AA51

Standards for the Conduct andEvaluation of Activities Carried Out bythe Office of Educational Research andImprovement (OERI)Evaluation ofApplications for Grants andCooperative Agreements andProposals for Contracts

AGENCY: Department of Education.ACTION: Final Regulations.

SUMMARY: The Assistant Secretary forEducational Research and Improvementestablishes final regulations that setstandards for the evaluation ofapplications for grants and cooperativeagreements and proposals for contracts.The development of these standards isrequired by the Office of EducationalResearch and Improvement'sauthorizing legislation, the "EducationalResearch, Development, Dissemination,and Improvement Act of 1994." Thestandards ensure that these applicationand proposal evaluation activities meetthe highest standards of professionalexcellence.EFFECTIVE DATE: These regulations takeeffect October 16, 1995.FOR FURTHER INFORMATION CONTACT:Edward J. Fuentes, U.S. Department ofEducation, 555 New Jersey Avenue,NW., Room 600, Washington, DC20208-5530. Telephone (202) 219-1895.Internet electronic mail address: [email protected]. Individuals whouse a telecommunications device for thedeaf (TDD) may call the FederalInformation Relay Service (FIRS) at 1800-877-8339 between 8 a.m. and 8p.m., Eastern time, Monday throughFriday.SUPPLEMENTARY INFORMATION: On March31, 1994, President Clinton signed Pub.L. 103-227, which includes Title IXthe "Educational Research,Development, Dissemination, andImprovement Act of 1994" (the "Act").The Act restructured the Office ofEducational Research and Improvement(OERI) and endowed it with a broadmandate to conduct an array of research,development, dissemination, andimprovement activities aimed atstrengthening the education of allstudents. The Act also required theestablishment of a National EducationalResearch Policy and Priorities Board(the "Board") to work collaborativelywith the Assistant Secretary to identifypriorities to guide the work of OERI.

The legislation directed the AssistantSecretary to develop, in consultation

with the Board, such standards as maybe necessary to govern the conduct andevaluation of all research, development,and dissemination activities carried outby the Office to ensure that suchactivities meet the highest standards ofprofessional excellence. The legislationrequired that the standards bedeveloped in three phases. Theseregulations implement the first phase ofthe standards. The Assistant Secretarywill publish at a later date additionalproposed regulations to implement theremaining standards in accordance withthe timelines established in the Act. Thelegislation requires the Board to reviewand approve the final standards.

On June 7, 1995, the AssistantSecretary for Educational Research andImprovement published a notice ofproposed rulemaking (NPRM) in theFederal Register (60 FR 30160).

Analysis of Comments and ChangesIn response to the Assistant

Secretary's invitation in the NPRM, fiveparties submitted comments on theproposed regulations. An analysis of thecomments and of the changes in theregulations since publication of theNPRM follows.

Issues are grouped according tosubject with appropriate sections of theregulations referenced in parentheses.Substantive issues are discussed underthe section of the regulations to whichthey pertain. In addition to the publiccomment, the comments of the Board'sCommittee on Standards are alsoaddressed. That Committee met inpublic session on August 4, 1995, toprovide final input for the Board and toact on the Board's behalf in approvingthe standards. Technical and otherminor changesand suggested changesthe Secretary is not legally authorized tomake under the applicable statutoryauthorityare not addressed.Qualifications of Peer Reviewers(§ 700.11)

Comments: Two commenters believedthat § 700.11 should require the majorityof reviewers for a given application tomeet the qualifications in§ 700.11(a)(1)(i). These commenterswere concerned that requiringindividual reviewers to possess onlyone or more of the qualifications listedunder §700.11(a)(1) might result in fewor no reviewers for a given applicationpossessing demonstrated expertise inthe subject of the competition(§700.11(a)(1)(i)). One of thesecommenters also felt that each group ofreviewers for a given application shouldinclude at least one reviewer with "in-depth knowledge of policy and practicein the field of education"

BEST 4(43Y AVAILABLEis

(§700.11(a)(1)(ii)), and at least onereviewer with "in-depth knowledge oftheoretical perspectives ormethodological approaches relevant tothe subject of the competition"(.§ 700.11(a)(1)(iii)). Another commenterfelt that all reviewers for researchprojects should possess technicalexpertise regarding the theoretical andmethodological aspects of the grantapplications.

Discussion: The Secretary believes itis important for all reviewers to possesseach of the qualifications in§700.11(a)(1). The Board agreed that itis important for all reviewers to possesseach of the qualifications in§700.11(a)(1) but recommended that§700.11(a)(1)(ii) be modified to allow areviewer to be deemed qualified on thebasis of in-depth knowledge of policy"or" practice in the field of educationrather than both.

Changes: The Secretary has revised§700.11(a)(1) to require each reviewerto possess each of the qualifications of(a)(1)(i), (a)(1)(ii) and (a)(1)(iii) andreplaced the word "and" in (a)(1)(ii)with the word "or".

Comments: One commenter expressedconcern that the word "relevant" in§700.11(a)(1)(i) was inadequate forspecifying that reviewers have directexpertise in the topic area of grantapplications they review.

Discussion: The Secretary agrees withthis commenter.

Changes: The Secretary replaced thewords "relevant to the subject area"with the words "in the subject area" in§700.11(a)(1)(i) and also in§700.11(a)(1)(iii).

Convening Reviewers to DiscussUnsolicited Applications (§ 700.21)

Comments: One commenter suggestedsubstituting the word "may" for theword "will" in § 700.21(c) so as not torequire the convening of reviewers in allinstances. The commenter believed thatit may not be necessary to convenereviewers to discuss the strengths andweaknesses of unsolicited applications.

Discussion: The Secretary believesthat discussions of each application'sstrengths and weaknesses allowsreviewers to share perspectives andprovide a higher quality of review.Therefore, the Secretary believes thatsuch discussions should, in general, berequired. However, the Secretary agreesthat in the case of unsolicitedapplications, it may not be necessary toconvene reviewers.

Changes: The Secretary has added anew §700.21(c)(2), which allows theSecretary to use discretion indetermining whether to convenereviewers of unsolicited applications.

Federal Register / Vol. 60, No. 178 / Thursday, September 14, 1995 / Rules and Regulations 47809

Comments: One commenter felt that itwas important that applicants receivethe written comments regarding thestrengths and weaknesses of theirapplications at the same time as theapplicants are notified of acceptance orrejection. Another commenterrecommended that the OERI standardsaddress the issue of OERI'scommunicating the results of thecompetition to the applicants and thelarger community.

Discussion: The Secretary recentlymodified Department procedures toprovide both successful andunsuccessful applicants earliernotification of funding decisions. Inmost cases, reviewer commentsaccompany these notifications.However, particularly for competitionsthat generate large numbers ofapplications, reviewer comments aresent at a later date so that this earlynotification of funding decisions neednot be delayed. The Secretary routinelyissues press releases to inform thepublic of the results of eachcompetition.

Changes: None.

Evaluating Grant and CooperativeAgreement Applications and ContractProposals (§ 700.21 and § 700.22)

Comments: One commenter statedthat it was not clear that the Secretarywill be constrained or informed by theresults of the evaluations carried out bythe peer reviewers. The commenterrecommended changes to clarify that:(1) Each reviewer will provide arecommendation to fund or not to fundeach application, accompanied by anumerical rating of the application; and(2) the Secretary will rely on numericalratings given by the peer reviewers inrank ordering the applications.

Discussion: The Secretary agrees that§ 700.21(e) should be revised to clarifythat the Secretary prepares a rank orderbased solely on the peer reviewers'ratings. However, the Secretary believesthat, in selecting applications for award,he must consider factors in addition tothe applicants' rankings and the fundingrecommendations of the peer reviewers.These other factors, specified in§ 700.40, include performance of theapplicant under a prior award, theamount of funds available for thecompetition, and any other informationrelevant to a priority or other statutoryor regulatory requirement applicable tothe selection of applications for newawards.

Changes: The Secretary has revised§ 700.21(e) to claiify that the rank orderis based solely on the peer reviewers'ratings for each application.

Comments: One commenter pointedout that § 700.22(d), regarding theevaluation of contract proposals,enables reviewers to assign proposals tothe category "capable of being madeacceptable." The commenterrecommended that a similar category beadded to § 700.21(d), relating to theevaluation of grants and cooperativeagreements. The commenter believesthat such a change could allowapplicants an opportunity to fix minorproblems in otherwise outstanding grantapplications and still be eligible forfunding.

Discussion: The Federal AcquisitionRegulations, which govern the Federalgovernment's contract procurementsexpressly recommend the establishmentof a category "capable of being madeacceptable." In that grant competitionsare held to determine which applicantsare to receive the benefit of Federalfinancial assistance for the activitiesapplicants propose, rather than todetermine which applicant orapplicants will be contracted to provideservices according to governmentspecifications, fairness would dictatethat if one grant applicant is allowed torevise its application, all other grantcompetitors should be provided thesame opportunity. In addition, grantcompetitions typically generate manymore applications than do contractcompetitions. There are often manymore highly rated applicants than thereare funds available for awards, and sothere is no need to allow competitors asecond chance to make theirapplications fundable. As a practicalmatter, applications that are trulyoutstanding, but have minor problems,are likely to be rated highly, with theminor problems addressed duringnegotiation of the grant award. TheBoard discussed the issue raised by thiscommenter. The Board was concernedthat reviewers of proposals for contractshad three categories in which theycould place contract proposals whilereviewers of grant and cooperativeagreement applications only had twocategories. The Board recommendedthat a third category be added under§ 700.21(d) that would allow reviewersto distinguish between projects that theywould recommend for funding andthose that they would highlyrecommend.

Changes: The Secretary has added anew category of "highly recommendedfor funding" under § 700.21(d).

Comment: After consultation with theBoard, the Secretary has determinedthat the evaluation criteria related toprior performance of applicants underpreviously funded grants or cooperativeagreements (§700.30(e)(3)(ii)(D) and

20

§700.30(e)(4)(ii)(E)) would requireapplicants to provide that informationfor evaluation by the peer reviewers.This information is already available tothe Secretary and will be another factorconsidered by the Secretary in makingaward decisions under §700.40(a)(3).

Discussion: The evaluation criteriaunder §700.30(e)(3)(ii)(D) and§ 700.30(e)(4)(ii)(E) duplicate the factorsin §700.40(a)(3) and thus impose anunnecessary burden on applicants.

Changes: § 700.30(e)(3)(ii)(D) and§700.30(e)(4)(ii)(E) are deleted.

Paperwork Reduction Act of 1980Section 700.30 contains information

collection requirements. As required bythe Paperwork Reduction Act of 1980,the Department of Education submitteda copy of this section to the Office ofManagement and Budget (OMB) for itsreview, and it was approved by OMB.(44 U.S.C. 3504(h))

These regulations affect the followingtypes of entities eligible to apply forgrants and cooperative agreements: Stateor local governments, businesses orother for profit organizations, nonprofitinstitutions, and any combinations ofthese types of entities. The Departmentneeds and uses the information toevaluate applications for funding.

Annual public reporting andrecordkeeping burden for this collectionof information is estimated to rangefrom 15 hours for each of theapproximately 750 applicationsexpected for a field initiated studycompetition to 150 hours for ten orfewer applications expected for anational research center. Therefore, theactual burden will be determined by thetype of project to be supported in theparticular competition.

Intergovernmental ReviewThis program is subject to the

requirements of Executive Order 12372and the regulations in 34 CFR Part 79.The objective of the Executive order isto foster an intergovernmentalpartnership and a strengthenedfederalism by relying on processesdeveloped by State and localgovernments for coordination andreview of proposed Federal financialassistance.

In accordance with the order, thisdocument is intended to provide earlynotification of the Department's specificplans and actions for this program.

Assessment of Educational hnpactBased on the response to the proposed

regulations and on its own review, theDepartment has determined that theregulations in this document do notrequire transmission of information that


is being gathered by or is available fromany other agency or authority of theUnited States.

List of Subjects in 34 CFR Part 700

Education, Educational research,Elementary and secondary education,Government contracts, Grantprograms-education, Libraries,Reporting and recordkeepingrequirements.

Dated: September 11, 1995.Richard W. Riley,Secretary of Education.Sharon P. Robinson,Assistant Secretary for Educational Researchand Improvement.(Catalog of Federal Domestic AssistanceNumber does not apply.)

The Secretary amends Chapter VII ofTitle 34 of the Code of FederalRegulations by adding a new Part 700 toread as follows:

PART 700-STANDARDS FOR THECONDUCT AND EVALUATION OFACTIVITIES CARRIED OUT BY THEOFFICE OF EDUCATIONALRESEARCH AND IMPROVEMENT(0ERO-EVALUATION OFAPPLICATIONS FOR GRANTS ANDCOOPERATIVE AGREEMENTS ANDPROPOSALS FOR CONTRACTS

Subpart A-GeneralSec.700.1 What is the purpose of these

standards?700.2 What activities must be governed by

these standards?700.3 What additional activities may be

governed by these standards?700.4 What definitions apply?700.5 What are the processes of open

competition?Subpart 13-Selection of Peer Reviewers700.10 When is the peer review process

used?700.11 Who may serve as peer reviewers?700.12 What constitutes a conflict of

interest for grants and cooperativeagreements?

700.13 What constitutes a conflict ofinterest for contracts?

Subpart C-The Peer Review Process700.20 How many peer reviewers will be

used?700.21 How are applications for grants and

cooperative agreements evaluated?700.22 How are proposals for contracts

evaluated?

Subpart D-Evaluation Criteria700.30 What evaluation criteria are used for

grants and cooperative agreements?700.31 What additional evaluation criteria

shall be used for grants and cooperativeagreements?

700.32 What evaluation criteria shall beused for contracts?

Subpart E-Selection for Award700.40 How are grant and cooperative

agreement applications selected foraward?

700.41 How are contract proposals selectedfor award?

Authority: 20 U.S.C. 6011(i).

Subpart A-General

§700.1 What is the purpose of thesestandards?

(a) The standards in this partimplement section 912(i) of theEducational Research, Development,Dissemination, and Improvement Act of1994.

(b) These standards are intended toensure that activities carried out by theOffice of Educational Research andImprovement (the Office) meet thehighest standards of professionalexcellence.(Authority: 20 U.S.C. 6011(i)(1))

§700.2 What activities must be governedby these standards?

(a) The standards in this part arebinding on all activities carried out bythe Office using funds appropriatedunder section 912(m) of the EducationalResearch, Development, Dissemination,and Improvement Act of 1994.

(b) Activities carried out with fundsappropriated under section 912(m) ofthe Act include activities carried out bythe following entities or programs:

(1) The National Research Institutes.(2) The Office of Reform Assistance

and Dissemination.(3) The Educational Resources

Information Center Clearinghouses.(4) The Regional Educational

Laboratories.(5) The Teacher Research

Dissemination Demonstration Program.(6) The Goals 2000 Community

Partnerships Program.(7) The National Educational Research

Policy and Priorities Board.(Authority: 20 U.S.C. 6011(i)(1))

§700.3 What additional activities may begoverned by these standards?

(a) The Secretary may elect to applythe standards in this part to activitiescarried out by the Department usingfunds appropriated under an authorityother than section 912(m) of the Act.

(b)(1) If the Secretary elects to applythese standards to a competition for newgrant or cooperative agreement awards,the Secretary announces, in a noticepublished in the Federal Register, theextent to which these standards areapplicable to the competition.

(2) If the Secretary elects to applythese standards to a solicitation for acontract award, the Secretary announces

21

in the request for proposals the extentto which these standards are applicableto the solicitation.(Authority: 20 U.S.C. 6011(i)

§700.4 What definitions apply?(a) Definitions in the Educational

Research, Development, Dissemination,and Improvement Act of 1994. Thefollowing terms used in this part aredefined in 20 U.S.C. 6011(1):DevelopmentDisseminationEducational Research

OafficoeNtinal Research InstituteTechnical Assistance

(b) Definitions in EducationDepartment General AdministrativeRegulations. The following terms usedin this part are defined in 34 CFR 77.1:ApplicantApplicationAwardDepartmentGrantProjectSecretary

(c) Definitions in the FederalAcquisition Regulation. The followingterms used in this part are defined in 48CFR Chapter 1:Contracting OfficerEmployee of an AgencyProposalSolicitation

(d) Other definitions. The followingdefinitions also apply to this part:

Act means the Educational Research,Development, Dissemination, andImprovement Act of 1994 (Title IX ofPub. L. 103-227, 108 Stat. 212).

EDAR means the EducationDepartment Acquisition Regulation, 48CFR Chapter 34.

EDGAR means the EducationDepartment General AdministrativeRegulations, 34 CFR Parts 74, 75, 77, 79,80, 81, 82, 85 and 86. FAR means theFederal Acquisition Regulation, 48 CFRChapter 1.(Authority: 20 U.S.C. 6011)

§700.5 What are the processes of opencompetition?

The Secretary uses a process of opencompetition in awarding or enteringinto all grants, cooperative agreements,and contracts governed by thesestandards. The processes of opencompetition are the following:

(a) For all new awards for grants andcooperative agreements, the Secretarywill make awards pursuant to theprovisions of EDGAR with the exceptionof the provisions in 34 CFR 75.100(c)(5),75.200(b)(3), (b)(5), 75.210, and75.217(b)(1), (b)(2), (c), and (d); and


(b) For contracts, the Department willconduct acquisitions pursuant to thispart in accordance with therequirements of the Competition inContracting Act, 41 U.S.C. 253, and theFAR.

(Authority: 20 U.S.C. 6011(i)(2); 41 U.S.C.253)

Subpart BSelection of PeerReviewers

§700.10 When is the peer review processused?

The Secretary uses a peer reviewprocess

(a) To review and evaluate allapplications for grants and cooperativeagreements and proposals for thosecontracts that exceed $100,000;

(b) To review and designateexemplary and promising programs inaccordance with section 941(d) of theAct; and

(c) To evaluate and assess theperformance of all recipients of grantsfrom and cooperative agreements andcontracts with the Office.(Authority: 20 U.S.C. 6011(i)(2)(B))

§700.11 Who may serve as peerreviewers?

(a) An individual may serve as a peerreviewer for purposes of reviewing andevaluating applications for new awardsfor grants and cooperative agreementsand contract proposals if theindividual

(1) Possesses the followingqualifications:

(i) Demonstrated expertise, includingtraining and experience, in the subjectarea of the competition.

(ii) In-depth knowledge of policy orpractice in the field of education.

(iii) In-depth knowledge of theoreticalperspectives or methodologicalapproaches in the subject area of thecompetition; and

(2) Does not have a conflict of interest,as determined in accordance with§ 700.12.

(b) For each competition for newawards for grants and cooperativeagreements

(i) Department staff may not serve aspeer reviewers except in exceptionalcircumstances as determined by theSecretary; and

(ii) The majority of reviewers may bepersons not employed by the FederalGovernment.

(2) For each review of an unsolicitedgrant or cooperative agreementapplication

(i) Department employees may assistthe Secretary in making an initialdetermination under 34 CFR 75.222(b);and

(ii) Department employees may notserve as peer reviewers in accordancewith 34 CFR 75.222(c).

(c) To the extent feasible, theSecretary selects peer reviewers for eachcompetition who represent a broadrange of perspectives.(Authority: 20 U.S.C. 6011(i)(2)(B))

§ 700.12 What constitutes a conflict ofinterest for grants and cooperativeagreements?

(a) Peer reviewers for grants andcooperative agreements are consideredemployees of the Department for thepurposes of conflicts of interestanalysis.

(b) As employees of the Department,peer reviewers are subject to theprovisions of 18 U.S.C. 208, 5 CFR2635.502, and the Department's policiesused to implement those provisions.(Authority: 20 U.S.C. 6011(i)(2)(B))

§700.13 What constitutes a conflict ofInterest for contracts.

(a) Peer reviewers for contractproposals are considered employees ofthe Department in accordance withFAR, 48 CFR 3.104-4(h)(2).

(b) As employees of the Department,peer reviewers are subject to theprovisions of the FAR, 48 CFR Part 3Improper Business Practices andPersonal Conflict of Interest.(Authority: 41 U.S.C. 423)

Subpart CThe Peer Review Process

§700.20 How many peer reviewers will beused?

(a) Each application for a grant orcooperative agreement award must bereviewed and evaluated by at least threepeer reviewers except

(1) For those grant and cooperativeagreement awards under $50,000, fewerthan three peer reviewers may be usedif the Secretary determines thatadequate peer review can be obtainedusing fewer reviewers; and

(2) For those grant and cooperativeagreement awards of more than$1,000,000, at least five reviewers mustbe used.

(b) Each contract proposal must beread by at least three reviewers unlessthe contracting officer determines thatan adequate peer review can be obtainedby using fewer reviewers.

(c) Before releasing contract proposalsto peer reviewers outside the FederalGovernment, the contracting officershall comply with FAR, 48 CFR 15.4132(f).

(Authority: 20 U.S.C. 6011(i)(2)(B))

22

§ 700.21 How are applications for grantsand cooperative agreements evaluated?

(a) Each peer reviewer must be givena number of applications to evaluate.

(b) Each peer reviewer shall(1) Independently evaluate each

application;(2) Evaluate and rate each application

based on the reviewer's assessment ofthe quality of the application accordingto the evaluation criteria and theweights assigned to those criteria; and

(3) Support the rating for eachapplication with concise writtencomments based on the reviewer'sanalysis of the strengths and weaknessesof the application with respect to eachof the applicable evaluation criteria.

(c) (1) Except as provided inparagraph (c)(2) of this section, aftereach peer reviewer has evaluated andrated each application independently,those reviewers who evaluated acommon set of applications areconvened to discuss the strengths andweaknesses of those applications. Eachreviewer may then independentlyreevaluate and re-rate an applicationwith appropriate changes made to thewritten comments.

(2) Reviewers are not convened todiscuss an unsolicited applicationunless the Secretary determines thatdiscussion of the application's strengthsand weaknesses is necessary.

(d) Following discussion and anyreevaluation and re-rating, reviewersshall independently place eachapplication in one of three categories,either "highly recommended forfunding," "recommended for funding"or "not recommended for funding."

(e) After the peer reviewers haveevaluated, rated, and made fundingrecommendations regarding theapplications, the Secretary prepares arank order of the applications basedsolely on the peer reviewers' ratings.(Authority: 20 U.S.C. 6011(i)(2)(C))

§ 700.22 How are proposals for contractsevaluated?

(a) Each peer reviewer must be givena number of technical proposals toevaluate.

(b) Each peer reviewer shall(1) Independently evaluate each

technical proposal;(2) Evaluate and rate each proposal

based on the reviewer's assessment ofthe quality of the proposal according tothe technical evaluation criteria and theimportance or weight assigned to thosecriteria; and

(3) Support the rating for eachproposal with concise writtencomments based on the reviewer'sanalysis of the strengths and weaknessesof the proposal with respect to each of


the applicable technical evaluationcriteria.

(c) After each peer reviewer hasevaluated each proposal independently,those reviewers who evaluated acommon set of proposals may beconvened to discuss the strengths andweaknesses of those proposals. Eachreviewer may then independentlyreevaluate and re-rate a proposal withappropriate changes made to the writtencomments.

(d) Following discussion and anyreevaluation and re-rating, reviewersshall rank proposals and advise thecontracting officer of each proposal'sacceptability for contract award as"acceptable," "capable of being madeacceptable without majormodifications," or "unacceptable."Reviewers may also submit technicalquestions to be asked of the offerorregarding the proposal.(Authority: 20 U.S.C. 6011(i)(2)(C))

Subpart DEvaluation Criteria

§700.30 What evaluation criteria are usedfor grants and cooperative agreements?

(a) Except as provided in paragraph(d) of this section, the Secretaryannounces the applicable evaluationcriteria for each competition and theassigned weights in a notice publishedin the Federal Register or in theapplication package.

(b) In determining the evaluationcriteria to be used in each grant andcooperative agreement competition, theSecretary selects from among theevaluation criteria in paragraph (e) ofthis section and may select from amongthe specific factors listed under eachcriterion.

(c) The Secretary assigns relativeweights to each selected criterion andfactor.

(d) In determining the evaluationcriteria to be used for unsolicitedapplications, the Secretary selects fromamong the evaluation criteria inparagraph (e) of this section, and mayselect from among the specific factorslisted under each criterion, the criteriawhich are most appropriate to evaluatethe activities proposed in theapplication.

(e) The Secretary establishes thefollowing evaluation criteria:

(1) National significance.(i) The Secretary considers the

national significance of the proposedproject.

(ii) In determining the nationalsignificance of the proposed project, theSecretary may consider one or more ofthe following factors:

(A) The importance of the problem orissue to be addressed.

(B) The potential contribution of theproject to increased knowledge orunderstanding of educational problems,issues, or effective strategies.

(C) The scope of the project.(D) The potential for generalizing

from project findings or results.(E) The potential contribution of the

project to the development andadvancement of theory and knowledgein the field of study.

(F) Whether the project involves thedevelopment or demonstration ofcreative or innovative strategies thatbuild on, or are alternatives to, existingstrategies.

(G) The nature of the products (suchas information, materials, processes, ortechniques) likely to result from theproject and the potential for theireffective use in a variety of othersettings.

(H) The extent and quality of plans fordisseminating results in ways that willallow others to use the information.

(2) Quality of the project design.(i) The Secretary considers the quality

of the design of the proposed project.(ii) In determining the quality of the

design of the proposed project, theSecretary may consider one or more ofthe following factors:

(A) Whether the goals, objectives, andoutcomes to be achieved by the projectare clearly specified and measurable.

(B) Whether there is a conceptualframework underlying the proposedactivities and the quality of thatframework.

(C) Whether the proposed activitiesconstitute a coherent, sustained programof research and development in thefield, including a substantial addition toan ongoing line of inquiry.

(D) Whether a specific research designhas been proposed, and the quality andappropriateness of that design,including the scientific rigor of thestudies involved.

(E) The extent to which the researchdesign includes a thorough, high-qualityreview of the relevant literature, a high-quality plan for research activities, andthe use of appropriate theoretical andmethodological tools, including those ofa variety of disciplines, whereappropriate.

(F) The quality of the demonstrationdesign and procedures for documentingproject activities and results.

(G) The extent to which developmentefforts include iterative testing ofproducts and adequate quality controls.

(H) The likelihood that the design ofthe project will successfully address theintended, demonstrated educationalneed or needs.

(I) How well and innovatively theproject addresses statutory purposes,

23

requirements, and any priority orpriorities announced for the program.

U) The quality of the plan forevaluating the functioning and impactof the project, including the objectivityof the evaluation and the extent towhich the methods of evaluation areappropriate to the goals, objectives, andoutcomes of the project.

(3) Quality and potentialcontributions of personnel.

(i) The Secretary considers the qualityand potential contributions of personnelfor the proposed project.

(ii) In determining the quality andpotential contributions of personnel forthe proposed project, the Secretary mayconsider one or more of the followingfactors:

(A) The qualifications, includingtraining and experience, of the projectdirector or principal investigator.

(B) The qualifications, includingtraining and experience, of key projectpersonnel.

(C) The qualifications, includingtraining and experience, of proposedconsultants or subcontractors.

(4) Adequacy of resources.(i) The Secretary considers the

adequacy of resources for the proposedproject.

(ii) In determining the adequacy ofresources for the proposed project, theSecretary may consider one or more ofthe following factors:

(A) The adequacy of support from thelead applicant organization.

(B) The relevance and commitment ofeach partner in the project to theimplementation and success of theproject.

(C) Whether the budget is adequate tosupport the project.

(D) Whether the costs are reasonablein relation to the objectives, design, andpotential significance of the project.

(E) The potential for continuedsupport of the project after federalfunding ends.

(5) Quality of the management plan.(i) The Secretary considers the quality

of the management plan of the proposedproject.

(ii) In determining the quality of themanagement plan of a proposed project,the Secretary may consider one or moreof the following factors:

(A) The adequacy of the managementplan to achieve the objectives of theproject, including the specification ofstaff responsibility, timelines, andbenchmarks for accomplishing projecttasks.

(B) The adequacy of plans forensuring high-quality products andservices.

(C) The adequacy of plans forensuring continuous improvement inthe operation of the project.


(D) Whether time commitments of theproject director or principal investigatorand other key personnel are appropriateand adequate to meet project objectives.

(E) How the applicant will ensure thata diversity of perspectives are brought tobear in the operation of the project,including those of parents and teachers,where appropriate.

(F) How the applicant will ensure thatpersons who are otherwise eligible toparticipate in the project are selectedwithout regard to race, color, nationalorigin, gender, age, or disability.

(G) The adequacy of plans forwidespread dissemination of projectresults and products in ways that willassist others to use the information.(Approved by the Office of Management andBudget under control number 1850-0723)(Authority: 20 U.S.C. 6011(i)(2)(D)(ii))

§ 700.31 What additional evaluationcriteria shall be used for grants andcooperative agreements?

In addition to the evaluation criteriaestablished in § 700.30(e), the Secretaryuses criteria or factors specified in theapplicable program statute to evaluateapplications for grants and cooperativeagreements.(Authority: 20 U.S.C. 6011(i)(2)(D)(ii))

§700.32 What evaluation criteria shall beused for contracts?

(a) The evaluation criteria to beconsidered in the technical evaluationof contract proposals are contained inthe FAR at 48 CFR 15.605. Theevaluation criteria that apply to anacquisition and the relative importanceof those factors are within the broaddiscretion of agency acquisitionofficials.

(b) At a minimum, the evaluationcriteria to be considered must includecost or price and quality. Evaluationfactors related to quality are calledtechnical evaluation criteria.

(c) Technical evaluation criteria mayinclude, but are not limited to, thefollowing:

(1) Technical excellence.(2) Management capability.(3) Personnel qualifications.(4) Prior experience.(5) Past performance.(6) Schedule compliance.

(Authority: 20 U.S.C. 6011(i)(2)(D)(ii))

Subpart ESelection for Award

§ 700.40 How are grant and cooperativeagreement applications selected for award?

(a) The Secretary determines the orderin which applications will be selectedfor grants and cooperative agreementawards. The Secretary considers the

2 4

following in making thesedeterminations:

(1) An applicant's ranking.(2) Recommendations of the peer

reviewers with regard to funding or notfunding.

(3) Information concerning anapplicant's performance and use offunds under a previous Federal award.

(4) Amount of funds available for thecompetition.

(5) Any other information relevant toa priority or other statutory or regulatoryrequirement applicable to the selectionof applications for new awards.

(b) In the case of unsolicitedapplications, the Secretary uses theprocedures in EDGAR (34 CFR 75.222(d)and (e)).(Authority: 20 U.S.C. 6022(i)(2)(D)(i))

§700.41 How are contract proposalsselected for award?

Following evaluation of the proposals,the contracting officer shall select foraward the offeror whose proposal ismost advantageous to the Governmentconsidering cost or price and the otherfactors included in the solicitation.(Authority: 20 U.S.C. 6011(i)(2)(D)(i))

[FR Doc. 95-22872 Filed 9-13-95; 8:45 am]BILLING CODE 4000-01-P

Appendix B

Part IIDepartment of Education34 CFR Part 701

Office of Educational Research andImprovement (OERI) Conduct andActivities Evaluation Standards;Designation of Exemplary andPromising Programs; Final Rule


25

MI1.IMP" M

411.

AMMil ..AIMIMAII ..... M.IMMMII!

MMUS...AIM'MIMM IM AMAM MI IM

IMII1M IMI MIM 1M AMIMA MI AMI

IMI

"AAM

IMI=111 11. I=NI In= I1W .1.11M 1=111.I. WI. II1 IM,.IIV.

MIIMbo.11111ML A IIfl

MMO

11.

MondayNovember 17, 1997

Part II


1M MIMN 2MI MI MbIM M....._..A=1MMIN = 34 CFR Part 701,-..=

Office of Educational Research and-- a"'...- ,. Improvement (OERI) Conduct and

Activities Evaluation Standards;.". 7M Mb. MIM Designation of Exemplary and Promising,AA1MIMII/ 1M ..1 Programs; Final Rule- - -- - ._- - --- - -. -t

AM110AAMm MEM111',MIMIN M

IMA. IM IMI.I. MI IMMI MI MtIMI MI AM

AMMI IML MI AIMAMAMI Mb. MI _IMP

M

2 6

61428 Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations


34 CFR Part 701

RIN 1850-AA52

Standards for Conduct and Evaluationof Activities Carried out by the Officeof Educational Research andimprovement (OERI)Designation ofExemplary and Promising Programs

AGENCY: Department of Education.ACTION: Final regulation.

SUMMARY: The Assistant Secretary forEducational Research and Improvement(OERI) is establishing final regulationspursuant to the "Educational Research,Development, Dissemination, andImprovement Act of 1994." Theregulations are intended to providequality assurance that programsdesignated by the Department ofEducation as either exemplary orpromising have met criteria that willallow educators, professionalorganizations, and others to use theseprograms with confidence.DATES: These regulations take effectDecember 17, 1997. However, affectedparties do not have to comply with theinformation collection requirement in§ 701.4 until the Department ofEducation publishes in the FederalRegister notification of the compliancedate and the control number assigned bythe Office of Management and Budget(OMB) to this information collectionrequirement. Publication of the controlnumber notifies the public that OMBhas approved this informationcollection requirement under thePaperwork Reduction Act of 1995.FOR FURTHER INFORMATION CONTACT:Stephen O'Brien, U.S. Department ofEducation, 555 New Jersey Avenue,NW, Room 502B, Washington, D.C.Telephone: (202) 219-2141. Internet:(Steve O'[email protected]). Individualswho use a telecommunication device forthe deaf (TDD) may call the FederalInformation Relay Service (FIRS) at 1-800-877-8339 between 8 a.m. and 8p.m., Eastern time, Monday throughFriday.

Individuals with disabilities mayobtain this document in an alternateformat (e.g., Braille, large print,audiotape, or computer diskette) onrequest to the person listed in thepreceding paragraph.SUPPLEMENTARY INFORMATION: On March31, 1994, President Clinton signedPublic Law 103-227, which includesTitle IX, the Educational Research,Development, Dissemination, andImprovement Act of 1994 (the Act). TheAct restructured OERI and provided it

with a broad mandate to conduct anarray of research, development,dissemination, and improvementactivities aimed at strengthening theeducation of all students.

The Act directed the AssistantSecretary to develop, in consultationwith the National Educational ResearchPolicy and Priorities Board (the Board),the highest standards of professionalexcellence necessary to govern theconduct and evaluation of all research,development, and disseminationactivities carried out by the OERI. Thelegislation requires that the standards bedeveloped in three phases.

In the first phase, standards werepromulgated to establish the peerreview process and evaluation criteria tobe used for reviewing applications forgrants and cooperative agreements andproposals for contracts. The Departmentpublished final regulations setting outthese standards on September 14, 1995(60 FR 47808). The regulations in thisannouncement address the secondphase of development by establishingthe criteria for panels to use inreviewing potentially exemplary andpromising educational programs. TheAssistant Secretary will later publishproposed regulations for phase three ofthe standards, which will govern howOERI evaluates performance of itsrecipients of grants, contracts, andcooperative agreements.

The OERI legislation requires thatexpert panels be established to revieweducational programs submitted byindividuals or organizations. Thelegislation also provides that theSecretary may identify educationalprograms for the panels to review. Thestatute requires the panels torecommend to the Secretary thoseprograms that should be designated asexemplary or promising anddisseminated through the Department'sNational Education DisseminationSystem. The law requires that eachpanel consist of appropriately qualifiedexperts and practitioners and requiresthe Secretary to develop standards thatdescribe the procedures the panels willuse in reviewing the educationalprograms. Section 941(a)(3) of the lawbroadly defines educational programs toinclude educational polices, researchfindings, practices, and products.Educational programs may range in sizeand complexity from an individualinstructional programsuch as anelementary school science programtoa comprehensive reform initiativeinvolving multiple goals andparticipants. Programs at all levels ofeducationpreschool, elementary,secondary, and postsecondaryareeligible for consideration.

2 7

In determining whether aneducational program should berecommended as exemplary orpromising, each panel is required by theAct to consider: (a) Whether, based onempirical data, the program is effectiveand should be designated as exemplaryor (b) whether there is sufficientevidence to demonstrate that theprogram shows promise for improvingstudent achievement and should bedesignated as promising. The Actexpressly states that a panel shall noteliminate a program from considerationbased on the lack of one type ofsupporting data such as test scores.

The evaluation process set forth in thefinal regulations will ensure thatprograms disseminated by theDepartment are high-quality, research-based programs that have providedevidence indicating they have improvedteaching, learning, or both, or hasdemonstrated other worthy educationalperformance outcomes. TheDepartment's dissemination system isdesigned to make information aboutthese promising and exemplaryprograms available to the public asquickly as possible. The system willenable the Department to respond to allforms of requests for information andassistance, and to support theapplications of research and bestpractice. The system will use electronicnetworking and the capabilities of:

National Research Institutes;Educational Resources Information

Center (ERIC);Regional Educational Laboratories;

Department-supported disseminationand technical assistance providers;

National Library of Education;Eisenhower Regional Consortia and

Clearinghouse, andOther public and private nonprofitentities, including educationassociations and networks.Until recently, the Department

validated exemplary programs throughits Program Effectiveness Panel (PEP)and disseminated them through theNational Diffusion Network (NDN).Since this program no longer exists,with the adoption of these standards theDepartment will evaluate anddisseminate promising educationalprograms in addition to exemplaryprograms. The Department will alsowork in partnership with constituencygroups who have expertise in thespecific topic areas represented by theexpert panels to develop coordinatedprocedures to maximize theirinvolvement in this work.

On June 3, 1996, the Secretarypublished a notice of proposedrulemaking (NPRM) for this part in the

Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations 61429

Federal Register (61 FR 27990-27993).These final regulations contain threemajor changes from the NPRM. Thesechanges are fully explained in the"Analysis of Comments and Changes"elsewhere in this preamble. The changespertain to the standing panel; thedistinction between "promising" and"exemplary"; and the factors listedunder the criteria expert panels will useto evaluate programs.

Analysis of Comments and ChangesIn response to the Secretary's

invitation in the NPRM, seven partiessubmitted comments on the proposedregulations. This included commentsfrom individual members of two pilotpanels (math/science and gender equity)that were appointed by the Secretary tofield test the expert panel process. Inaddition to the public comment,comments from the Board'sSubcommittee on Standards areaddressed as required by the legislation.The full Board approved the finalregulations at a meeting on September26, 1997. An analysis of the commentsand of the changes in the regulationssince publication of the NPRM follows.

Major issues are grouped according tosubject with appropriate sections of theregulations referenced in parentheses.Technical and other minor changesand suggested changes the Secretary isnot legally authorized to make under theapplicable statutory authorityare notaddressed.

Eligibility (§ 701.3)Comments: One commenter asked for

clarification on who is eligible to submiteducational programs for designation aspromising or exemplaty. Specifically,this commenter asked whetherfederally-funded entities, such as theRegional Laboratories, will be requiredto go through this process; whether localagencies that receive Federal fundingthrough states, such as under Title I ofthe Elementary and SecondaryEducation Act (ESEA), may submitprograms on their own; whethersponsors need to be invited to submit ormay submit on their own initiative; andwhether for-profit entities may submit.

Discussion: The law provides that"individuals" or "organizations" maysubmit educational programs for review.Since the law is silent on the specificnature of the organizations, theSecretary believes that for-profitagencies would be eligible to submitprograms for review. With respect to theOERI-funded Regional EducationalLaboratories, the law provides that theSecretary may identify those programsfor panel review. In addition, theSecretary believes that the Laboratories

could submit one or more of theirprograms on their own initiative. Thequestion of whether local agencies thatreceive Federal funding through a Stateor Federal entity, such as under Title Iof the ESEA, can submit on their ownor must go through their fundingagency, will be addressed inadministrative guidance.

Changes: None.

Content of Submissions (§ 701.4)Comments: Three commenters made

suggestions about this section. Twocommenters believed that requiringfunding and staffing information wasburdensome and not germane to thedesignation of a program as promisingor exemplary. One commenter believedthat this section should require programsponsors to submit specific materialsrelated to content and methods. Anothercommenter believed that this sectionshould include the requirement that theprogram include evidence ofsustainability of improvement withtargeted student populations.

Discussion: The Secretary believesthat funding and staffing informationshould be included to help determinewhether an educational program shouldbe recommended as either exemplary orpromising. The Secretary agrees thatsponsors should be required to submitinformation or materials specific tocontent and methods, as available andappropriate. The Secretary believes thatthe evidence of sustainability of studentimprovement should be evaluated bypeer reviewers in accordance with§ 701.22.

Changes: Section 701.4(b)(7) has beenrenumbered as §701.4(b)(8) and a new§701.4(b)(7) has been added to includea provision for specific materialsrelevant to content and methods.

Procedures for Submitting EducationalPrograms (New § 701.5)

Comments: One commenter believedthat the regulations should contain morespecificity about the procedure forsubmitting programs to the expertpanels. This commenter requestedspecifics on who receives thesubmissions and whether they may be

\submitted at any time or only onspecific dates.

Discussion: The Secretary agrees thatthe general submission proceduresshould be included in regulations. Asponsor seeking the exemplary orpromising designation for itseducational program may submit itsprogram at any time for consideration tothe Assistant Secretary, who will assignthe submitted program to theappropriate panel for review. Theindividual expert panels will set

28

appropriate timelines for programsubmissions. In addition, the AssistantSecretary will periodically establish andannounce in the Federal Registerspecific topic areas of high priority.Sponsors of educational programs inthese areas will be invited to submitthem for consideration.

Changes: A new § 701.5 has beenadded to include general procedures forsubmitting educational programs forreview by an expert panel.

Establishment of Panels (§ 701.10)Comments: The Board's

Subcommittee on Standardsrecommended a change to the expertpanel system. The Subcommitteethought that the structure of havingmembers of the expert panels drawnfrom a separate standing panel ofeducational experts was an unwieldy,overly-complicated structure. The Boardrecommended that the expert panels beformed separately from a standingpanel, which would instead provide anadministrative oversight and monitoringfunction for the expert panels.

Discussion: The Secretary agrees thatthe expert panel should be formedseparately from a standing panel. TheSecretary will determine the feasibilityof establishing a separate standing panelfor the oversight and monitoringfunctions referred to by the Boardfunctions which are administrative innature and could also be performed byOERI staff. Elimination of a reference toa standing panel in the regulationswould not alter the composition andfunction of the expert panels as outlinedin the NPRM.

Changes: Section 701.10(a) has beenremoved, § 701.10(b) has been revised,§ 701.11 has been removed, § 701.12(a)has been revised, and § 701.12 has alsobeen renumbered as § 701.11.

Panel Membership (§§ 701.11 and701.12)

Comments: One commenter observedthat §§ 701.11 and 701.12 in the NPRMdid not explicitly state that thoseserving on the panels would representboth the community of practice and thatof research. One commenter believedthat each panel should include one ormore members with evaluationexpertise in order to help evaluateevidence of effectiveness.

Discussion: The Secretary agrees withthese comments.

Changes: A sentence has been addedat the end of the renumbered § 701.11(a)(formerly § 701.12(a)) stating that themembership of the expert panels willrepresent both the community ofpractice and the community of research.Additionally, §§701.11(b)(3) and


701.11(b)(4) have been renumbered as§§701.11(b)(4) and 701.11(b)(5),respectively; and a new provision for§701.11(b)(3) has been added to includethe selection of at least one individualwith expertise in evaluating educationalprograms.

Difference Between Promising andExemplazy Programs (§ 701.21)

Comments: Five comments werereceived on the distinction betweenpromising and exemplary programs. Asproposed in § 701.21, the distinctionwas based upon the generalizability ofthe educational programs. Promisingprograms had to meet each of thecriteria of educational effectiveness in§ 701.22 (success, quality, educationalsignificance, and usefulness to others)with respect to only one "context orpopulation." Exemplary programs hadto meet each of the criteria "withrespect to multiple contexts or multiplepopulations."

Two commenters believed that thedistinction should stay the way it wasin the NPRM, although one of thosesuggested some clarifying language.However, three commenters questionedthe distinction on the basis that it wastoo narrowly and artificially drawn anddid not reflect the commonlyunderstood meaning of the words"promising" and "exemplary." In thisregard, one commenter believed thatpromising programs should not have tomeet every criterion in § 701.22 at thesame level as exemplary programs. Twocommenters believed that promisingprograms should have to meet thecriteria at the same level as exemplary,but that the evidence required ofpromising programs should be lessstringent and that exemplary programsshould be held to a higher standard ofevidence.

Discussion: The Secretary agrees withthose commenters who questioned theproposed distinction and advised OERIto give a more common sense meaningto the terms "promising" and"exemplary." The Secretary believesthat the distinction between promisingand exemplary programs specified inlegislation is sufficient to cover theseconcerns. The Secretary relies upon theexpert judgment of the expert panelreviewers in determining the nature andweight of evidence necessary todesignate a program as either promisingor exemplary, and in applying thecriteria listed in § 701.22 in making thisdetermination.

Changes: A revision has been made tothe distinction between "promising"and "exemplary."

Criteria (§ 701.22)

Comments: Five commentersprovided comments on this section andsuggested revisions to either thewording of the criteria or to the contentof the factors listed under each criterionor both. These comments includedcomments from one member of themath/science pilot panel and threemembers of the gender equity pilotpanel. Although the math/science panelmember did provide comments specificto the proposed criteria and factors, theconsensus of this panel was that theexpert panel process would be betterserved if each panel developed its ownfactors specific to the content ordiscipline or both under review by theindividual panel. One commentersuggested that the word "replicability"would better capture the concept for thecriterion entitled "usefulness to others."In addition, OERI's Board (TheSubcommittee on Standards) thoughtthat the regulations should be as simpleas possible and should give the expertpanels as much discretion as possible inevaluating programs submitted forreview.

Discussion: In addition to the math/science and gender equity panels, theSecretary will establish pilot panels intechnology and early reading in the nextyear. The Secretary has determined thatuntil the work of all four pilot panels isconcluded, the regulation should retainonly the four criteria outlined in theNPRM in § 701.22 and allow each panelthe flexibility to establish its ownindividual factors under each criterionthat are specific to its content ordiscipline. The fact that the commentsfrom the public suggested variouschanges to the factors underscores thedesirability of this approach. While thefinal regulations will therefore no longerrequire the expert panels to apply thefactors listed in the NPRM, the Secretaryencourages each panel to look at thesefactors as suggested examples. TheSecretary will review the factorsdeveloped by all of the panels to see ifthe criteria set forth in the finalregulations need to be modified.

Changes: The factors specified undereach of the four criteria have beeneliminated and the criterion,"usefulness to others" has been changedto "replicability."Assessment of Educational Impact

In the NPRM the Secretary requestedcomments on whether the proposedregulations would require transmissionof information that is being gathered byor is available from any other agency orauthority of the United States.

29

Based on the response to the NPRMand on its own review, the Departmenthas determined that the regulations inthis document do not requiretransmission of information that is beinggathered by or is available from anyother agency or authority of the UnitedStates.

Electronic Access to This DocumentAnyone may view this document, as

well as all other Department ofEducation documents published in theFederal Register, in text or portabledocument format (pdf) on the WorldWide Web at either of the followingsites:http://ocfo.ed.gov/fedreg.htmhttp://www.ed.gov/news.html

To use the pdf you must have theAdobe Acrobat Reader Program withSearch, which is available free at eitherof the previous sites. If you havequestions about using the pdf, call theU.S. Government Printing Office tollfree at 1-888-293-6498.

Anyone may also view thesedocuments in text copy only on anelectronic bulletin board of theDepartment. Telephone: (202) 219-1511or, toll free, 1-800-222-4922. Thedocuments are located under OptionGFiles/Announcements, Bulletins andPress Releases.

Note: The official version of this documentis the document published in the FederalRegister.

List of Subjects in 34 CFR Part 701Education, Educational research,

Reporting and recordkeepingrequirements.(Catalog of Federal Domestic AssistanceNumber does not apply)

Dated: November 11, 1997.Ricky T. Takai,Acting Assistant Secretary for EducationalResearch and Improvement.

The Secretary amends chapter WI oftitle 34 of the Code of FederalRegulations by adding a new part 701 toread as follows:

PART 701STANDARDS FORCONDUCT AND EVALUATION OFACTIVITIES CARRIED OUT BY THEOFFICE OF EDUCATIONALRESEARCH AND IMPROVEMENTPERODESIGNATION OFEXEMPLARY AND PROMISINGPROGRAMS

Subpart AGeneralSec.701.1 What is the purpose of these

standards?701.2 What definitions apply?701.3 Who is eligible to submit an

educational program for review?

Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations 61431

701.4 What must a program sponsor submitfor review?

701.5 What are the procedures forsubmitting an educational program forreview by an expert panel?

Subpart BSelection of Panel Members701.10 How are panels established?701.11 How is the membership of expert

panels determined?

Subpart CThe Expert Panel ReviewProcess

701.20 How does an expert panel evaluateprograms?

701.21 What is the difference between anexemplary and a promising program?

701.22 What criteria are used to evaluateprograms for exemplary or promisingdesignation?

Authority: 20 U.S.C. 6011(i), unlessotherwise noted.

Subpart AGeneral

§701.1 What is the purpose of thesestandards?

(a) The standards in this partimplement section 941(d) of theEducational Research, Development,Dissemination, and Improvement Act of1994.

(b) These standards are intended toprovide quality assurance thateducational programs designated by theU.S. Department of Education as eitherexemplary or promising have metcriteria that will allow educators,professional organizations, and others touse these programs with confidence.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

§701.2 What definitions apply?The following definitions apply to

this part:Assistant Secretary means the

Assistant Secretary for the Office ofEducational Research and Improvement.

Educational programs meaneducational policies, research findings,practices, and products.

Program sponsor means a partysubmitting an educational program fordesignation by the Secretary as eitherpromising or exemplary.

Secretary means the Secretary of theDepartment of Education or an officialor employee of the Department actingfor the Secretary under a delegation ofauthority.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

§701.3 Who is eligible to submit aneducational program for review?

Any public or private agency,organization or institution, or anindividual may submit an educationalprogram for review.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

§701.4 What must a program sponsorsubmit for review?

(a) To have an educational programconsidered for designation as exemplaryor promising, a sponsor must submit tothe Secretary a description of theprogram, program materials, and adiscussion of the program that isresponsive to the criteria in § 701.22.

(b) Information submitted mustinclude, to the extent relevant to theparticular program,

(1) A program abstract of 250 wordsor less;

(2) A description of the salientfeatures of the program;

(3) A description of the program'sphilosophy and history;

(4) Site information, includingdemographics;

(5) A description of evaluation results;(6) Funding and staffing information;(7) Specific materials relevant to

content and methods, as appropriate;and

(8) Organization name, address,telephone and fax numbers, e-mailaddress (if available), and contactperson.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

§701.5 What are the procedures forsubmitting an educational program forreview by an expert panel?

(a) An applicant seeking theexemplary or promising designation forits educational program may submit itsprogram at any time for consideration tothe Assistant Secretary, who will assignthe submitted program to theappropriate expert panel for review.

(b) The Assistant Secretary willperiodically establish and announce inthe Federal Register specific topic areasof high priority. Sponsors of educationalprograms in these areas will be invitedto submit their programs forconsideration.

(c) The individual expert panels willset appropriate timelines for reviewingprogram submissions.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

Subpart BSelection of PanelMembers

§ 701.10 How are panels established?The Assistant Secretary selects

individuals, based on their areas ofexpertise, to serve on expert panels inspecific topic areas for the purpose ofreviewing and evaluating educationalprograms and recommending, to theSecretary, those programs that should bedesignated as exemplary or promising.

30


§ 701.11 How Is the membership of expertpanels determined?

(a) For the review of each program orgroup of programs, the AssistantSecretary establishes an expert panel.The membership of the expert panelswill represent both the community ofpractice and the community of research.

(b) In establishing the membership ofeach expert panel, the AssistantSecretary

(1) Selects individuals who have in-depth knowledge of the subject area orcontent of the program or group ofprograms to be evaluated;

(2) Selects at least one current teacher,principal, or other school-based orcommunity-based professional;

(3) Selects at least one individual withexpertise in evaluating educationalprograms;

(4) Ensures that no more than one-third of the panel members areemployees of the Federal Government;and

(5) Ensures that each panel memberdoes not have a conflict of interest, asdetermined in accordance withparagraph (c) of this section, withrespect to any educational program thepanel member is asked to review.

(c) Panel members are consideredemployees of the U.S. Department forthe purposes of conflicts of interestanalysis and are subject to theprovisions of 18 U.S.C. 208, 5 CFR2635.502, and the Department's policiesused to implement those provisions.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

Subpart CThe Expert Panel ReviewProcess

§701.20 How does an expert panelevaluate programs?

(a) Each panel member shall(1) Independently review each

program based on the criteria in§ 701.22;

(2) Provide written comments basedon an analysis of the strengths andweaknesses of the program according tothe criteria;

(3) Participate in site visits or otherverification activities, if appropriate;and

(4) Participate in a meeting of theexpert panel, if appropriate, to discussthe reviews.

(b) A panel may not eliminate aneducational program from considerationbased solely on the fact that the programdoes not have one specific type ofsupporting data, such as test scores.

(c) Each expert panel shall make arecommendation to the Secretary as to


whether the program is exemplary,promising, or neither.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

§701.21 What is the difference between anexemplary and a promising program?

(a) In determining whether aneducational program should berecommended as exemplary orpromising, the panel shall consider

(1) Whether, based on empirical data,the program is effective and should bedesignated as exemplary; or

(2) Whether there is sufficientevidence to demonstrate that theprogram shows promise for improving

student achievement and should bedesignated as promising.

(b) The Secretary relies upon thejudgment and expertise of peerreviewers, as established in § 701.11, todetermine the nature and extent ofevidence required to distinguishbetween promising and exemplaryprograms and to apply the four criteriaestablished in § 701.22, and their ownindividual factors under each criterionin making this determination.(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and(E), 6041(d))

3 1

§701.22 What criteria are used to evaluateprograms for exemplary or promisingdesignation?

The Secretary establishes thefollowing evaluation criteria for expertpanels to use in determining whether aneducational program should berecommended as exemplary, promising,or neither:

(a) Evidence of success.(b) Quality of the program.(c) Educational significance.(d) Replicability.


(FR Doc. 97-30051 Filed 11-14-97; 8:45 am]BILLING CODE 4000-01-P

Appendix C

Part VIIDepartment of Education34 CFR Part 702

Standards for Conduct and Evaluationof Activities Carried Out by the Officeof Educational Research andImprovement (0ER1)Evaluation ofthe Performance of Recipients ofGrants, Cooperative Agreements, andContracts; Final Rule

Attaining Excellence25

32

,M1

..11111111PrIl

EA

.1111,

TuesdayOctober 27, 1998

Part VII


MM34 CFR Part 702I11MINEII". Standards for Conduct and Evaluation ofActivities Carried Out by the Office ofEducational Research and Improvement(0ER1)Evaluation of the Performance of

IM/ Recipients of Grants, Cooperative- Agreements, and Contracts; Final Rule

10

IMPIN MY

OW. INIIMI NSMa 1=1NM =1I11,--

ma

3 3

57570 Federal Register / Vol. 63, No. 207 / Tuesday, October 27, 1998 / Rules and Regulations


34 CFR Part 702

RIN 1850AA54

Standards for Conduct and Evaluationof Activities Carried Out by the Officeof Educational Research andImprovement (0ERI)Evaluation ofthe Performance of Recipients ofGrants, Cooperative Agreements, andContracts

AGENCY: Office of Educational Researchand Improvement, Department ofEducation.ACTION: Final regulations.

SUMMARY: The Assistant Secretaryestablishes regulations pursuant toOERI's authorizing legislation, theEducational Research, Development,Dissemination, and Improvement Act of1994. The major purpose of thesestandards is to ensure that the research,development, and disseminationactivities carried out by the recipients ofgrants from and contracts andcooperative agreements with OERI meetthe highest standards of professionalexcellence.EFFECTIVE DATE: These regulations takeeffect November 27, 1998.FOR FURTHER INFORMATION CONTACT:Sharon Bobbitt, U.S. Department ofEducation, 555 New Jersey Avenue,NW, Room 508C, Washington, D.C.Telephone: (202) 219-2126. Internet:([email protected]). Individualswho use a telecommunications devicefor the deaf (TDD) may call the FederalInformation Relay Service (FIRS) at 1800-877-8339 between 8 a.m. and 8p.m., Eastern time, Monday throughFriday.

Individuals with disabilities mayobtain this document in an alternateformat (e.g., Braille, large print,audiotape, or computer diskette) onrequest to the contact person listed inthe preceding paragraph.SUPPLEMENTARY INFORMATION:

BackgroundOn March 31, 1994, President Clinton

signed Pub. L. 103-227, which includesTitle IX, the Educational Research,Development, Dissemination, andImprovement Act of 1994 (the Act). TheAct restructured OERI and provided itwith a broad mandate to conduct anarray of research, development,dissemination, and improvementactivities aimed at strengthening theeducation of all students.

Statutory RequirementsThe Act directed the Assistant

Secretary to develop, in consultation

with the National Educational ResearchPolicy and Priorities Board (the Board),such standards as may be necessary togovern the conduct and evaluation of allresearch, development, anddissemination activities carried out byOERI to ensure that these activities meetthe highest standards of professionalexcellence. The Board is responsible forreviewing and approving the standards.The legislation requires that thestandards be developed in three phases.

In the first phase, standards werecreated and promulgated to establish thepeer review process and evaluationcriteria to be used for the review ofapplications for grants and cooperativeagreements and proposals for contracts.The final regulations setting out thesestandards were published on September14, 1995 (60 FR 47808). In the secondphase, standards were created andpromulgated to establish the criteria tobe used in reviewing potentiallyexemplary and promising educationalprograms. The final regulations settingout these standards were published onNovember 17, 1997 (62 FR 61427).

In the third phase, which is thesubject of these final regulations, theAct requires that OERI developstandards for evaluating and assessingthe performance of all recipients ofgrants from and cooperative agreementsand contracts with OERI. Thisevaluation must take place both duringand at the conclusion of theperformance of the grant, cooperativeagreement, or contract, and mustinclude the use of a system of peerreview for the final assessment.

In developing the standards, theAssistant Secretary was required toreview the procedures utilized by theNational Institutes of Health (NIH), theNational Science Foundation (NSF), andother Federal departments or agenciesengaged in research and developmentand to solicit recommendations fromresearch organizations and members ofthe general public. OERI has reviewedthe procedures used to evaluate theperformance of recipients of grants,contracts, or cooperative agreements byseveral offices within NIH and NSF, theOffice of Energy Research in theDepartment of Energy, the Food andDrug Administration, the NationalInstitute of Standards and Technology,the National Aeronautics and SpaceAdministration, and the UniversityResearch Initiative of the Department ofDefense. Recommendations concerningthese standards have been obtainedfrom the American EducationalResearch Association, the Council forEducational Development and Research,and the Organization of ResearchCenters.

3 4

StandardsThe standards have been developed

by the Assistant Secretary inconsultation with the Board. Thesestandards cover all grants, cooperativeagreements, and contracts administeredby OERI, ranging from the smallestpurchase orders and commissionedpapers to the largest research projectsand research centers. The standards:

Require at least one interimassessment as well as a final assessmentof the performance of recipients ofgrants, cooperative agreements, andcontracts.

Establish procedures for selectingpeer review panels to conduct theassessments.

Establish procedures and criteriathat the peer review panels use inconducting the assessments.

Establish specific additional criteriathat peer review panels use inconducting the assessments for NationalResearch and Development Centers,Regional Educational Laboratories,Field-Initiated Studies, and ERICClearinghouses.

In an effort to fulfill the law'sintention of ensuring high-qualityresearch, development, and evaluation,OERI has developed standards in whichinterim and final assessments may besupplemented by a self-assessment bythe recipient of a grant, cooperative,agreement, or contract. The Board andthe Assistant Secretary believe that thecollection and review of evidence onone's own performance is itself a usefultool for improvement.

The Government Performance andResults Act requires the establishmentof performance indicators forDepartment activities. Informationcollected pursuant to those indicatorswill be considered, as appropriate, inthe evaluation of individual recipients.

On February 24, 1998, the AssistantSecretary published a notice ofproposed rulemaking (NPRM) for thesestandards in the Federal Register (63 FR9393). These final regulations containfour major changes from the NPRM.These changes are fully explained in the"Analysis of Comments and Changes"elsewhere in this preamble. The majorchanges pertain to clarification of thepurpose of the regulation, how OERIdetermines the number of interimassessments necessary, the role ofDepartment of Education staff in theassessments, and the use of interimassessments as a source of informationfor the final assessment.

Analysis of Comments and ChangesIn response to the Secretary's

invitation in the NPRM, four parties

Federal Register / Vol. 63, No. 207 / Tuesday, October 27, 1998 / Rules and Regulations 57571

submitted comments on the proposedregulations. In addition to the publiccomment, comments from the Board'sSubcommittee on Standards areaddressed as required by the legislation.The full Board approved the finalregulations at a meeting on September18, 1998. An analysis of the commentsand of the changes in the regulationssince publication of the NPRM follows.

Major issues are grouped according tosubject with appropriate sections of theregulations referenced in parentheses.Technical and other minor changesand suggested changes the Secretary isnot legally authorized to make under theapplicable statutory authorityare notaddressed.

Purpose (5 702.1)Comments: Three commenters

suggested that the purpose of thestandards be clarified. One commentersuggested that the standards themselvescannot ensure the highest standards ofprofessional excellence. Anothercommenter asked specifically whetherthe purpose for conducting assessmentswas to make decisions about futurefunding or to provide a system formonitoring and enhancing current andfuture projects.

Discussion: The Secretary agrees thatthe purpose of the standards should beclarified to go beyond their statedstatutory purpose, which is to "ensurethe highest standards of professionalexcellence," to include the objectives ofcontinuously improving the quality offunded activities and of considering theresults as one of the factors indetermining continuation funding formulti-year awards.

Changes: Section 702.1 has beenmodified to include a provision that thepurpose of the standards is to providefeedback to help improve the quality offunded activities and to provideinformation for consideration ascontinuation funding decisions aremade.

Additional Activities that May beEvaluated (§ 702.3)

Comment: One commenter thoughtthat the statement that these standardscould be applied to other activitiesfunded by the Department was toobroad and should be deleted.

Discussion: The Secretary believesthat this statement is necessarily broadto allow all Department programs to usethese standards, when appropriate, toassess the performance of any of theirfunded activities without developingtheir own unique regulations. Thisstatement is also consistent with theearlier standards which established thepeer review process and evaluation

criteria to be used for the review ofapplications for grants and cooperativeagreements and proposals for contracts.

Changes: None.

Number of Interim Assessments(§ 702.4)

Comments: Two commenterssuggested changes to this provision. Onecommenter suggested since there maybe more than one interim assessment,that it be clear in § 702.4(d)(1). TheOERI Board suggested that therequirements for a single interimassessment for total awards of$5,000,000 or less be modified to reflecttotal awards of $3,000,000 or less.

Discussion: In response to thecomments, the Secretary now believesthat considerations such as difficulty inachieving project objectives rather thanthe dollar levels of awards shoulddetermine whether a particular projectmerits more than one interimassessment. Elimination of the dollarthreshold clarifies the original intent ofthis section which is to require that allawards receive one interim assessment.More than one interim assessment willbe performed only when a recipient ishaving difficulty achieving projectobjectives as determined by the initialinterim assessment or through themonitoring efforts of Department ofEducation staff. The Assistant Secretarywill make the determination of thenumber of interim assessments on acase-by-case basis.

Changes: Section 702.4(b) has beenmodified to delete the dollar thresholdand to reflect that all awards willreceive at least one interim assessment.A new paragraph 702.4(c) has beenadded to clarify that the AssistantSecretary will require more than oneinterim assessment when a recipient hasbeen identified, either in the initialinterim review or through monitoringefforts of Department of Education staff,as having difficulty in achieving projectobjectives. Former paragraph 702.4(c)has been redesignated as § 702.4(d).Section 702.4(d)(1) has been modified todefine an interim assessment as "anyassessment" conducted during arecipient's period of performance.

Definitions (5 702.5)

Comment: One commenter suggestedthat the terms referred to in this sectioninclude the specific definitions and notreferences to the OERI statute and to theEducation Department GeneralAdministrative Regulations.

Discussion: The Secretary believesthat providing the citations for specificterms rather than the definitionsthemselves keeps regulations short andconcise while still cross referencing

3 5

easily accessible resources for thedefinitions.

Changes: None.

Characteristics of Peer Reviewers(§ 702.10)

Comment: One commenter suggestedthat paragraph 702.10(a) "(4) knowledgeof a broad range of education policiesand practices;" be deleted from the listof knowledge and expertise required ofpeer reviewers, because it is redundantwith the other criteria and is very vague.

Discussion: The Secretary believesthat this criterion provides for a balancebetween specific program knowledgeand a broader perspective of educationpolicies and practices and is thereforenot redundant with the other, morefocused, characteristics required of peerreviewers.

Changes: None.

Role of Department Staff (5 702.10)Comments: Two commenters

expressed concern over the appropriaterole of the OERI staff in the reviewprocess. One commenter urged theDepartment to use all outside reviewers.The other commenter acknowledged theknowledge and skills of the OERI staffbut suggested that staff not serve as peerreviewers within the primary division ofan agency in which they work and thateach peer review panel be limited to oneDepartment staff person. Thiscommenter suggested that the staff focuson the important role of mentoring anddesigning competitions.

Discussion: The Secretary agrees thatthe primary role of the OERI staff shouldbe management of competitionsincluding assessing the results of peerreviews and monitoring awards. TheSecretary believes that the purpose ofthe peer review process should be toacquire the perspective of outsideexperts independent of OERI. TheSecretary also believes that there may beexceptional circumstances whereexpertise resides in OERI or in theDepartment, or where outside reviewersare not required such as in the reviewof small purchase orders. Theexceptions should be determined by theAssistant Secretary.

Changes: Section 702.10(d) has beenreworded to preclude OERI and otherDepartment staff from serving as peerreviewers except in exceptionalcircumstances as determined by theAssistant Secretary.

Conflict of Interest (§ 702.11)

Comment: One commenter wasconcerned that while the conflict ofinterest requirements were "legallycorrect" they failed to address theproblem occasioned by reviewers who

57572 Federal Register / Vol. 63, No. 207 / Tuesday, October 27, 1998 / Rules and Regulations

may have ideological or methodologicalview points that differ from those of therecipient to be evaluated, or who areaffiliated with competing institutionalorganizations.

Discussion: The commenter appearsto be concerned that the proposedconflict of interest provision does notaddress the potential problem of bias onthe part of a panel against a particulargrantee on ideological or other grounds.The Secretary first believes that it isessential to retain the present language,which parallels the provision in thestandards at 34 CFR 701.11(c), becauseit highlights the important issue ofimproper financial gain or theappearance of improper gain. However,the Secretary agrees that adding arequirement to the effect that panelsselected by the Assistant Secretaryreflect a broad range of perspectivescould strengthen the regulation.

Changes: A new paragraph "(c)" hasbeen added to § 702.13 requiring theAssistant Secretary, to the greatestextent feasible, to select peer reviewersfor each evaluation who represent abroad range of perspectives.

Sources of Information (§§ 702.22 and702.23)

Comment: One commenter suggestedthat the use of Govermnent Performanceand Results Act (GPRA) informationshould be encouraged rather thanrequired for both interim and finalassessments. The commenter isconcerned that information currentlybeing collected under GPRA to evaluatethe effectiveness of a program or asystem-level activity will not provideinformation relevant to the assessmentof individual awards under thatprogram or system-level activity andtherefore should not be required.

Discussion: The Secretary agrees thatinformation obtained by GPRA-relatedreports on the effectiveness of a programor system level activity, e.g., howeffectively a program is meeting theoverall objectives defined for it in itsauthorizing legislation, may notnecessarily include information relatedto an individual award being reviewedunder this regulation. However, theSecretary believes that information onthe effectiveness of the particularprogram under which a recipientreceives funding will help to provide acontext for the review of an individualaward and must be considered by thepanel. Moreover, these regulations makeit clear that the GPRA information is

only one of a number of sources used inconducting the review.

Changes: None.Comment: One commenter suggested

that the findings and information frominterim assessments would be animportant source of information for thefinal assessments and should beincluded under § 702.23(a).

Discussion: The Secretary agrees thatthe results of interim assessmentsshould be a source of information forfinal assessments.

Change: Section 702.23(a) has beenmodified to add a new paragraph(§702.23(a)(5)) to require that the resultsof interim assessments be considered asa source of information for finalassessments.

Evaluation Criteria (§ 702.24)

Comments: Two commenterssuggested changes to this section. Onecommenter suggested that there be asingle menu of criteria for the standards,because the proposed menu is too long.The second commenter suggested thatsince Field Initiated Studies are notlikely to provide services, the word"services" be deleted from the criterionin §702.24(e)(4)(ii): "* * * addressesissues of national significance throughits products or services, or both."

Discussion: The Secretary believes thecurrent menu approach provides acomprehensive strategy for assessing theperformance of all activities, rangingfrom the smallest purchase order to thelargest research investments. Thecategories in the regulation reflect thespecific authorities in the OERI statute.In addition, the menu provides for othercriteria for future research investmentsthat do not fit within the statutoryauthorities yet also must be assessed. Asingle menu would, of necessity, be toogeneric to apply to the wide range ofactivities covered by these standards.The Secretary agrees that assessing"services" is not appropriate for FieldInitiated Studies projects.

Change: Section 702.24(e)(4)(ii) hasbeen modified to delete the word,"services."

Paperwork Reduction Act of 1995Under the Paperwork Reduction Act

of 1995, no persons are required torespond to a collection of informationunless it displays a valid OMB controlnumber. The valid OMB control numberassigned to the collection of informationin these final regulations is displayed atthe end of the affected sections of theregulations.

3 6

Assessment of Educational Impact

In the NPRM the Secretary requestedcomments on whether the proposedregulations would require transmissionof information that is being gathered byor is available from any other agency orauthority of the United States.

Based on the response to the NPRMand on its own review, the Departmenthas determined that the regulations inthis document do not requiretransmission of information that is beinggathered by or is available from anyother agency or authority of the UnitedStates.

Electronic Access to This Document

Anyone may view this document, aswell as all other Department ofEducation documents published in theFederal Register, in text or portabledocument format (pdf) on the WorldWide Web at either of the followingsites:

http://ocfo.ed.gov/fedreg.htmhttp://www.ed.gov/news.html

To use the pdf you must have theAdobe Acrobat Reader Program withSearch, which is available free at eitherof the previous sites. If you havequestions about using the pdf, call theU.S. Government Printing Office at (202)512-1530 or, toll free, at 1-888-2936498.

Anyone may also view thesedocuments in text copy only on anelectronic bulletin board of theDepartment. Telephone: (202) 219-1511or, toll free, 1-800-222-4922. Thedocuments are located under OptionGFiles/Announcements, Bulletins andPress Releases.

Note: The official version of this documentis the document published in the FederalRegister.

List of Subjects in 34 CFR Part 702

Education, Educational research,Reporting and recordkeepingrequirements.(Catalog of Federal Domestic AssistanceNumber does not apply.)

Dated: October 22, 1998.C. Kent McGuire,Assistant Secretazy for Educational Researchand Improvement.

The Secretary amends Chapter VII ofTitle 34 of the Code of FederalRegulations by adding a new Part 702 toread as follows:


PART 702STANDARDS FORCONDUCT AND EVALUATION OFACTIVITIES CARRIED OUT BY THEOFFICE OF EDUCATIONALRESEARCH AND IMPROVEMENT(0EROEVALUATION OF THEPERFORMANCE OF RECIPIENTS OFGRANTS, COOPERATIVEAGREEMENTS, AND CONTRACTS

Subpart AGeneralSec.702.1 What is the purpose of these

standards?702.2 What activities must be evaluated by

these standards?702.3 What additional activities may be

evaluated by these standards?702.4 When is performance assessed under

these standards?702.5 What definitions apply?

Subpart BSelection of Peer ReviewPanels

702.10 What are the characteristics of peerreviewers?

702.11 What constitutes a conflict ofinterest for grants and cooperativeagreements?

702.12 What constitutes a conflict ofinterest for contracts?

702.13 How are peer reviewers selected forpanels?

Subpart CThe Evaluation Process702.21 How does a peer review panel

evaluate the performance of a recipient?702.22 What information does a peer

review panel consider for an interimassessment?

702.23 What information does a peerreview panel consider for a finalassessment?

702.24 What evaluation criteria must beused for performance assessments?

Authority: 20 U.S.C. 6011(i), unlessotherwise noted.

Subpart AGeneral§ 702.1 What Is the purpose of thesestandards?

(a) The standards in this partimplement section 912(i) of theEducational Research, Development,Dissemination, and Improvement Act of1994 (the Act).

(b) These standards establish criteriaand a peer review process to providerecipients of OERI grants, cooperativeagreements and contract awards withassessments of their projects.

(1) The purpose of the assessments isto provide feedback to recipients toimprove the quality of funded activitiesand to provide information to OERI asit determines if a recipient of a multi-year award merits continuation funding.

(2) The criteria and peer reviewprocess are intended to address thestatutory requirement that the research,development, and disseminationactivities carried out by the recipients of

grants from and contracts andcooperative agreements with the Officeof Educational Research andImprovement (OERI) meet the higheststandards of professional excellence,(Authority: 20 U.S.C. 6011(i)(2)(F))

§702.2 What activities must be evaluatedby these standards?

These standards apply to activitiescarried out by OERI using fundsappropriated under section 912(m) ofthe Act including activities carried outby the following entities or programs:

(a) The National Education ResearchInstitutes.

(b) The Office of Reform Assistanceand Dissemination.

(c) The Educational ResourcesInformation Center.

(d) The Regional EducationalLaboratories.

(e) The Teacher ResearchDissemination Demonstration Program.

(f) The Goals 2000 CommunityPartnerships Program.

(g) The National Educational ResearchPolicy and Priorities Board.(Authority: 20 U.S.C. 6011(i)(1))

§ 702.3 What additional activities may beevaluated by these standards?

The Secretary may apply thesestandards to other activities funded bythe Department, as appropriate.(Authority: 20 U.S. C 6011 (i)(1))

§ 702.4 When Is performance assessedunder these standards?

(a) The Secretary will assess theperformance of recipients of OERIgrants, contracts, and cooperativeagreements subject to these standardsduring and at the conclusion of theirperiod of performance.

(b) The Department requires at leastone interim assessment by a peer reviewpanel for all awards.

(c) The Assistant Secretary willapprove and require more than oneinterim assessment when an award isidentified, either by the initial interimreview or by Department of Educationstaff monitoring the award, as havingdifficulty in achieving projectobjectives.

(d) A final assessment by a peerreview panel is required for all awards.

(e) As used in this part(1) Interim assessment is any

assessment conducted during arecipient's period of performance.

(2) Final assessment is one conductedat the conclusion of a recipient's periodof performance.(Authority: 20 U.S.C. 6011(i)(2)(F))

§ 702.5 What definitions apply?(a) Definitions in the Educational

Research, Development, Dissemination,

3 7

and Improvement Act of 1994. Thefollowing terms used in this part aredefined in 20 U.S.C. 6011(1)(1):DevelopmentDisseminationEducational Research

(b) Definitions in the EducationDepartment General AdministrativeRegulations. The following terms usedin this part are defined in 34 CFR 77.1:ApplicationAwardDepartmentGrantProjectSecretary

(c) Definitions in the FederalAcquisition Regulation. The followingterm used in this part is defined in 48CFR Chapter 1: Contract Proposal.(Authority: 20 U.S.C. 6011(i)(2)(F)

Subpart BSelection of Peer ReviewPanels

§ 702.10 What are the characteristics ofpeer reviewers?

(a) The Assistant Secretary selectseach peer reviewer. Each peer reviewermust have the necessary knowledge andexpertise in the area of the project beingreviewed to evaluate the performance ofa recipient. This experience mayinclude

(1) Expert knowledge of subject matterin the area of the activities to bereviewed;

(2) Expert knowledge of theory ormethods or both in the area of theactivities to be reviewed;

(3) Practical experience in the area ofthe activities or type of institution orboth to be reviewed;

(4) Knowledge of a broad range ofeducation policies and practices;

(5) Experience in managing complexorganizations; or

(6) Expertise and experience inevaluation theory and practice.

(b) Each peer reviewer must be free ofconflict of interest, as determined inaccordance with § 702.11 or § 702.12.

(c) The Assistant Secretary may solicitnominations for peer reviewers fromprofessional associations, nationallyrecognized experts, and other sources.

(d) OERI and other Department staffwho possess the qualifications inparagraphs (a) and (b) of this sectionmay serve as peer reviewers only inexceptional circumstances asdetermined by the Assistant Secretary.(Authority: 20 U.S.C. 6011(i)(2)(B))

§702.11 What constitutes a conflict ofinterest for grants and cooperativeagreements?

A peer reviewer assessing theperformance of the recipient of a grant

57574 Federal Register / Vol. 63, No. 207/Tuesday, October 27, 1998/Rules and Regulations

from or cooperative agreement withOERI is considered an employee of theDepartment for the purposes of conflictof interest analysis. As an employee ofthe Department, the peer reviewer issubject to the provisions of 18 U.S.C.208, 5 CFR 2635.502, and theDepartment's policies used toimplement those provisions.(Authority: 20 U.S.C. 6011(i)(2)(B))

§702.12 What constitutes a conflict ofInterest for contracts?

A peer reviewer assessing theperformance of the recipient of acontract with OERI is considered anemployee of the Department inaccordance with the Federal AcquisitionRegulation (FAR), 48 CFR 3.104-4(h)(2).As an employee of the Department, thepeer reviewer is subject to theprovisions of the FAR, 48 CFR Part 3,Improper Business Practices andPersonal Conflict of Interest.(Authority: 41 U.S.C. 423)

§702.13 How are peer reviewers selectedfor panels?

(a) The Assistant Secretary assignspeer reviewers to panels that conductthe performance assessments.

(b) The Assistant Secretary mayestablish panels by category of recipient,such as a panel to review theperformance of all Regional EducationalLaboratories. Each recipient is evaluatedindividually by reviewers who havebeen assigned to this type of panel.

(c) In establishing panels, theAssistant Secretary, to the greatestextent feasible, selects peer reviewersfor each evaluation who represent abroad range of perspectives.(Authority: 20 U.S.C. 6011(i)(2)(B))

Subpart CThe Evaluation Process

§702.21 How does a peer review panelevaluate the performance of a recipient?

(a) In each evaluation, a peer reviewpanel

(1) Considers relevant informationabout the recipient's performance, asdescribed in §§ 702.22 and 702.23; and

(2) Makes judgments about therecipient's performance, using thecriteria in § 702.24.

(b) Each peer reviewer prepares areport based on the reviewer'sassessment of the quality of the projectaccording to the evaluation criteria.

(c) After each peer reviewer hasevaluated each project independently,the panel may be convened to discussthe strengths and weaknesses of theproject. Each reviewer may thenindependently re-evaluate each projectwith appropriate changes made to thewritten report.

(d) The report of the interimassessment must include anyrecommendations the peer reviewermay have for improving the recipient'sperformance.

(e) The report of the final assessmentmust contain each peer reviewer'sevaluative summary of the recipient'sperformance, from the beginning of thecontract, grant, or cooperativeagreement to its conclusion.(Authority: 20 U.S.C. 6011(i)(2)(F))

§702.22 What information does a peerreview panel consider for an interimassessment?

(a) Sources of information for theinterim assessment must include

(1) The original request for proposalsor grant announcement and the contractproposal or grant application;

(2) Documentation of any changes inthe work described in the contract,grant, or cooperative agreement,including reasons for the changes;

(3) Any progress reports delivered tothe Department or made available to thepublic by the recipient;

(4) Examples of products delivered tothe Department or made available to thepublic by the recipient;

(5) Any relevant reports written byOERI staff, including reports of sitevisits by OERI staff;

(6) Any performance evaluationsconducted under the FAR or theEducation Department GeneralAdministrative Regulations (34 CFR Part75).

(7) Any relevant information providedby the recipient in response toGovernment Performance and ResultsAct (GPRA) (Pub. L. 103-62)requirements; and

(8) Any reports from programevaluations commissioned by theDepartment.

(b) Sources of information for theinterim assessment may also include

(1) A self-assessment, prepared by therecipient, addressing the criteria in§ 702.24;

(2) One or more site visits by the peerreview panel;

(3) One or more oral or writtenpresentations to the panel by therecipient describing its performance; or

(4) Other information about therecipient's performance.(Approved by the Office of Management andBudget under control number 1850-0746)(Authority: 20 U.S.C. 6011(i)(2)(F))

§702.23 What Information does a peerreview panel consider for a finalassessment?

(a) Sources of information for the finalassessment must include

(1) The original request for proposalsor application notice and the contrac

proposal or grant application, togetherwith documentation of any changes inthe work described in the proposal orapplication, including reasons for thechanges;

(2) If consistent with the recipient'scontract, grant, or cooperativeagreement with OERI, a written reportor oral presentation or both by therecipient summarizing its activities andaccomplishments;

(3) Any relevant information providedby the recipient in response toGovernment Performance and ResultsAct (GPRA) (Pub. L. 103-62)requirements;

(4) Any reports from programevaluations commissioned by theDepartment; and,

(5) Any relevant information providedby the interim assessment.

(b) The final assessment may alsoinclude other sources of information,such as one or more of those listed in§ 702.22.(Approved by the Office of Management andBudget under control number 1850-0746)(Authority: 20 U.S.C. 6011(i)(2)(F))

§ 702.24 What evaluation criteria must beused for performance assessments?

(a) Peer reviewers (and thoserecipients who conduct self-evaluations)shall use the criteria in paragraph (b) ofthis section to assess performance and,in case of interim assessments, toidentify areas in which the performanceof recipients may need improvement.

(b) The following evaluation criteriaare to guide the assessment processundertaken by peer reviewers. The peerreviewers determine the extent to whichrecipients meet these criteria:

(I) Implementation and management.(i) Peer reviewers shall consider thedegree to which the recipient has fullyexecuted its program of work. In doingso, peer reviewers shall considerevidence on the extent to which therecipient completes the work describedin the approved application or contract,including any approved modifications,in the time period proposed and in anefficient manner.

(ii) In examining the degree ofimplementation, peer reviewers mayalso consider evidence on the extent towhich

(A) The recipient implements andutilizes a quality assurance system forits products or services or both; and

(B) The recipient conducts self-assessment or self-evaluation activities,including periodically seeking outindependent critiques and evaluationsof its work, and uses the results toimprove performance.

(2) Quality. (i) Peer reviewers shallconsider the degree to which the


recipient's work approaches or attainsprofessional excellence. In determiningquality, peer reviewers shall considerevidence on the extent to which

(A) The recipient utilizes processes,methods, and techniques appropriate toachieve the goals and objectives for theprogram of work in the approvedapplication; and

(B) The recipient applies appropriateprocesses, methods, and techniques in amanner consistent with the higheststandards of the profession.

(ii) In determining quality, peerreviewers may also consider the extentto which the recipient conducts acoherent, sustained program of workinformed by relevant research.

(3) Utility. (i) In determining theutility of the recipient's products orservices or both, peer reviewers shallconsider evidence on the extent towhich the recipient's work (includinginformation, materials, processes,techniques, or activities) is effectivelyused by and is useful to its customersin appropriate settings.

(ii) In determining utility, peerreviewers may also consider the extentto which the recipient has receivednational recognition; e.g., articles inrefereed journals and presentations atprofessional conferences.

(4) Outcomes and impact. (i) Peerreviewers shall consider the results ofthe recipient's work. In examiningoutcomes and impact, peer reviewersshall consider evidence on the extent towhich

(A) The recipient meets the needs ofits customers; and

(B) The recipient's work contributesto the increased knowledge orunderstanding of educational problems,issues, or effective strategies.

(ii) In examining outcomes andimpact, peer reviewers may alsoconsider the extent to which recipientsaddress issues of national significancethrough its products or services or both.

(c) For National Research andDevelopment Centers, peer reviewersalso shall consider evidence on theextent to which recipients meet thefollowing criteria:

(1) Quality. (i) The recipient uses awell-conceptualized framework and

sound theoretical and methodologicaltools in conducting professionallyrigorous studies; and

(ii) The recipient conducts work ofsufficient size, scope, and duration toproduce sound guidance forimprovement efforts and futureresearch.

(2) Utility. The recipient documents,reports, and disseminates its work inways to facilitate the effective use of itswork in appropriately targeted settings.

(3) Outcomes and impact. (i) Therecipient's work contributes to thedevelopment and advancement oftheory in the field of study, including itspriority area; and

(ii) The recipient addresses issues ofnational significance through itsproducts or services or both.

(d) For the Regional EducationalLaboratories, peer reviewers also shallconsider evidence on the extent towhich recipients meet the followingcriteria:

(1) Quality. (i) The recipient utilizes awell-conceptualized framework andsound theoretical and methodologicaltools in conducting professionallyrigorous studies;

(ii) The recipient conducts work ofsufficient size, scope, and duration toproduce sound guidance forimprovement efforts; and

(iii) The recipient's products are welltested and based on sound research.

(2) Utility. The recipient documents,reports, and disseminates its work inways to facilitate its effective use inappropriately targeted settings,particularly in school improvementefforts of States and localities.

(3) Outcomes and impact. (i) Therecipient assists States and localities toimplement comprehensive schoolimprovement strategies through theprovision of research-based information(including well-tested models andstrategies), materials and assistance; and

(ii) The recipient's work results inwidespread access to informationregarding research and best practices,particularly within its region.

(e) For Field-Initiated Studies, peerreviewers also shall consider evidenceon the extent to which recipients meetthe following criteria:

3 9

(1) Implementation and management.The recipient's work responds to thegoals, objectives and mission of theNational Institute from which it isfunded.

(2) Quality. The recipient utilizes awell-conceptualized framework andsound theoretical and methodologicaltools in conducting professionallyrigorous studies.

(3) Utility. The recipient documents,reports, and disseminates its work inways to facilitate its effective use inappropriately targeted settings.

(4) Outcomes and impact. (i) Therecipient's work contributes to thedevelopment and advancement oftheory and knowledge in the field ofstudy; and

(ii) The recipient addresses issues ofnational significance through itsproducts.

(f) For the ERIC Clearinghouses, peerreviewers also shall consider evidenceon the extent to which recipients meetthe following criteria:

(1) Quality. The recipient applies anintegrated approach to acquiring anddisseminating significant and high-quality educational literature andmaterials to maintain and enhance theERIC database.

(2) Utility. The recipient contributesto the development of the ERIC databaseas a source of literature and materialsthat reflects trends and issues within itsscope.

(3) Outcomes and impact. (i) Therecipient meets the informational andeducational needs of its customersthrough dissemination and outreachapproaches and the development of anarray of print and non-print materials;and

(ii) The recipient provides nationalleadership on the use of currentcomputer, networking, and informationtechnology,(Approved by the Office of Management andBudget under control number 1850-0746)(Authority: 20 U.S.C. 6011(i)(2)(F))

[FR Doc. 98-28729 Filed 10-26-98; 8:45 am)BILLING CODE 4000-01-U

United StatesDepartment of Education

Washington, DC 20208-5651

Postage and Fees PaidU.S. Department of Education

Permit No. G-17

Official BusinessPenalty for Private Use, $300 Standard Mail (A)

&Orli*04.:

,4:1- Ulm 1,90ter:

* 'OP /1 Ø :0 *NeJ

PATES OV

u

vi

,

U.S. Department of EducationOffice of Educational Research and Improvement (0ERI)

National Library of Education (NLE)Educational Resources Information Center (ERIC)

NOTICE

REPRODUCTION BASIS

®

ERICTM029698

This document is covered by a signed "Reproduction Release(Blanket) form (on file within the ERIC system), encompassing allor classes of documents from its source organization and, therefore,does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission toreproduce, or is otherwise in the public domain and, therefore, maybe reproduced by ERIC without a signed Reproduction Release form(either "Specific Document" or "Blanket").

EFF-089 (9/97)

document resume ed 429 129 attaining excellence: a ... · document resume. ed 429 129 tm 029 698....

Documents