In Depth Panel Review Training

Activity: Mock Panel ReviewTo evaluate the Need for Assistance, reviewers will consider the extent to which the application includes: 1. A concise problem statement that identifies the current condition(s) to be addressed by the project2. Supporting information or data detailing the scope and nature of the problem3. Current challenges standing in the way of addressing the problem4. A clear description of the community to be served and who the intended beneficiaries are

What score, out of 10 possible points, did you give the application narrative?

Why?

Panel Review Roles

Reviewer: A member of the panel charged with evaluating

the application and documenting their evaluation

by scoring and commenting on the application strengths and

weaknesses

Facilitator: A member of the panel charged with managing

the panel discussions and writing the PSR

Subject Area Manager (SAM): An ANA representative charged with assisting the panel on the process and the editing of the

PSR

Program Area Manager (PAM): An ANA representative with the

responsibility to approve the PSR

Review Director: oversee the entire panel review process

Logistics Contractor: handles payment and ARM support

Panel Review Director

PAM

SAM

Facilitator 1 3 Reviewers

Facilitator 2 3 Reviewers

SAM

Facilitator 1 3 Reviewers

Facilitator 2 3 Reviewers

PAM

SAM

Facilitator 1 3 Reviewers

Facilitator 2 3 Reviewers

SAM

Facilitator 1 3 Reviewers

Facilitator 2 3 Reviewers

Panel Review Structure

Approval Process of PSRs

Calendar of EventsPre-Panel

(April 1st – April 27th)

Reviewers and facilitators are identified and trained

Session 1: P&M, EMI, and ERE(April 25th – May 9th)

Reviewers and facilitators evaluate and panel applications

Session 2: SEDS and SEEDS(May 9th – May 23rd)

Reviewers and facilitators evaluate and panel applications

Post Panel

Payment is sent to reviewers and facilitators

The Application Review Module (ARM)

Do Enter the Score First Use the ARM text box

to write your comments

Spellcheck Save every 20-30

minutes

Do Not× Use Google Chrome× Click the ARM support

link on the ARM login page

Email ANAReviewer@acf.hhs.gov

with all ARM questions!

Reviewers read application & evaluate using

FOA criteria

Reviewers write comments &

determine preliminary score

Panel members discuss

applications

Reviewers revise comments & scores; PSR compiled by facilitator & submitted to

SAM/PAM

Three Principles for Reviewing Applications

Evaluate the extent to which the application responds to the FOA Evaluation Criteria

The FOA Evaluation Criteria is the only measure used to evaluate the application

Do not compare applications against each other

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary score

Panel members

discuss applicatio

ns

Reviewers revise

comments &

scores; PSR

compiled by

Facilitator &

submitted to

SAM/PAM

Narrative Descriptor Score

Excellent 93-100

Very Good 86-92

Good 78-85

Fair 70-77

Needs Significant Improvement 0-69

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary score

Panel members

discuss applicatio

ns

Reviewers revise

comments &

scores; PSR

compiled by

Facilitator &

submitted to

SAM/PAM

How Do You Determine the Significance of a Weakness and a Strength?

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary score

Panel members

discuss applicatio

ns

Reviewers revise

comments &

scores; PSR

compiled by

Facilitator &

submitted to

SAM/PAM

The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach)

The applicant did not describe anywhere in the application how community input was used in the development of the project, making it unclear if the project proposal is truly community-driven. (page 25, Approach)

The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be evaluated, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected)

Examples of Good Weakness Comments

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary score

Panel members

discuss applicatio

ns

Reviewers revise

comments &

scores; PSR

compiled by

Facilitator &

submitted to

SAM/PAM

What is a panel

discussion like?

Not everyone agrees at the

beginning

Initial scores vary.

Sometimes by 20 points or

more

Structured and focused

We spent about an hour talking

about each application

Hearing the other reviewers

point of view made me

reconsider my score

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary score

Panel members

discuss applicatio

ns

Reviewers revise

comments &

scores; PSR

compiled by

Facilitator &

submitted to

SAM/PAM

The FACILITATOR sets group norms, focuses

the meeting, asks probing questions, shares information, and documents the

discussion

Panel Group Norms• Attend all

meetings• Keep Focused• Listen and

show respect

Focus the meeting by creating a meeting plan

Probing Questions• What criteria did you use? • How did you determine this

score? • What documentation did the

application provide that substantiates the approach?

Share initial reviewer

comments

Document the discussion to help write the Panel Summary Report

Requirements of a Panel Summary Report (PSR)

1. Factually Accurate2. Adheres to the FOA Evaluation

Criteria3. Consistent4. All comments justify the score5. Polished and Well-Written

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary

score

Panel members discuss applicati

ons

Reviewers revise commen

ts & scores;

PSR compile

d by Facilitat

or & submitte

d to SAM/PA

M

Example of an Inconsistency in a PSR

Strength CommentThe applicant clearly showed how the current staff will manage and complete the project on time. (page 75, Objective Work Plan)

Weakness CommentThe applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach)

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary

score

Panel members discuss applicati

ons

Reviewers revise commen

ts & scores;

PSR compile

d by Facilitat

or & submitte

d to SAM/PA

M

Writing a PSR that Represents the View of the Panel

Create an initial PSRTake notes during the panel discussionDraft the final PSRSubmit the final PSR to the PAM/SAM

for comments and approvalShare the final PSR with the reviewers

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary

score

Panel members discuss applicati

ons

Reviewers revise commen

ts & scores;

PSR compile

d by Facilitat

or & submitte

d to SAM/PA

M

Tips on Writing a PSR that will be Approved

Reference Application Page NumbersUse the wording of the relevant FOA

evaluation criteriaMake it grammatically correct

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary

score

Panel members discuss applicati

ons

Reviewers revise commen

ts & scores;

PSR compile

d by Facilitat

or & submitte

d to SAM/PA

M

Example of a Good PSRCommentThe applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be measured, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected)

FOA Text in Outcomes Expected Evaluation CriteriaTo evaluate the projects intended impact, reviewers will consider the extent to which: 1. The condition(s) identified in the problem statement will be addressed2. The lives of community members and beneficiaries will improve

To evaluate the impact indicator(s), reviewers will consider the extent to which: How impact will be measured with at least one impact indicator using the same measure at three points in time, baseline (beginning of project), end of project, and three years post-project.

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary

score

Panel members discuss applicati

ons

Reviewers revise commen

ts & scores;

PSR compile

d by Facilitat

or & submitte

d to SAM/PA

M

PSR Approval

Reviewers read

application &

evaluate using FOA

criteria

Reviewers write

comments &

determine

preliminary

score

Panel members discuss applicati

ons

Reviewers revise commen

ts & scores;

PSR compile

d by Facilitat

or & submitte

d to SAM/PA

M

Yes No

Is the PSR factually accurate?

Does the PSR adhere to the FOA Evaluation Criteria?

Is the PSR Consistent?

Do the comments justify the score?

Is the PSR polished and well-written?

Approved!

Roles and Responsibilities RecapReviewer

1. Evaluate the Application2. Document evaluation by scoring

and justifying score with comments

3. Submit score and comments to facilitator through ARM

Facilitator4. Create initial PSR by compiling all

reviewer comments5. Holds panel meeting with

reviewers6. Revises PSR based on reviewer

comments, discussion or SAM/PAM recommendations

7. Submit PSR to SAM through ARM

SAM/PAM Responsibility8. Determine whether the PSR can be approved. If not, back to step 6.

Facilitator Responsibility9. Share final PSR with reviewers to ensure it reflects panel opinion.

SAM Responsibility10. Submit PSR to PAM for approval through ARM

PAM Responsibility11. Approve the PSR