just-in-time irb review: capitalizing on scientific merit review to improve human subjects research...

Just-in-Time IRB Review: Capitalizing on Scientific Merit Review to Improve Human SubjectsResearch ComplianceAuthor(s): P. Adam Kelly and Michael L. JohnsonSource: IRB: Ethics and Human Research, Vol. 27, No. 2 (Mar. - Apr., 2005), pp. 6-10Published by: The Hastings CenterStable URL: http://www.jstor.org/stable/3564480 .

Accessed: 09/06/2014 16:47

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

The Hastings Center is collaborating with JSTOR to digitize, preserve and extend access to IRB: Ethics andHuman Research.

http://www.jstor.org

This content downloaded from 195.78.109.193 on Mon, 9 Jun 2014 16:47:49 PMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=hastings

http://www.jstor.org/stable/3564480?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


IM7CASIE STUDY

BY P. ADAM KELLY AND MICHAEL L. JOHNSON

Just-In-Time IRB Review: Capitalizing on Scientific Merit Review to Improve Human Subjects Research Compliance

he concept of "Just-In-Time" production migrat- ed from manufacturing to scientific merit review in May zooo, when the National Institutes of

Health (NIH) issued a statement that grant applications submitted to NIH henceforth would be subject to "Just- In-Time" Institutional Review Board (IRB) review.' The

concept made sense: only those applications deemed by NIH peer review to be worthy of funding would under-

go IRB review at the principal investigator's home insti-

tution, thereby in theory reducing the review burden on IRBs. Considering the size of NIH's extramural research

portfolio, $16.8 billion in FY zooz,2 the upside potential of Just-In-Time review for IRBs seemed incontrovert- ible. And indeed, after about a year had passed, anec- dotal evidence emerged that the NIH policy was popular with IRBs, although no formal policy evaluation had been conducted.

In April zooz, the Department of Veterans Affairs Health Services Research and Development Service (VA HSR&D) adopted a Just-In-Time IRB review policy sim-

ilar to the NIH policy for its $104 million (FY zooz) research portfolio.3 A difference was that VA HSR&D made compliance with its policy voluntary at the level of

VA's regional administrative centers and, for those

regions adopting the voluntary approach, at the individ- ual medical center level as well. Because VA HSR&D wanted to proactively manage and respond to implementation of the Just-In-Time review policy, it commis- sioned a simultaneous evaluation of the policy.

The VA HSR&D research program is an intramural

program for VA clinician and non-clinician researchers that provides funding for research at the interface of health care systems, patients, and health care outcomes. VA HSR&D research specifically addresses quality, access, and cost-effectiveness of services and therapies,4 through projects that address such areas as pain man-

agement, ethnic and cultural variations in care, evalua-

tion of prognostic tools and screening procedures for diseases such as prostate disease, issues of access to care, quality of care, and patient satisfaction.s

Under the VA HSR&D Just-In-Time IRB review policy (JIT policy), grant applications are accepted for scientific merit review without prior IRB approval.7 Applicants are advised that IRB approval must be received by VA HSR&D before funds for funded applications will be disbursed, but are not told specifically when to seek IRB approval. At the time of the policy's implementation, VA HSR&D anticipated that the JIT policy would potentially i) decrease the review burden on IRBs, z) decrease the grant application preparation burden on principal investigators, 3) increase the grant funds disbursement workload of VA HSR&D fiscal

operations (although initially the specific nature of that increase was not known), and 4) decrease the effectiveness of VA HSR&D's scientific merit review by depriv- ing reviewers of IRB review outcome information that

previously they in most instances had received prior to

evaluating a grant application. The fourth potential effect of the JIT policy was of

particular concern. VA HSR&D has been protective of its scientific merit review program, a program consis-

tently highly regarded both within VA and nationally, and was wary of any threat to its performance from the

JIT policy. The evaluation was intended to investigate and help VA HSR&D proactively manage the impact of each of these four potential effects.

Study Methods

Tn April 2ooz, at approximately the same time as the

IJIT policy was implemented, investigators from within VA HSR&D commenced an evaluation of the policy. VA HSR&D's Management Consultation Program, the

sponsor of the evaluation, specified that the timeline for

completion be limited to 15 months. The evaluation

incorporated both qualitative and quantitative approaches to data collection and analysis. Qualitative

P. Adam Kelly and Michael L. Johnson, "Just-In-Time IRB Review: Capitalizing on Scientific Merit Review to Improve Human Subjects Research Compliance," IRB: Ethics & Human Research 27 No. 2 (2005): 6-1o.

I MARCH-APRIL 2005 IRB: ETHICS & HUMAN RESEARCH



methods included collection of verbal self-report data from structured interviews of key stakeholders in the research program. Quantitative methods included visual surveys conducted by non-participant observers of two significant research administration activities at VA HSR&D, the scientific merit review process, and the grant funds disbursement process.

For the verbal self-report data collection, the team selected an "extreme case" sampling design,8 focusing intentionally on institutions and individuals having the largest and smallest volume of funded research through VA HSR&D. According to Patton, this sampling approach is useful in that "extreme cases may be information-rich cases precisely because, by being unusual, they can illuminate both the unusual and the typical."9 To identify the largest and smallest VA HSR&D research program sites, we examined a national database of VA HSR&D grant application volumes. We determined the quantities of grant applications submitted to VA HSR&D, applications reviewed by the semiannual VA HSR&D scientific merit review meeting, and applications assigned "fundable" and "non-fundable" scores for the years 1999-zoo2001. We then used these numbers to rank all research program sites from largest to smallest and to rank order the most prolific applicants (principal investigators) from each research program site.

Qualitative (Verbal) Data. We conducted telephone interviews with members of 27 "key" institution IRBs, including the affiliate IRBs of the VA medical facilities hosting the largest HSR&D research programs and the on-site IRBs of VA medical facilities hosting several of the smallest programs. We also conducted three- month follow-up interviews with members of each of three randomly selected largest-program affiliate IRBs. We asked IRB members 1) what effect, if any, they perceived the JIT policy having had on the volume and quality of grant applications coming through their IRB from VA HSR&D, and z) whether any perceived effects were sizable relative to the overall grant application mix (VA plus non-VA) their IRB regularly sees. We also conducted telephone interviews of nearly all the scientific merit reviewers who participated in the June 2002zooz and

January 20zoo3 VA HSR&D scientific merit review meetings. These individuals were asked questions similar to the two questions asked of IRB members, and also were asked 1) whether they perceived any change in the burden on themselves or their colleagues during merit review, and z) whether grant application review discus- sions had taken any new directions, in their opinion,

since implementation of the JIT policy. Lastly, we conducted telephone interviews with 32

principal investigators, representing VA facilities hosting the largest and smallest HSR&D programs. We selected respondents based on two criteria: the investigator's numerical rank in quantity of grant applications submitted during the period of 1999-20ol, and the investigator's location, to ensure geographic diversity of the sample. In these interviews, we asked investigators about their perceptions of the grant submission process and of how the JIT policy had impacted that process for them. Investigators at key institution VA HSR&D program sites were asked 1) whether the volume of grant applications they submitted had changed since the implementation of the JIT policy, z) what benefits, if any, they had realized as a result of the policy, and 3) whether these benefits appeared sizable relative to the overall mix of grant applications (VA plus non-VA) the investigator regularly produced.

I Quantitative (Observational) Data. We conducted a site visit of fiscal operations at the VA HSR&D cen- tral office. During that visit, we observed the grant funds disbursement process in detail. We also observed the June zooz and January 2003 scientific merit review meetings. Members of our team were located in each of the several different reviewing rooms that typically oper- ate simultaneously throughout the meeting. In order to quantify the average amount of time reviewers invested on human subjects-related issues, we recorded the frequency, type, and length (in minutes and seconds) of human subjects-related comments made by reviewers during each grant application review.

Study Results

JIT Policy's Impact on IRB Review Burden. Of the 15 IRB members we interviewed who were affiliated with the largest VA HSR&D research programs, nine reported having experienced grant application volume in

zooz that was equal to or slightly increased from Zool levels, while six of 15 reported having experienced a slight to significant decrease. Twelve of 15 perceived no

change in "overall workload" (e.g., intensity of grant application reviews, duration of reviews, clerical duties, education/training) of their IRB from April zooz through January 2003, and this sentiment was reaffirmed in May

20zoo3 by the follow-up interviews with three largest-program IRB members.

Five of 12 smallest-program IRB members experienced a moderate increase in grant application volume,

IRB: ETHICS & HUMAN RESEARCH MARCH-APRIL 2005

IU This content downloaded from 195.78.109.193 on Mon, 9 Jun 2014 16:47:49 PM

All use subject to JSTOR Terms and Conditions


while seven of 12 experienced no change. Four of 12

perceived a moderate increase in "overall workload" since April zooz; two of these linked their perception to an increase in pressure on them to approve grant applications quickly. They stated, "the PIs rush us to get the

approval because the [applications] were funded," and

"when we get an application after notice of score, we're under a little more time pressure to get it through."

Several respondents from both largest- and smallest-

program IRBs stated that grant application reviews seem to have become more equally distributed over the calen- dar year. One respondent expressed a common sentiment: "We're not crunched trying to meet the submission deadline dates." Neither largest- nor smallest-program IRB members reported detecting a change in grant application quality. Several stated that it was "still too

early to tell" whether the JIT policy would have an

impact on application quality, and this sentiment

emerged again during the follow-up interviews. Lastly, the majority of largest-program IRB members (13 of 15) and smallest-program IRB members (1o of 1 z) were unable to distinguish any effects of the VA HSR&D JIT policy from effects of the analogous policy of the much

larger NIH research programs. In the words of one

respondent, "We have an active program ... the number of those [applications] going through this [VA] process is a very small number."

f JIT Policy's Impact on Principal Investigators' Application Preparation Burden. Of the 32 principal investigators we interviewed, z29 perceived that the JIT policy had made submitting grant applications easier for them. Respondents said, "there are fewer steps in terms of getting [an application] out the door and through the VA system," and "it's definitely more user friendly for the researchers ... and it makes sense." However, only 11 of 25 perceived the JIT policy as actually having reduced their grant application submission workload.

According to one respondent, "in the big scheme of getting a grant funded, it's not a huge piece." Lastly, none of the principal investigators stated that the JIT policy impacted their handling of human subjects-related issues in grant application preparation. As one investigator remarked, "I do it the same way, in terms of what I do for human subjects protection."

JIT Policy's Impact on Funds Disbursement. During our site visit to the VA HSR&D research program fiscal operations, we discovered that the JIT policy had the potential to cause delay in funds disbursement for those grant applications receiving a "fundable" score

in scientific merit review, since the arrival of IRB

approval would occur well after merit review. Fiscal

operations staff indicated that this potential for delay was of particular concern for managing funds disbursement on multi-site studies, since disbursing funds across

multiple sites invariably would mean withholding fund-

ing from some sites until IRB approval letters arrive, while disbursing funds to other sites. The staff indicated that computerization of funds tracking would mitigate the problem to a degree, but that monitoring of IRB

approval across sites in multi-site studies would become a larger and more time-consuming task, both for fiscal

operations staff and for investigators. To quantify the magnitude of this new delay in proj-

ect funds disbursement, we worked with the fiscal operations staff to track funds disbursement for 23 grant applications assigned a fundable score during the

January, 2003 VA HSR&D scientific merit review meet-

ing. We found that as of October , 2oo003, the JIT policy had caused delays in funds disbursement for approximately 6o% (14 of 23) of these fundable projects. However, as of January , oo004, nine of these 14 delayed projects had received home institution IRB

approval and, by February 1, 2oo004, all had received home institution IRB approval.

JIT Policy's Impact on Scientific Merit Review Effectiveness. The results of our scientific merit review observation are shown in Table 1. On average, human

subjects-related issues were mentioned 1.4 times per grant application, consuming 3 % of an application's total discussion time. In the telephone interviews, reviewers indicated being aware that the grant applications they were reviewing lacked advance IRB approval, but they expressed confidence that their professional attentiveness to human subjects-related issues would screen out unac-

ceptable grant applications. As one respondent explained, "I think, as in the past, there are IRB concerns that come up, that are identified in the review ... I

don't think that part has changed." Most reviewers perceived no significant change in the

quality of grant applications they reviewed. A common sentiment of those few who did notice a decrease in

application quality is captured in this response: "There were one or two that came through that should not have

gotten through to begin with, that locally were not scru- tinized, and should have been weeded out at the institutional level."

Lastly, about one-third (12 of 38) of reviewers reported perceiving more discussion of human subjects-related

a8 MARCH-APRIL 2005 IRB: ETHICS & HUMAN RESEARCH



Table 1. Summary of Human Subjects-Related Comments Made and Discussion Time Elapsed During Scientific Merit Review

(statistics shown are review group averages per grant application reviewed)

Human Subjects-Related Comments*

Review Session Frequency Elapsed Review Time Total Review Time % Human Grant Applications (date, review group) Subjects-Related Reviewed

6/02, Room 1 1.31 1:01 27:56 3.5% 16 6/02, Room 2 1.22 1:03 30:06 3.1% 18 1/03, Room 1 1.19 0:22 28:06 1.3% 21 1/03, Room 2 1.54 0:26 21:30 2.3% 26 1/03, Room 3 2.50 1:47 21:37 8.0% 4 Weighted Average 1.39 0:43 26:08 3.0% -

of Group Averages

* Comments recorded were limited to those relating to the following issues: preserving participant confidentiality, obtaining proper informed consent, ensuring no undue participant burden, and general ethical concerns of reviewers.

issues after JIT policy implementation, while two-thirds (26 of 3 8) reported no change in the level of human subjects-related discussion. Said one respondent, "Reviewers are perhaps particularly conscious that [an application] doesn't have another external approval, but these [human subjects-related] issues were raised before the [Just-In-Time policy]."

Conclusions

WTT e found support for VA HSR&D's assertion that its reviewers look at IRB-related issues and

address them explicitly in the review process, regardless of whether IRB approval has been obtained prior to merit review. In fact, many merit reviewers indicated that their review system by design operates independently of the prior decisionmaking of other evaluative bodies in

making its funding decisions. This strong confidence in the VA HSR&D scientific merit review process proved well founded, as none of the grant applications assigned a fundable score in FY zooz or FY 2003 under the JIT policy was subsequently denied home institution IRB

approval. We believe that we also encountered early evidence

that the JIT policy reduces IRBs' grant application review burden, even though IRB members may not have

perceived it as such. We posit that, over time, if a JIT policy is successful, its effect should be a gradual slowing of the steep increase in overall grant application volume and an increase in the quality of human subjects compliance evident in the applications that do reach IRBs. This is, in fact, what at least one IRB member we interviewed has observed: "We're seeing more grant applications,

and we would have seen even more, if not for [the Just- In-Time policy]."

The two reasons for this are evident: 1) IRBs no longer see grant applications assigned an "unfundable" score by merit reviewers; and 2) those applications that IRBs do receive have been screened for human subjects compliance by the merit review process. As described in the Belmont Report, acts of beneficence are those that, all else equal, "maximize possible benefits and minimize possible harms."'10 We believe that Just-In-Time IRB review fits this description, constituting an act of beneficence by a funding agency towards a key partner in the scientific research process, namely, the IRB.

Generalizing our findings to Just-In-Time IRB review by any funding agency, IRBs' review workload stands to be reduced 1) absolutely, because IRBs see fewer applications than they would see without Just-In-Time review, and z) conditionally, to the extent that a given IRB trusts the judgment of the funding agency's merit review process that an application is scientifically sound. That is, even though IRBs have a statutory requirement to conduct diligent human subjects reviews to ensure compliance in all research at their institutions, IRBs neverthe- less have an opportunity to capitalize on prior merit review in order to increase the efficiency of their own scientific review of the proposed project. More generally, IRBs and funding agency merit review boards have responsibilities that overlap to a small but significant degree, and these responsibilities cannot be delegated by one entity to the other. However, the entities can help ease each other's burden by supplying the results of their work to the other. In the past, it largely has been IRBs

IRB: ETHICS & HUMAN RESEARCH MARCH-APRIL 2005 a9 This content downloaded from 195.78.109.193 on Mon, 9 Jun 2014 16:47:49 PM

All use subject to JSTOR Terms and Conditions


supplying their results to merit review boards; Just-In- Time IRB review illustrates that this relationship works in the opposite direction as well. In the case of a funding agency such as VA HSR&D, which places strong empha- sis on both scientific value and human subjects compliance in merit review, the two workload-reduction effects of Just-In-Time review described above may provide IRBs a real benefit over time.

As for concerns about the increase in complexity of

grant funds disbursement and project management under

Just-In-Time IRB review, we recommend that funding agencies employing Just-In-Time IRB review carefully attend to the financial disbursement process, and make

appropriate arrangements in advance to accommodate a

likely increase in processing workload. We also suggest that principal investigators view the new time lapse between grant funds approval and actual funds disbursement as a permanent procedural change, rather than as a

negative consequence of the JIT policy. As one research

manager explained to us:

Other funding agencies have these sorts of delays in funding ... As people get used to [the JIT policy], they will know to put in start dates of four to six months after [merit review] rather than two to four months. Hopefully, they also learn to adopt phased budgets for multi-site studies (e.g., skeleton staff after home IRB has been approved but before all sites are approved and real data collection can begin).

We note several limitations to our study findings. First, given the large size and geographic spread of the VA HSR&D research program, we were compelled ulti-

mately to sample from among many small research program sites, and believe we may have experienced a

degree of non-response bias from this subpopulation in

particular. Second, the evaluation approach we employed did not include quantification of the review workloads of the IRBs we contacted; therefore, we have no quantitative evidence of IRBs' workload change due to the JIT policy. However, we do have verbal evidence that IRBs affiliated with certain research program sites experienced a slight reprieve from the upsurge in grant applications experienced across VA HSR&D during the evaluation

period. Lastly, we recognize that the voluntary nature of the JIT policy, which resulted in varied implementation of the policy across the VA HSR&D sites we examined, limited our ability to capture and report on the policy's full impact. However, we suggest that the findings pre- sented here represent what we expect to emerge as the

policy becomes implemented more broadly across VA HSR&D. We recommend a follow-up evaluation to cap-

ture what by now is likely substantial further development in the effects of Just-In-Time IRB review.

Acknowledgments W e wish to thank Abeer A. Alsarraj, BS, for her

extraordinary dedication as the research assistant for this study; Cynthia Boudreaux, MEd, for her assistance in

designing the evaluation plan; and Dianna Bufkin, AS, for administrative assistance. We are grateful to Martha Bryan, EdD, and Linda McIvor, MS, for guiding us through the scientific merit review process, and to Mary Jones, BS, for her help with fiscal operations issues. Many thanks also to Neil Thakur, PhD, our VA HSR&D program officer, for his assistance in making the research plan feasible and for his

helpful insights in reviewing this manuscript. This research was supported by the Health Services Research and

Development Service of the U.S. Department of Veterans Affairs.

Disclaimer: The views expressed in this article are those of the authors and do not necessarily represent the views of the U.S. Department of Veterans Affairs or Baylor College of Medicine.

E P. Adam Kelly, PhD, MBA, and Michael L. Johnson, PhD, are employed by the Health Services Research and Development Service of the U.S. Department of Veterans Affairs and by Baylor College of Medicine. Drs. Kelly and Johnson are supported by career development awards from the Health Services Research and Development Service of the U.S. Department of Veterans Affairs.

References 1. National Institutes of Health. Revised Policy for IRB Review of

Human Subjects Protocols in Grant Applications. (May zooo). http://grants.nih.gov/grants/guide/notice-files/NOT-OD-00oo-03 1.htm.

2. National Institutes of Health. NIH Awards (competing and non- competing) by Fiscal Year and Funding Mechanism, Fiscal Years 1992- zooz. (March 2004). http://grantsz.nih.gov/grants/award/trends/ fund92oz.htm.

3. Department of Veterans Affairs, Health Services Research and Development Service. A Message From the Director of HSR&D. (December 2oo003). http://www.hsrd.research.va.gov/about/ directorletter.cfm.

4. Department of Veterans Affairs, Health Services Research and Development Service. About Health Services Research and Development Service (HSR&D). (December 2003). http://www. hsrd.research.va.gov/about.

s. See ref. 3, Department of Veterans Affairs 2003. 6. See ref. 3, Department of Veterans Affairs 2oo003. 7. Department of Veterans Affairs, Memorandum. (August 2oo003).

http://www. i.va.gov/resdev/directive/Proposal-Limit.pdf. 8. Patton MQ. Qualitative Research & Evaluation Methods. 3d ed.

Thousand Oaks, CA: Sage Publications, 2ooz. 9. See ref. 8, Patton zooz, p. 234. io. National Commission for the Protection of Human Subjects of

Biomedical and Behavioral Research. The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Washington, DC: Government Printing Office, 1979, p. 5.

U MARCH-APRIL 2005 IRB: ETHICS & HUMAN RESEARCH



just-in-time irb review: capitalizing on scientific merit review to improve human subjects research...

Documents