Learning by Doing: CRCHD Scholars Gain Early Experience in Mock Review Session
, by CRCHD Staff
Dr. Albert Farias and Dr. Bisrat Debeb
It felt real. These were actual R01 applications. An NIH Center for Scientific Review (CSR) Scientific Review Officer (SRO) presided over the panels. The same process applied.
And that was the intent—for the 21 scholars participating and for those observing the mock review to experience a “simulation training” of the NIH peer review process. The mock review session was one component of the 2019 Professional Development Workshop (PDW) and Mock Review (MR), held by the National Cancer Institute’s (NCI) Center to Reduce Cancer Health Disparities (CRCHD) for its Continuing Umbrella of Research Experiences (CURE) scholars on June 3-4, 2019, at the NIH Natcher Conference Center.
It would be difficult to overstate the significance of the NIH peer review process, which ensures that grant applications are assessed based on their scientific and technical merit.
“The NIH peer review system is the foundation of the NIH extramural research enterprise, so it is crucial to understand how it works,” says Behrous Davani, Ph.D., Program Director with the National Institute of General Medical Sciences and previously of CRCHD.
Given the peer review process’s potential impact on the prospects of an independent research career, it is integral for scholars to be introduced to the process early in their careers.
“The PDW Mock Review settings are special, with participants who have just started navigating their path through NIH peer review process,” says Svetlana Kotliarova, Ph.D., Scientific Review Officer, CSR, Cancer Prevention Study Section.
CSR reviews 77% of all NIH proposals, engaging 18,500 distinct reviewers in this work, according to the Center’s website.
“…The Mock Review provided me with an insider’s view of what a real-life NCI peer review process and discussion looks like,” says Oreofe O. Odejide, MD, MPH, who was one of the MR chairs and who is Assistant Professor of Medicine, Harvard Medical School, and Physician, Dana-Farber Cancer Institute.
Dr. Svetlana Kotliarova and Dr. Oreofe Odejide
Preparing for the MR
For MR participants, the work began before the PDW and MR commenced. All 21 CURE scholars—who had no experience or limited experience in NIH peer review—were assigned to one of the three panels and one of three roles—chair, reviewer or discussant. These scholars received training in the process, says Dr. Davani, including orientation sessions organized by Dr. Kotliarova and the opportunity to ask questions of him or Dr. Kotliarova.
The Day of the MR
Over the course of three live panels, reviewers experienced in person the NIH peer review process, including exposure to the scoring system and review criteria, and had the opportunity to develop needed skills to review and critique a grant application. Throughout the MR, participants received feedback ranging from administrative or logistical guidance into how reviews are structured or run to dealing with discrete, substantive issues. For example:
“In high-impact applications, the strengths should prevail; only essentially flawless applications, which also are expected to have a large and wide impact, should get score of 1,” says Dr. Kotliarova, addressing a point that is underscored at study section meetings.
After each panel, Dr. Davani or CRCHD Program Director Sara L. Hargrave, Ph.D., pulled back the curtain, revealing the actual NIH study section discussion points and scores for these previously submitted R01 applications.
“Seeing how our critiques and scores compared to that of the actual NIH study section was also a useful way to gauge the strengths and limits of our approach to the grant review process,” says Colin A. Flaveny, Ph.D., who was one of the reviewers and who is Assistant Professor, Department of Pharmacology and Physiology, Saint Louis University College of Medicine.
Dr. Julian Burks, Dr. Colin Flaveny and Dr. Magdalena Bieniasz
Lessons and Takeaways
Looking back, MR participants identified a variety of specific lessons and takeaways from the experience including, among others, differing opinions among reviewers, understanding what reviewers are looking for, and appreciating the value and importance of each portion of the grant application. Most of all, they were nearly unanimous in expressing that the experience has or will improve their grant writing skills.
“By exposing me to the peer review process and getting an insider’s view of how grant applications are evaluated, the Mock Review provided me an invaluable experience on how to craft a winning grant proposal,” says Bisrat Debeb, D.V.M., Ph.D., who was one of the reviewers and who is Assistant Professor, Department of Breast Medical Oncology, The University of Texas MD Anderson Cancer Center.
In this sense, serving on a MR panel is doubly beneficial, as participants gain experience in and refine their skills as grant reviewers and writers.
“It's been said that we learn by doing, and that definitely applies to the peer review of grant applications,” says Dr. Davani, later continuing, “Being a reviewer gives a researcher the best education that one can get in what makes a grant application more competitive.”
Dr. Albert Farias and Dr. Bisrat Debeb
Programs Directors Emphasize Communicating with NCI Staff, Early Career Reviewer Opportunities
During the ensuing discussion, participants and observers could ask questions of a panel of NCI Program Directors: Dr. Davani and Tiffany Wallace, Ph.D., of CRCHD, Susan A. McCarthy, Ph.D., of the Division of Cancer Biology, and Damali N. Martin, Ph.D., M.P.H, of the Division of Cancer Control and Population Sciences, as well as Dr. Kotliarova, of CSR. These experts are well positioned to provide specific feedback and tips to prospective grant applicants.
“Program Officers can provide a plethora of guidance, including highlighting NCI’s programmatic priorities, clarifying the requirements of an FOA, and providing general grantsmanship guidance,” Dr. Wallace says.
Understanding NCI’s current priorities may indicate in which areas NCI “is most willing to invest in for the coming year(s),” explains Dr. Wallace, which “therefore may help inform the direction of your proposed research.” In addition to speaking with NCI program staff, Dr. Wallace says researchers can try to stay apprised of NCI’s current priorities by watching public NCI advisory meetings via live videocast.
Dr. Yamilé Molina
CURE scholars heard and registered these important messages. Yamilé Molina Ph.D., M.P.H., who was one of the discussants and who is Assistant Professor, University of Illinois at Chicago, shared several take-home messages, including that "It is crucial to be 'up to date' on what will be required for applications and what criteria will be used during reviews."
Dr. Susan McCarthy emphasizes the importance of, and the great opportunity for, researchers to get in the door on serving as a reviewer earlier in their careers, including through NIH’s Early Career Reviewer (ECR) Program and reviewer programs operated by local associations and American Cancer Society chapters.
The MR offers a natural progression for CURE scholars to be exposed to the peer review process over time, and to learn and engage with it in a friendly atmosphere.
“One particularly poignant feature of the study section overall is that it provided a less-intimidating environment for young investigators to voice their opinions than other programs would provide,” Dr. Flaveny says.