IOM Description of CIRM Grant Review Process

Here is the text of the Institute of Medicine's description of how the grant review process works at California stem cell agency. You can find CIRM's version as exhibit B in this document.

"CIRM staff are available to potential applicants to discuss ideas and to answer questions about published RFAs and the conformity of a particular proposal to the goals of announced programs. From responses to a questionnaire submitted by the committee to the California stem cell scientific community, 4 it appears that views on discussions of this type vary, with some individuals being highly appreciative of these preliminary discussions and others finding the CIRM staff less accessible (IOM, 2012d). The committee agrees that having a system for communicating with potential applicants early in the process is important, in particular to ensure that neither applicants nor CIRM staff are spending large amounts of time writing or assessing proposals that are not in keeping with the goals of any particular RFA. The committee also suggests that CIRM continue making its scientific staff available to potential applicants and working with this constituency to maximize the effectiveness of this aspect of the grant submission process.

"CIRM staff recognized that the number of applications that would potentially be received for a given RFA could overwhelm the Institute’s ability to review each rigorously for scientific merit. Accordingly, during its early years, CIRM restricted the number of applications that would be accepted from any one institution in response to a particular RFA. The reasoning was that doing so would limit the overall number of applications, making the review process manageable while guaranteeing that applications would represent the scientific communities at a wide range of California institutions. This was especially important given that CIRM’s enabling legislation limited administrative expenditures, requiring that the process for grant-making decisions be streamlined. However, there was considerable pushback from potential grantees, as it was thought that some individuals, in particular junior investigators or those new to stem cell biology, were at a disadvantage in competing with colleagues at their home institutions for the right to submit a proposal and hence had limited access to possible CIRM support.

"To address this concern while keeping the number of proposals sent for full review manageable, CIRM established a pre-application procedure and eliminated the restriction on the number of applications that could be submitted from any single institution (CIRM, 2011d). The preapplication procedure is similar to a process used by a number of private foundations that provide support for biomedical research. Applicants are asked to provide a shortened version of their proposal through the CIRM website. CIRM staff evaluate these shortened proposals to ensure that they are in keeping with the RFA. Those deemed responsive to the RFA are then sent to three outside reviewers, who are also provided the RFA. Each reviewer is asked to evaluate the preapplication, indicating whether it should definitely, possibly, or definitely not be invited as a full proposal. Additionally, each reviewer is asked to identify proposals that are among the two to three best in the group being evaluated by that reviewer (each reviewer typically is given 10-25 pre-applications to consider). No written critique is requested of the evaluators. Using these initial external evaluations, CIRM staff determine which applicants will be invited to submit full proposals. Once invited, proposals must be based on the pre-application proposal that was submitted. There is no appeal process for pre-applications that are not invited for a full proposal submission (CIRM, 2011e).

"After the pre-application process was piloted, applicants, reviewers, CIRM staff, and the ICOC board members were surveyed regarding its acceptability (CIRM, 2011e). As might be expected, applicants often expressed frustration that there was no feedback on why their pre-application was not selected to move forward. Additionally, in responses to the committee’s questionnaire 5 , some principal investigators raised concern about whether a short proposal contains sufficient detail for an informed review (IOM, 2012d). On balance, however, there appeared to be overwhelming support for the pre-application process, especially in comparison with the previous model whereby there was a limit on the number of applications that could be submitted from any single institution (CIRM, 2011e). The committee agrees that, despite its limitations, the current preapplication procedure opens up the opportunity for CIRM funding to a broader cohort of investigators and is, in principle, an appropriate process. The committee recognizes the tension between providing applicants as much information as possible and not overburdening reviewers, and suggests that CIRM consider ways of offering applicants more information on the shortcomings perceived in preapplications that were not selected for further consideration.

"The Scientific and Medical Research Funding Working Group, designated in most CIRM materials as the Grants Working Group (GWG), is the entity charged with reviewing scientific proposals and making recommendations to the ICOC (the CIRM governing board) with respect to those that should be funded.

"The GWG is appointed by the ICOC and consists of 23 members, including the chair of the ICOC, 7 of the 10 ICOC patient advocates, and 15 non-California scientists known for their expertise in stem cell biology (CIRM, 2009f, 2012g). The 15 scientists are selected based on the particulars of the individual RFAs and are drawn from a pool of more than 150 individuals chosen by CIRM as highly qualified to review proposals. Participation of these experts, none of whom, as non-Californians, are eligible for CIRM funding and stand to gain directly from CIRM, is instrumental in providing the rigorous scientific review required for making funding decisions. The success CIRM has had in commissioning outstanding review committees for each of its RFAs is a testament both to the Institute’s stature in the eyes of the stem cell community and the willingness of stem cell scientists outside of California to contribute their time and effort to facilitate the work of their California colleagues.

"Full proposals received by CIRM by the RFA deadline are entered into the CIRM database, and all GWG members assigned to this review cycle declare any conflicts of interest with any of the applications (CIRM, 2009g). Any GWG member in conflict for a particular application is recused during discussions, scoring, and final voting. The GWG members are then assigned applications for which no conflict exists based on their unique expertise. Typically, three external scientists review each application. The GWG can call on additional specialist reviewers as needed if its own expertise is insufficient to evaluate the science in any individual application adequately. Prior to the GWG’s face-to-face meeting, each reviewer and ad hoc specialist submits a scientific score (1-100, with 100 being best) and a written critique for each assigned application. A meeting of the GWG is then announced on the CIRM website. This meeting starts with a session open to the public, during which GWG business is conducted. The GWG then meets in closed session for a two-stage review of the applications (CIRM, 2011g).

"The first stage of the review is scientific in nature, led by the chair of the GWG (an external scientist member appointed to this role by the ICOC). The assigned reviewers declare their scores for the application being discussed and briefly summarize the basis for their recommendations. This is followed by full discussion of the application by GWG members, ending with the assigned reviewers suggesting revised scores based on the discussion. Each scientific member of the GWG not in conflict with that application then submits a final scientific score. Although ad hoc specialist reviewers can suggest scores in their written evaluations and, if present, during the discussion, only GWG members can submit a final score. The final scientific score is the arithmetic mean of the reviewers’ scores. If there is a wide divergence in scores with a sizable proportion (greater than 35 percent) of the GWG being in disagreement with the majority view, a minority report is forwarded to the ICOC along with the final score (CIRM, 2011g).

"The next stage is the programmatic review, chaired by one of the patient advocate members of the GWG appointed to this position by the ICOC (CIRM, 2011g). The purpose of this review is to evaluate all of the applications taking into account not only their scientific scores but also the overall purpose of the RFA, with the goal of segregating the applications into three tiers— recommended, provisionally recommended, or not recommended for funding. This process has two steps. First, a histogram of the scores of all of the applications is generated. Of note, at this stage the applications are deidentified, and only the scores are revealed. The GWG examines this histogram and identifies natural breaks to divide the applications into the three tiers based on their scores. Next, the applications are identified so that the scientific score (and tier) of each is made known. GWG members (except those with conflicts, who leave the room) begin a discussion to determine whether any of the applications should be moved from one tier to another in an effort to achieve a balanced portfolio representing a spectrum of priority disease areas, scientific approaches, innovation, and so forth. For an application to be moved from one tier to another, a majority vote of the GWG is required; all members of the GWG not in conflict (scientists and patient advocates) participate in this vote. Once the GWG is satisfied with the final ranking of proposals, a final vote is taken, and the rank order is proposed to the ICOC for approval. For each application, in addition to its final ranking, the scientific score voted by the scientists on the GWG is provided to the ICOC (CIRM, 2011g; IOM, 2012e).

"The ICOC makes funding decisions at a meeting scheduled and publicized in advance. As with other ICOC agenda items, deliberations on the funding of applications begin in a session that is open to the public. ICOC board members in conflict with any particular application are recused from both this public discussion and any subsequent private deliberations. Prior to the ICOC meeting, summary information about each application is available on the CIRM website, including how that particular application ranked relative to the others and its tier designation.

"Applications are redacted, however, to remove information that would identify applicants or institutions. Individual applicants are aware of how their proposal scored and how likely it is to be funded, and have the opportunity to make an “extraordinary petition” to the ICOC. Any ICOC board member may request that the petition be heard. In such cases, petitioners are invited to the ICOC meeting to explain why they believe the assigned score and priority ranking are not appropriate.(California Stem Cell Report note: The preceding sentence is in error. Petitioners are not invited to appear. In fact, some have not understood their right to appear. Others do not even understand that they can file a petition.)

"The ICOC takes this information (the petitions) into consideration as it deliberates about the final ranking of applications. If it is necessary to discuss proprietary information, the ICOC may meet in closed session before a final vote is taken on which applications will be funded. As a result of its private and public deliberations, the ICOC may move applications from one tier to another before taking a final vote, after which applicants are notified about funding decisions. Examination of ICOC records indicates that the shifting of applications from one tier to another does occur. For example, as of October 22, 2012, 62 extraordinary petitions were heard by the ICOC, of which 20 (32 percent) were successfully funded (CIRM, 2012h). While most of this shifting is between adjacent tiers, there have been cases in which applications have been moved from tier 3 to tier 1 (CIRM, 2011g; IOM, 2012e); this has occurred with applications for major programs with large budgets. As discussed in greater detail below, the committee is troubled by the extraordinary petition mechanism and suggests that this practice be eliminated. The committee recognizes that CIRM has recently initiated a self-study regarding all aspects of extraordinary petitions."