Here is the text of the
Institute of
Medicine's description of how the grant review process works at
California stem cell agency. You can find
CIRM's version
as exhibit B in this document.
"CIRM staff are available to potential
applicants to discuss ideas and to answer questions about published
RFAs and the conformity of a particular proposal to the goals of
announced programs. From responses to a questionnaire submitted by
the committee to the California stem cell scientific community, 4 it
appears that views on discussions of this type vary, with some
individuals being highly appreciative of these preliminary
discussions and others finding the CIRM staff less accessible (IOM,
2012d). The committee agrees that having a system for communicating
with potential applicants early in the process is important, in
particular to ensure that neither applicants nor CIRM staff are
spending large amounts of time writing or assessing proposals that
are not in keeping with the goals of any particular RFA. The
committee also suggests that CIRM continue making its scientific
staff available to potential applicants and working with this
constituency to maximize the effectiveness of this aspect of the
grant submission process.
"CIRM staff recognized that the number
of applications that would potentially be received for a given RFA
could overwhelm the Institute’s ability to review each rigorously
for scientific merit. Accordingly, during its early years, CIRM
restricted the number of applications that would be accepted from any
one institution in response to a particular RFA. The reasoning was
that doing so would limit the overall number of applications, making
the review process manageable while guaranteeing that applications
would represent the scientific communities at a wide range of
California institutions. This was especially important given that
CIRM’s enabling legislation limited administrative expenditures,
requiring that the process for grant-making decisions be streamlined.
However, there was considerable pushback from potential grantees, as
it was thought that some individuals, in particular junior
investigators or those new to stem cell biology, were at a
disadvantage in competing with colleagues at their home institutions
for the right to submit a proposal and hence had limited access to
possible CIRM support.
"To address this concern while keeping
the number of proposals sent for full review manageable, CIRM
established a pre-application procedure and eliminated the
restriction on the number of applications that could be submitted
from any single institution (CIRM, 2011d). The preapplication
procedure is similar to a process used by a number of private
foundations that provide support for biomedical research. Applicants
are asked to provide a shortened version of their proposal through
the CIRM website. CIRM staff evaluate these shortened proposals to
ensure that they are in keeping with the RFA. Those deemed responsive
to the RFA are then sent to three outside reviewers, who are also
provided the RFA. Each reviewer is asked to evaluate the
preapplication, indicating whether it should definitely, possibly, or
definitely not be invited as a full proposal. Additionally, each
reviewer is asked to identify proposals that are among the two to
three best in the group being evaluated by that reviewer (each
reviewer typically is given 10-25 pre-applications to consider). No
written critique is requested of the evaluators. Using these initial
external evaluations, CIRM staff determine which applicants will be
invited to submit full proposals. Once invited, proposals must be
based on the pre-application proposal that was submitted. There is no
appeal process for pre-applications that are not invited for a full
proposal submission (CIRM, 2011e).
"After the pre-application process was
piloted, applicants, reviewers, CIRM staff, and the ICOC board
members were surveyed regarding its acceptability (CIRM, 2011e). As
might be expected, applicants often expressed frustration that there
was no feedback on why their pre-application was not selected to move
forward. Additionally, in responses to the committee’s
questionnaire 5 , some principal investigators raised concern about
whether a short proposal contains sufficient detail for an informed
review (IOM, 2012d). On balance, however, there appeared to be
overwhelming support for the pre-application process, especially in
comparison with the previous model whereby there was a limit on the
number of applications that could be submitted from any single
institution (CIRM, 2011e). The committee agrees that, despite its
limitations, the current preapplication procedure opens up the
opportunity for CIRM funding to a broader cohort of investigators and
is, in principle, an appropriate process. The committee recognizes
the tension between providing applicants as much information as
possible and not overburdening reviewers, and suggests that CIRM
consider ways of offering applicants more information on the
shortcomings perceived in preapplications that were not selected for
further consideration.
"The Scientific and Medical Research
Funding Working Group, designated in most CIRM materials as the
Grants Working Group (GWG), is the entity charged with reviewing
scientific proposals and making recommendations to the ICOC (the CIRM
governing board) with respect to those that should be funded.
"The GWG is appointed by the ICOC and
consists of 23 members, including the chair of the ICOC, 7 of the 10
ICOC patient advocates, and 15 non-California scientists known for
their expertise in stem cell biology (CIRM, 2009f, 2012g). The 15
scientists are selected based on the particulars of the individual
RFAs and are drawn from a pool of more than 150 individuals chosen by
CIRM as highly qualified to review proposals. Participation of these
experts, none of whom, as non-Californians, are eligible for CIRM
funding and stand to gain directly from CIRM, is instrumental in
providing the rigorous scientific review required for making funding
decisions. The success CIRM has had in commissioning outstanding
review committees for each of its RFAs is a testament both to the
Institute’s stature in the eyes of the stem cell community and the
willingness of stem cell scientists outside of California to
contribute their time and effort to facilitate the work of their
California colleagues.
"Full proposals received by CIRM by the
RFA deadline are entered into the CIRM database, and all GWG members
assigned to this review cycle declare any conflicts of interest with
any of the applications (CIRM, 2009g). Any GWG member in conflict for
a particular application is recused during discussions, scoring, and
final voting. The GWG members are then assigned applications for
which no conflict exists based on their unique expertise. Typically,
three external scientists review each application. The GWG can call
on additional specialist reviewers as needed if its own expertise is
insufficient to evaluate the science in any individual application
adequately. Prior to the GWG’s face-to-face meeting, each reviewer
and ad hoc specialist submits a scientific score (1-100, with 100
being best) and a written critique for each assigned application. A
meeting of the GWG is then announced on the CIRM website. This
meeting starts with a session open to the public, during which GWG
business is conducted. The GWG then meets in closed session for a
two-stage review of the applications (CIRM, 2011g).
"The first stage of the review is
scientific in nature, led by the chair of the GWG (an external
scientist member appointed to this role by the ICOC). The assigned
reviewers declare their scores for the application being discussed
and briefly summarize the basis for their recommendations. This is
followed by full discussion of the application by GWG members, ending
with the assigned reviewers suggesting revised scores based on the
discussion. Each scientific member of the GWG not in conflict with
that application then submits a final scientific score. Although ad
hoc specialist reviewers can suggest scores in their written
evaluations and, if present, during the discussion, only GWG members
can submit a final score. The final scientific score is the
arithmetic mean of the reviewers’ scores. If there is a wide
divergence in scores with a sizable proportion (greater than 35
percent) of the GWG being in disagreement with the majority view, a
minority report is forwarded to the ICOC along with the final score
(CIRM, 2011g).
"The next stage is the programmatic
review, chaired by one of the patient advocate members of the GWG
appointed to this position by the ICOC (CIRM, 2011g). The purpose of
this review is to evaluate all of the applications taking into
account not only their scientific scores but also the overall purpose
of the RFA, with the goal of segregating the applications into three
tiers— recommended, provisionally recommended, or not recommended
for funding. This process has two steps. First, a histogram of the
scores of all of the applications is generated. Of note, at this
stage the applications are deidentified, and only the scores are
revealed. The GWG examines this histogram and identifies natural
breaks to divide the applications into the three tiers based on their
scores. Next, the applications are identified so that the scientific
score (and tier) of each is made known. GWG members (except those
with conflicts, who leave the room) begin a discussion to determine
whether any of the applications should be moved from one tier to
another in an effort to achieve a balanced portfolio representing a
spectrum of priority disease areas, scientific approaches,
innovation, and so forth. For an application to be moved from one
tier to another, a majority vote of the GWG is required; all members
of the GWG not in conflict (scientists and patient advocates)
participate in this vote. Once the GWG is satisfied with the final
ranking of proposals, a final vote is taken, and the rank order is
proposed to the ICOC for approval. For each application, in addition
to its final ranking, the scientific score voted by the scientists on
the GWG is provided to the ICOC (CIRM, 2011g; IOM, 2012e).
"The ICOC makes funding decisions at a
meeting scheduled and publicized in advance. As with other ICOC
agenda items, deliberations on the funding of applications begin in a
session that is open to the public. ICOC board members in conflict
with any particular application are recused from both this public
discussion and any subsequent private deliberations. Prior to the
ICOC meeting, summary information about each application is available
on the CIRM website, including how that particular application ranked
relative to the others and its tier designation.
"Applications are redacted, however, to
remove information that would identify applicants or institutions.
Individual applicants are aware of how their proposal scored and how
likely it is to be funded, and have the opportunity to make an
“extraordinary petition” to the ICOC. Any ICOC board member may
request that the petition be heard. In such cases, petitioners are
invited to the ICOC meeting to explain why they believe the assigned
score and priority ranking are not appropriate.(California Stem Cell
Report note: The preceding sentence is in error. Petitioners are not
invited to appear. In fact, some have not understood their right to
appear. Others do not even understand that they can file a petition.)
"The ICOC takes this information (the
petitions) into consideration as it deliberates about the final
ranking of applications. If it is necessary to discuss proprietary
information, the ICOC may meet in closed session before a final vote
is taken on which applications will be funded. As a result of its
private and public deliberations, the ICOC may move applications from
one tier to another before taking a final vote, after which
applicants are notified about funding decisions. Examination of ICOC
records indicates that the shifting of applications from one tier to
another does occur. For example, as of October 22, 2012, 62
extraordinary petitions were heard by the ICOC, of which 20 (32
percent) were successfully funded (CIRM, 2012h). While most of this
shifting is between adjacent tiers, there have been cases in which
applications have been moved from tier 3 to tier 1 (CIRM, 2011g; IOM,
2012e); this has occurred with applications for major programs with
large budgets. As discussed in greater detail below, the committee is
troubled by the extraordinary petition mechanism and suggests that
this practice be eliminated. The committee recognizes that CIRM has
recently initiated a self-study regarding all aspects of
extraordinary petitions."
