Thursday, March 22, 2007

The CHA Example: How CIRM Decides Who Gets the Big Bucks

The $2.6 million California stem cell grant involving the CHA Regenerative Medicine Institute received a score of 77 from a panel of grant reviewers, although they commented that it "can be easily qualified as overly ambitious."

Approval of the application last week by the Oversight Committee of the California Institute for Regenerative Medicine has resulted in calls for an investigation into CHA RMI's nonprofit status and its links to a Korean scientist involved in an international plagiarism case, among other things.

The CHA application first came up for a review last January by a CIRM working group, chaired by Stuart Orkin of the Dana Farber Cancer Institute. Fourteen other scientists held seats on the group. Seven members of the Oversight Committee sat on the panel. Only one is from Los Angeles, where CHA RMI has its office. It is not known whether she was in attendance when the CHA application was discussed. All of the scientists are from out-of-state.

Meeting privately, the reviewers recommended the CHA application and others for funding. The CHA application was placed in the first tier of grants that were sent on to the Oversight Committee. The scores of the first tier grants ranged from 95 to 66. The reviewers received detailed information on the proposal, including the names of the principal researcher as well as its methodology. Only one reviewer was recused from considering the grant. He was Jeffrey Rothstein of John Hopkins, who works in ALS research, a field that was also targeted by the grant.

Prior to action by the Oversight Committee, the names of all CIRM grant applicants and their institutions are secret except during the private meetings of reviewers, according to CIRM policies. The Oversight Committee is also not told their names during the votes on the reviewers' recommendations. The names of the winning applicants are only disclosed after the vote. The names of the losers will never be disclosed by CIRM.

CIRM says its secrecy is justified for a number of reasons. The agency says it is the traditional way grant applications are handled in the scientific community. It is professionally damaging, CIRM also says, for scientists to be publicly identified as not being able to win grants. It is also damaging to be criticized in public. Maintaining secrecy means that scientists are more likely to propose more ambitious and riskier research than would otherwise be the case. The results of science will be better in the aggregate, thus benefitting the public more than would identifying the applicants and their institutions, CIRM says.

During last week's Oversight Committee meetings when the grants were approved, the 29 members of that panel were not told the names of the applicants or the institutions. They were given a summary that is also available to the public. Individual members were given a list of the grants by number on which they could not vote or participate in the debate. Those lists were withheld from the public at the meeting. Just prior to voting on or discussing an individual grant, a list was read of the committee members who could not participate in the debate. At that point, well-informed members of the audience and probably many members of the committee could identify the actual institutions involved and often the individual researchers. The persons who could not are ones who are not as well informed on stem cell research.

The Oversight Committee voted on the first tier of grants as a block. At that point, no list of recused members was read to the public. Rather each member announced that they were voting in favor of the block with exception of grants where they had a conflict. CIRM's outside counsel recommended the procedure.

Following the vote, CIRM posted a list on the Internet of Oversight Committee members recused from voting on the CHA grant. They are Ricardo Azziz, chair of Department of Obstetrics and Gynecology at Cedars-Sinai in Los Angeles, Jeanne Fontana, a surrogate for John Reed, head of the Burnham Institute in La Jolla, and Richard Murphy, president of the Salk Institute, also in the La Jolla area. Reasons for their recusal were not posted.

(The California Stem Cell Report has argued often against much of the secrecy in the grant-making process for a variety of reasons. We will write more about the issue later.)

In response to a query, Dale Carlson, chief communications officer for CIRM, supplied the following:
"The CIRM grant review and administration process does not end with the ICOC's vote on deciding which applications to approve or not approve for funding. To that point, the review process by the Grants Working Group is focused on scientific merit. After that, there is an internal administrative review by Institute staff to ensure that each approved application is from an institution and principal investigator that meet the eligibility requirements of the specific Request for Applications (RFA); of the requested budget and proposed facilities for the proposed project; and of the institution's mechanisms for complying with our grants administration policy and medical and ethical standards.

"The administrative review process can take several weeks (we are still working on the SEED grants approved in mid-February, for example) and only after it's completed to our satisfaction do Notice of Grant Awards (NGAs) go out to recipient institutions and researchers. Checks follow NGAs.

"The NIH grants review process is similar."
The principal investigator on the CHI RMI grant is Jang-Won Lee. Little information is available about him on the CIRM web site. Carlson said the score of 77 on his grant is an average of each score by each reviewer. Here are the rankings of the grants.

Below is the text of the strengths and weaknesses of his application based on the CIRM reviewers assessment. More information on the grant can be found at this location.

"STRENGTHS: The proposal is well-written and includes preliminary data in pigs and novel methods. The research plan is nicely developed and the PI has the appropriate expertise, at least in animal cloning (less with hESCs), to be successful in this endeavor. Success of the PI in the porcine model adds strength to the plan. A large collection of letters of support provides evidence of enthusiastic collaboration with the PI that will add critically needed expertise to the project. The plan to differentiate and transplant hESC-derived neural cells in a well-established mouse model with experts in the field strengthens the lack of experience with hESC culture (but not derivation) by the rest of the group.

"WEAKNESSES: This is a proposal that can be easily qualified as overly ambitious. The author provides a shopping list of all the experiments that will happen after the ALS SCNT embryos have successfully been established and characterized. This seems premature. The proposal would be successful if the derivation is first done accurately and convincingly to generate a handful of lines that will be available for the community. Preliminary data on enucleation, SCNT and hESC derivation in an animal model should be done before proposing these studies. Specifically, SCNT on frozen oocytes in an animal model should be done before using completely viable, clinically useful human oocytes. The use of frozen oocytes for SCNT has not been established, and is likely to be a significant technical problem for enucleation and whole cell injection. There is no indication of a plan to enucleate the oocytes in the proposal and a clear rationale for using one or both of the methods used previously by the collaborator who developed the method is required. A plan for the derivation of hESCs is also needed along with a rationale for the use of ALS cells for tranplantation studies, rather than normal cells. It also appears that no one on this project has experience with this hESC derivation, or the derivation of any ESC lines.

"The section on clinical grade ESCs is not necessary for the proposal and should be removed. These ESCs are not stable lines that have been shown to be maintained in vitro. In fact, they appear by the literature and preliminary data to be a mixture of hESCs and hESC-derived differentiated populations. The plans to differentiate hESCs for transplantation do not require this intermediate step. It is unfortunate, because the application of novel SCNT techniques is a reasonable way to move the field of SCNT and hESC biology forward. If the rest of the proposal was as well-designed as the pig studies, the score would be very high."

4 comments:

  1. I find this whole issue of opennness in grant review fascinating in a variety of ways. Some of the arguments given for why grant review should not be open seem spurious (e.g., "it is the traditional way grant applications are handled in the scientific community" or "It is professionally damaging for scientists to be publicly identified as not being able to win grants. It is also damaging to be criticized in public.")

    But there is one issue which is certainly worth thinking about more deeply. This is the issue of "Maintaining secrecy means that scientists are more likely to propose more ambitious and riskier research than would otherwise be the case. The results of science will be better in the aggregate, thus benefitting the public more than would identifying the applicants and their institutions, CIRM says." I think in general this is probably true for grant proposals, at least with the way science works now. In some aspects of science, just the IDEA of doing some type of work is kept under wraps for many reasons. And if one knew that a grant review system was going to be completely open, then some people would not put their best ideas in the grants. This would truly hurt the progression of science, since then this might limit the scope of what is being proposed, which in turn would limit the scope of what is funded. I think it would be worth looking into if there are any fully open grant review systems and if so, whether that has stifled creativity.

    ReplyDelete
  2. Jonathan makes some very good points. However, can we say for sure that the usual way of making grants is the best? Has it been tested against a competing model? Probably not. I doubt a fully open model exists. The NIH model has major weaknesses, one of which surfaced earlier this weak. Here is a quote from a Washington Post story about how the situation at the NIH is stifling good science:

    "'Researchers shy away from real discoveries. They're becoming worriers, not explorers,' said Dr. Stephen Strittmatter, a professor of neurobiology at the Yale University School of Medicine.

        "'When scientists have to spend most of their time trying to get funded, caution wins out over cutting-edge ideas, creativity sacrifices to convention, and scientific progress gives way to meetings and grant applications,' said Dr. Robert Siliciano, an infectious disease specialist with Johns Hopkins University School of Medicine."

    Perhaps it is time to open up the game and freshen and enliven the scientific community.

    The NIH model also does not apply in California. CIRM is fundamentally a different organization. As we all know, the NIH can be regulated by both the president and Congress, who have the power of the purse over it. Enshrined in the State Constitution, CIRM is virtually untouchable by the governor or the legislature, which is all the more reason why its affairs should be more transparent than the NIH.

    Here is a link to the Post story:
    http://www.washingtontimes.com/functions/print.php?StoryID=20070319-104820-8232r

    ReplyDelete
  3. I don't know of a completely open review process. I do know of one that is more transparent than California's. In Connecticut the oversight committee receives summaries like those in California, except that they include the name and the institution of the applicant. These are avaialble to the public at the meeting where the awards are made. That would be an improvement on California's system.

    John Simpson
    Foundation for Taxpayer and Consumer Rights

    ReplyDelete
  4. I think without a doubt it is time to open up the game and freshen and enliven scientists. This is why I support Open Access to the scientific literature (see e.g., http://www.plos.org) as well as Open Peer Review of scientific publication (see e.g., http://www.plosone.org/home.action and http://www.biology-direct.com/). I would unquestionably support an Open model of grant review. There is no doubt it is worth trying. However, as I said, given the way the scientific community currently works, I am not sure it is ready for Open grant review.

    However, I am not certain this will help scientists do more creative work. If you really want creative work, in my mind, you need to ditch the grant system almost entirely. You need to do something like Howard Hughes does with their "Investigators". Basically, they have a competition to select excellent scientists, and then they given them money and say "Do excellent work. Come back in five years. And if you were good. We will probably give you money again" .

    Since for most funds we are stuck with grant based systems, then we need to make these as good as possible. As I said in my other comments I think CIRM deserves some credit for being more Open than many other systems. And I understand you want them to be more Open even than that. Maybe there are ways to do this without too much resistance and without directly forcing change. For example, you could see if any of the people who had grants rejected would be willing to post their grants and reviews online. Basically, you could create a YouTube for grants. Not sure if you would get any buy in. But if some people out there felt unfairly treated, they might want to post their grants. As I said in my other comments, I would bet most scientists would not want to do this. But you might get a few. And then it might spread.

    Or you could create a journal - the journal of rejected grant proposals.

    ReplyDelete

Search This Blog