California's Stem Cell Genomics Award: Text of Remarks by Jeanne Loring of Scripps

Here is the text of comments from Jeanne Loring, director of the Center for Regenerative Medicine at the Scripps Research Institute, in response to questions from the California Stem Cell Report concerning the review process for applications in the $40 million stem cell genomics round.

Boldface within Loring's response is hers. Parenthetical identifications of terms or individuals have been inserted in Loring's remarks by the California Stem Cell Report.

California Stem Cell Report: What questions do you think need to be addressed considering the events surrounding the award?

Loring:

 "1. What instructions were given to first round applicants about merging applications?

"Alan (Trounson, CIRM's president) visited all of the first round applicants. On his visit to San Diego, Alan met separately with me and my Illumina partners, Larry Goldstein of UCSD and Craig Venter of the Venter Institute. Alan told me that Larry had taken him to a mens' club and Craig had taken him to a motorcycle dealership. I took him for a tour of Illumina's new facilities, which I thought was appropriate for a scientific site visit. 

"At his visit to all of the first round applicants, Alan Trounson suggested that we merge our applications if we could. My interpretation of that suggestion was that we actually merge our center proposals- UCSF's with ours, for example. I was not told that addition of subcontracts from other institutions could be interpreted as a merger with those institutions. The Stanford group's claim that Scripps and Illumina were consortium members of theirs was disingenuous, since our Scripps/ Illumina consortium was a competing genome center application, and we did not merge any part of our application with Stanford's. 

"To make the claim that they had 'merged' with Scripps and Illumina, Stanford added one project from a junior faculty member at Scripps, and contracted Illumina to do sequencing for them- they have had long-standing contracts with Illumina for services.

"The difference between Stanford's relationship with Illumina and my relationship with Illumina is analogous to Stanford paying a Tesla dealer to repair a car and me teaming up with Tesla to design a new type of car.

"We were not told who the other applicants were, which made it a challenge to determine who we might merge with. We chose to contact Pui Kwok, through my collaborator Susan Fisher at UCSF, Joe Ecker at Salk, Mike Snyder at Stanford, and Josh Stuart at UCSC about potential partnering.

"When I spoke with one of the people we contacted, he said that I could get a place in the consortium put together by Stanford if I were to get my institution to contribute $2 million in matching funds. This was not entirely a surprise; I had heard from colleagues at UCSD that UCSD's vice chancellor, David Brenner, had initiated the collection of matching funds by pledging $2 million to the consortium.

"2. At the Grants Working Group(GWG, the grant review group): what differences were there between the stated requirements in the RFA and what the reviewers were told to do during the meeting?

"Matching funds

"The RFA did NOT require contributions from grantees, and certainly did not suggest that such contributions would be considered to be items for the GWG to judge, since they were tasked with only on the quality of the science, the 'scientific merit.' To quote from the message sent to me from Gil Sambrano(the CIRM staffer who handles most of the review process), announcing my score and the GWG report, 'Applications were reviewed using the criteria detailed in the RFA and scored on scientific merit.'

"That makes me wonder if the reviewers told at the GWG meeting to include monetary contributions from the potential grantees positively in their scoring, in spite of the fact that their scoring is stated by CIRM as being solely on the basis of scientific merit?

"During the ICOC (the agency's governing board) meeting, Alan Trounson said that he had told us during his visit to all of the first round grantees that it would be important provide money for 'matching' funds. I state unequivocally that he did not tell me or anyone in my lab about this. 

"Scoring applications after removal of projects that the GWG scored poorly.

"We learned when the reviews were posted that the GWG scored two of the Stanford center projects very poorly. We do not know the actual scores they gave those projects, nor do we know what scores were given to our own projects. We do know that CIRM instructed the GWG to score the entire Stanford grant after removing the two low scoring projects. All three of our projects scored high enough to be included in the overall review of our application.

"The Stanford project received a range of scores from 70 to 95 after the two projects were removed. Our application received a range of scores from 70 to 88 with all projects left in place. It is not much of a stretch to imagine that if the low-scoring Stanford projects had been kept in, there would not have been scores of 95 by any of the reviewers. It is not too speculative to suggest that their scores would have been lower, perhaps lower than ours, if the low-scoring projects had not been removed.
We were not told of the practice of the GWG altering grants in order to improve the scores of those grants. I am also appalled that this was done, since it is not allowed in NIH review of multi project U and P awards, the closest equivalent to the CIRM genome center award. 'P' awards are Program Project Grants, in which several investigators write sub-proposals to be done in concert with each other. I review these grants, and we are instructed that we cannot remove subprojects in order to change the scores. Similarly, 'U' awards are for consortia that are to be coordinately managed. I also review these applications, and again, it is forbidden for us to alter the applications as written. The goal of the reviewers for these NIH awards is to 'review the grant we are given'.
CIRM staff indicated at the ICOC meeting that the GWG had recommended funding of the Stanford project. 

This is simply untrue. As the message from Gil Sambrano states: 'The GWG understood that this initiative will support only one or two centers and only a single data coordination and management component within a total budget of $40M.  However, as the GWG's scores and recommendations were based solely on scientific merit, the group did not select which center(s) should ultimately be funded as this is a programmatic assessment.'

The GWG was tasked with scoring based on 'scientific merit' (which CIRM instructed them would include monetary contributions). Programmatic assessment is required to choose an application, and CIRM is not part of the Programmatic committee, according to the following quote from the message from Sambrano:
'ICOC/Application Review Subcommittee MeetingFunding decisions will be made by the Application Review Subcommittee of CIRM’s governing board, the Independent Citizens Oversight Committee (ICOC), at a public meeting that will be held in Berkeley, CA on January 29, 2014.  The Subcommittee, which meets concurrently with the Board, is composed of 16 voting members (the Patient Advocate and Industry members of the Board, along with the Chair and statutory Vice Chair of the Board) and 13 non-voting members (the 13 members of the Board who are appointed from institutions that are eligible to receive CIRM funding).
'The Applications Review Subcommittee will conduct a programmatic assessment of applications reviewed by the GWG. The Subcommittee may consider any factors (such as availability of funds, overall grant portfolio, RFA priorities, strategic considerations) that might impact on their decision to fund or not fund applications. The Subcommittee aims to fund applications that are both scientifically meritorious and that bring programmatic value to the CIRM portfolio.'

In the meeting, CIRM did not present information about any application except Stanford's, giving the strong impression that this application was the only one of merit. The ICOC members of the Application Review Subcommittee were not provided with the applications in order to assess the factors they were charged to assess.

California Stem Cell Report: Are there structural issues created by Proposition 71, the measure that created the agency, that made things more or less difficult?

Loring: 

"The 29-member board (ICOC) is difficult enough to deal with, but now that most of the members are considered to be conflicted and are not allowed to even discuss the applications, we are left with a small number of non-scientists making decisions about scientific merit.

"I know that many members of the ICOC were very upset that they were unable to voice their opinions about what should be their mission- to guide CIRM's policies and choices for funding so that they are in the best interest of the voters."

California Stem Cell Report: Are there processes at CIRM that show weaknesses or need to be re-examined? Do you have any specific recommendations for changes?

Loring: 

"The ICOC members should be provided with all of the grant applications as well as the reviews. They can choose to ignore them, but if they find that certain grant apps or disease-specific areas require more high level consideration, they should have the tools to provide that guidance.

"I am concerned about the interference of the CIRM president in influencing the ICOC decisions. He has de facto power to promote or defeat specific applications, and he often wins by promoting one applicant over another. Stanford and Stanford faculty-founded companies such as Stem Cells Inc, are blatantly promoted over others. The relationship between the president and the head of the stem cell program at Stanford involves personal favors which make him conflicted and he should at the very least recuse himself from any discussion or recommendation of Stanford faculty's applications."

Loring also included the following:

Text of message from Gil Sambrano on Friday, January 10, 11:06 am
"Dear Dr. Loring:

Thank you for submitting your application under the California Institute for Regenerative Medicine’s (CIRM) Stem Cell Genomics Centers of Excellence Awards RFA 12-06. We are providing you this report as the Program Director (PD) and designated point of contact on the application, but it is your responsibility to share this information as appropriate with members of your team.

The applications underwent peer review at a meeting held on November 7-8, 2013 by the members of CIRM’s Grants Working Group (GWG). Applications were reviewed using the criteria detailed in the RFA and scored on scientific merit.

Review ReportBelow, please find the Review Report of your application.  This report includes the average scientific score and the funding recommendation of the GWG. Applications are scored on a scale that ranges from 1 – 100, with 100 being the highest achievable score.

Applications are separated into three funding tiers.  An application’s average score determines the funding tier as follows:

75-100 = Tier 1 - recommended for funding
65-74 = Tier 2 - moderate scientific quality or consensus on scientific merit cannot be reached, and may be suitable for programmatic consideration by the ICOC
1-64 = Tier 3 - not recommended for funding

The Review Report provides only a brief summary of the evaluation of your application by the GWG. The report is not an exhaustive critique and does not cover all of the factors that may have contributed to the final score or the final recommendation. The report highlights key points relevant to the review criteria that were captured from reviewers’ written comments and from the discussion of your proposal by the GWG during the review meeting.

ICOC/Application Review Subcommittee MeetingFunding decisions will be made by the Application Review Subcommittee of CIRM’s governing board, the Independent Citizens Oversight Committee (ICOC), at a public meeting that will be held in Berkeley, CA on January 29, 2014.  The Subcommittee, which meets concurrently with the Board, is composed of 16 voting members (the Patient Advocate and Industry members of the Board, along with the Chair and statutory Vice Chair of the Board) and 13 non-voting members (the 13 members of the Board who are appointed from institutions that are eligible to receive CIRM funding).

The Applications Review Subcommittee will conduct a programmatic assessment of applications reviewed by the GWG. The Subcommittee may consider any factors (such as availability of funds, overall grant portfolio, RFA priorities, strategic considerations) that might impact on their decision to fund or not fund applications. The Subcommittee aims to fund applications that are both scientifically meritorious and that bring programmatic value to the CIRM portfolio.

Under California’s open meeting laws, members of the public, including applicants for CIRM funding, may provide written and oral comments to the ICOC regarding items on the Board’s agenda. Applicants may attend and observe the ICOC meeting. Applicants may contribute oral comments for not more than three (3) minutes during the public comment periods. The ICOC Chairman will announce the public comment period, which typically occurs prior to the Board’s voting on any motion. Applicants may also provide written comments to the ICOC. All correspondence to the ICOC must be submitted to the Executive Director of the ICOC, Maria Bonneville, at mbonneville@cirm.ca.gov. Please do not send correspondence to the ICOC that relates to an appeal of a funding recommendation by the GWG as it will be redirected to the CIRM Review Office (see “Response to Review” below).

In preparation for the ICOC meeting, the Review Report (with PI and institution identities removed) will be posted no later than Friday, January 17, 2014 on our website at http://www.cirm.ca.gov/ReviewReports. Additional information for CIRM’s public meetings can also be found on our website.

Award NotificationCIRM will notify you by email of the ICOC’s funding decisions following the ICOC meeting.

Response to ReviewThe GWG conducts the scientific evaluation of proposals submitted to CIRM. If the applicant (PI/PD) wishes to appeal the scientific review by the GWG (or seek reconsideration of the recommendation), the PI/PD must first consult with the CIRM Review Office. All appeal requests must be made through the CIRM Review Office within 10 days of CIRM making this report available (i.e., deadline is January 21, 2014). Grounds for an appeal are limited to the circumstances described in “Guidance for Appeal of Scientific Review and Reconsideration Policy” available via this link: http://www.cirm.ca.gov/board-and-meetings/guidance-appeals-and-requests-reconsideration-grants-working-group-funding
--
Gilberto R. Sambrano, Ph.D.
Associate Director, Review
California Institute for Regenerative Medicine
210 King Street
San Francisco, CA 94107
(415) 396-9103
To:  jloring@scripps.edu; wynne@scripps.edu
REVIEW REPORT FOR CIRM RFA 12-06R GENOMICS CENTERS OF EXCELLENCE AWARDS (R)
Application: Center for Advanced Stem Cell Genomics

PI: Jeanne Loring
Institution: Scripps Research Institute

Recommendation Overview: The GWG provided two final scores for each application as follows: 1) an overall center score (covering the center-initiated projects, collaborative research activities, and center organization and operations plan) and 2) a data coordination and management component score.
The overall scientific merit and quality of the proposals submitted under this RFA were viewed by the GWG to be deserving of high scores. Overall center scores placed four proposals in Tier 1, one proposal in Tier 2 and none in Tier3. The separate data management and coordination component scores placed two proposals in Tier 1, two in Tier 2 and one in Tier 3.
The GWG understood that this initiative will support only one or two centers and only a single data coordination and management component within a total budget of $40M.  However, as the GWG's scores and recommendations were based solely on scientific merit, the group did not select which center(s) should ultimately be funded as this is a programmatic assessment. CIRM staff is recommending that the ICOC fund only the highest scoring genomics center and the corresponding highest data coordination and management center which together will fulfill the goals of this initiative.
The scores, GWG Tier recommendation and CIRM staff recommendation are as follows:

GWG Overall Center Recommendation: Tier 1GWG Overall Final Score: 76

GWG Data Center Recommendation: Tier 2GWG Data Center Final Score: 72
CIRM Staff Recommendation: Do not fund
EXECUTIVE SUMMARYThis Genomics Center will be led by a program director (PD) from an academic institution and a co-PD from an industry organization. Three Center-Initiated Projects (CIPs) are proposed, and, as required by the RFA, a plan for inclusion of Collaborative Research Projects and a Data Coordination and Management Center are described.


Center Organization and Operational Plan - The organization of the proposed Genomics Center is well conceived as a collaboration between highly qualified investigators from an academic institution and an industry partner, representing a diversity of competencies. The balance between expertise in stem cell biology and genomics technologies is a particular strength.

- The PD has extensive research experience at the interface of stem cell biology and genomics and is committed to serving the stem cell community; he/she is well suited to lead this program.

- The industry partner institution, and especially the co-PD, is well positioned to develop novel cutting edge genomics technologies and make them accessible to customers.
- The teams from the two applicant institutions have a well-established, strong working relationship; reviewers considered this an important attribute of this proposal.

- All elements necessary for the establishment and operation of a successful Genomics Center are in place; the structure and composition of the proposed administrative and oversight committees are appropriate and should ensure both delivery of projects and high standards of work.

- The three CIPs are designed to support the service aspects of this Genomics Center by focusing on the development of tools that can be generally used to explore genomics data. Reviewers considered it a strength that, if successful, these projects will both create novel tools and technologies and validate them. There was some concern that some of the tools may not be made easily and widely available.

- Although a letter from leadership indicates enthusiastic institutional support from the academic institution, no additional funds or specific dedicated space have been designated. Reviewers expressed serious concern about this lack of material commitment.

- Some reviewers expressed concern that both the PD and co-PD are already heavily committed individuals and questioned whether they would have the capacity to fully provide a strong commitment to this project.


Center Organization and Operational Plan - The proposed Genomics Center appears well designed to support collaborative research projects and to make relevant state-of-the-art genomics technologies readily accessible to investigators with primary expertise in stem cell biology or translational research.

- The proposed application process is appropriate, review procedures and criteria are well thought out.

- The offer to culture cells for external collaborators in the Genomics Center's core lab is especially appealing, as that would remove a variable from the experiments and thus help with standardization of conditions for genomics assays.

- Concern was expressed about whether potential collaborators who have limited experience in genomics would receive adequate assistance in designing their proposed studies.


CIP-1The applicants propose to develop an updated and expanded version of an existing genomics tool. They plan to make available global gene expression and epigenomic data, obtained through a series of systematic analyses of human pluripotent stem cells and their derivatives, to serve as reference for future experiments. They also propose to analyze the heterogeneity of stem cell populations and to develop genomic tools for the assessment of stem cell quality. Finally, they intend to use disease-specific induced pluripotent stem cells (iPSC) to study the molecular basis of two neurodevelopmental diseases and identify disease-modifying compounds.

- This project adopts a broad approach toward developing pragmatic, accessible tools for basic research on stem cells in vitro, and to lay the groundwork for more effective clinical translation. This would represent a valuable resource to the stem cell community.

- Enthusiasm was diminished by the notion that most of the activities are applications of existing tools or extensions of existing work. While important goals, the activities were not viewed as particularly innovative.

- The goals of this project are overly ambitious, raising doubt that all aims can be achieved. Given the tremendous track record of the principal investigator (PI), though, it is expected that substantial progress will be made.

- Reviewers' opinions about the utility of an already existing analytical tool, to be further developed under this award, were divided. Some judged it positively as an important tool that has been made freely available in its current form and were enthusiastic about the plan for dissemination of the updated version. Conceptually, they considered the proposed approach to be very valuable, as it has the potential to provide objective standards for assessing cell fate and for quality control of cell populations. Other reviewers expressed concern that the current tool has not been widely adopted in the stem cell community, calling into question its usefulness.
- The proposed work on neurodevelopmental diseases is disconnected from the central focus of this project and might have been better developed as a separate project.
- Reviewers criticized the general lack of experimental detail, particularly in aims 4 and 5, which impeded assessment of feasibility.
- The project plays to the strengths of the PI as a well-established leader in the stem cell field with a strong record of productivity and innovation.

- The broad scope of the project is matched by the experience and expertise of the team involved.

CIP-2This project addresses the integrity of stem cells for clinical transplantations and their utility in translational and clinical research. The goals are to establish informatics tools for determining the functional significance of genome wide molecular variations in therapeutic stem cell populations and to develop and validate methods for assessing the prevalence of deleterious alterations in stem cell populations. The applicants also plan to develop and validate a workflow for integrating genomics information with identification of potential therapeutic compounds and their effects on patient-derived induced pluripotent stem cells and on patient treatment outcomes. Finally, the plan is to disseminate all developed tools and protocols to the stem cell community.

- The utility of the proposed tools and protocols for translational genomics-based research would be high.
- The first goal is straightforward and feasible. Reviewers emphasized that input data must be high quality, as noted by the applicants, and suggested that applicants consider that the cellular differentiation state may affect functional significance of specific genomics variation.

- Other goals are more risky, but if successfully developed, they should be widely applicable and help stimulate the use of stem cell-based systems to explore both disease mechanisms and potential therapies.
- Toward assessing prevalence of deleterious alterations in stem cell populations, reviewers recommended that a variety of additional stem cell datasets, especially some originating outside the team, be included in the project.

- The feasibility of a key component of the study, linking genomic information from patients to potential therapeutics and individualized treatments, was difficult to assess, since the applicants did not specify the types of diseases to be studied.
- Reviewers observed that parts of this proposal are vague and hard to follow, and it was unclear what some of the deliverables would be.

- There is a clear plan for dissemination of the acquired expertise and knowledge.

- The PI is well qualified for this work and has assembled a powerful leadership team that possesses the necessary expertise.


CIP-3This project is led by the industry partner organization and is focused on the development of several single cell genomic technologies and tools for large-scale epigenetic analyses.

- Reviewers noted that only one of the technologies under development is truly stem cell-specific, but the proposed work would nevertheless deliver technologies extremely valuable to the stem cell community.

- The tools to be improved or developed are at the forefront of technical advances.

- Reviewers geatly appreciated the novelty of one of the proposed technologies.
- Some reviewers were concerned that the project essentially constitutes commercial development of genomic products and questioned whether it was appropriate for CIRM to support this activity.  Others felt that the developed technologies would provide valuable research tools and could have great potential impact on stem cell science.
- Concern was expressed about whether the new technologies would be specifically disseminated to California investigators and whether their cost might be prohibitive to many researchers.

- The basic technologies underlying this proposal have already been developed and it should therefore be feasible to complete the proposed developments in the proposed time scale.

- The PI and team are exceptionally well qualified to deliver on this project.


Data Coordination and Management - The DCM team is led by two individuals. One has a track record of developing a highly successful, adaptable, user-friendly platform. The other is an expert in medical informatics, although reviewers expressed concern that a biosketch for this individual was not included in the proposal.

-The proposed DCM structure and leadership would likely ensure solid database structure, data access and visualization capabilities.
- Reviewers considered the details provided on the data management plan to be inadequate; there was little description of how the DCM Center will participate in data integration and analysis or interact with various projects.

- Reviewers acknowledged the importance of patient privacy protection but felt the focus on this issue in the application was overemphasized and distracting.

- The descriptions of data visualization tools are reasonable but not particularly innovative or tailored to the specific needs of the stem cell community."