Showing posts with label peer review. Show all posts
Showing posts with label peer review. Show all posts

Thursday, March 26, 2015

$25 Million for Stem Cell Research Plus Sharp Criticism of California Stem Cell Board

The California stem cell agency today approved $25.2 million for preclinical research aimed at speeding development of therapies for afflictions that include arthritis, Alzheimer's and “bubble boy” syndrome.

The action came during an agency governing board session that was marked by sharp criticism from an executive with a La Jolla firm associated with one of the rejected grant applicants, Jill Helms of Stanford. The executive, Sanford Madigan of Ankasa Regenerative Therapeutics, told the board he was "disturbed by board members not willing to do their jobs."

Sanford Madigan, LinkedIn photo
Several board members said they were offended by Madigan's remarks, although he later apologized. He appeared to be irritated by the reluctance of the board to override its reviewers. Five board members subsequently abstained from voting to overturn staff and reviewer rejection of Helms' application. The motion that would have funded her proposal failed on a vote of 5-1-5.

The public flare-up involving the board and a member of the public was unusual, although comments such as Madigan's are sometimes heard in private.

His remarks came at the end of a discussion about the weaknesses of the agency's closed-door review process. The discussion involved other applications in addition to Helms' proposal. Randy Mills, president of the agency, acknowledged the review process was "sub-optimal" and was being replaced by new procedures he has dubbed CIRM 2.0.

Ankasa was created by Avalon Ventures, which is raising $16 million to help commercialize Helm's work to develop a stem cell therapy to improve skeletal healing of elderly people.

The governing board of the agency handed out awards to seven scientists with the goal of moving their research closer to federal approval for a start of a clinical trial. The board approved applications from two researchers who made special appeals to the board after reviewers and staff nixed their proposals.

Prior to the vote on her application, Helms was told that she could revise her $7 million application and submit it for later consideration. She said the soonest that would occur would be in about 10 months, given the agency's timetable. By that time, she said the research team would be disbanded through lack of funding and the effort would "disintegrate." 

CIRM has already put $7 million into Helms' research. She has been something of a bright star for the agency, which has featured her on its Web site and asked her to appear on its panels.

The rejection of the Helms proposal highlighted some of the vagaries of the agency's longstanding review process.  She received an average scientific score  of 72 which knocked her out of the funding category(tier one). However, her median score was 75, a figure that two reviewer-approved grants received. What skewed the average figure was a low score of 45. No other grant in the top 10 received that a ranking that low. Eight reviewers recommended funding the proposal, Four did not with three recommending it be placed in "tier two," a grey area for applications.

No reviewers were listed as having conflicts of interest on the Helms proposal. However, their finanicial and professional interests are not disclosed to the public. Nor are they identified to the public.

On another proposal, John Cashman, CEO of Human BioMolecular Research Institute of San Diego, asked for a re-review of his firm's application for a treatment involving spinal fusion. He said that his firm has worked with CIRM for several years to make changes that the agency had requested. Nonetheless, he said reviewers do not seem to agree.  Cashman said,
"Certainly the system is broken."
The board did not act on Cashman's request for a re-review.

Here is the roll call vote on the Helms' application: Yes -- David Higgins, Steve Juelsgaard, Kathy LaPorte, Lauren Miller, Art Torres. No -- Jonathan Thomas. Abstain -- Francisco Prieto, Robert Quint, Al Rowlett, Jeff Sheehy, Diane Winokur.

Summaries of the reviews of the winning applications can be found on this 90-page document.

(Editor's note: An earlier version of this item did not contain the two paragraphs that deal with the vagaries of the grant review process and conflicts of interest.) 

Wednesday, July 02, 2014

Expectations, Ballyhoo and Stem Cell Research

Two seemingly unrelated biotech stories popped up this morning on the news.  One involved an international stem cell research brouhaha. The other involved what could amount to a nearly $2 billion biotech deal for a California firm.  

What brings them together is the diaphanous nature of some of the work in these much ballyhooed fields. But first, let’s look at the latest reports about the STAP stem cell flap concerning research in Japan and Massachusetts that seemed to promise a fast and easy way to make pluripotent stem cells.

After five months and major questions, the journal Nature has decided to retract the STAP paper despite the fact that the journal had it vetted by some of the best scientists in the world. Even with the review, Nature said “extensive” errors have surfaced along with “inexplicable discrepancies.”

It is fair to say that 20 years ago, that paper would still be widely accepted and remain firmly entrenched in Nature’s archive as reliable. What has changed is the Internet and impact of social media on evaluation of research. That has given researchers the unfettered ability to discuss and publish their findings dealing with replication of results and other issues.  At the same time, the speed in which this cyber review takes place is remarkable.  The change from 20 years ago is the equivalent of the move from hand-cranked printing presses to the high-speed presses of today that can spit out thousands of pages an hour.

(We should note that California stem cell researcher Paul Knoepfler of UC Davis played an important role in probing the scientific reliability of the STAP research with responsible reporting and commentary on his blog, ipscell.com.)

Now, about that nearly $2 billion deal, Wall Street Journal columnist Helen Thomas this morning wrote about the acquisition of Seragon Pharmaceutical by Roche, describing it as “disconcerting.” She said it could be a case of shelling   “out vast sums for assets that could quite possibly amount to nothing.”  San Diego-based Seragon “was formed only last year and has one breast cancer drug in early stage trials,” Thomas wrote.

She continued,
“The global pharma sector's forward earnings multiple has expanded to almost 16 times, up from less than 11 times two years' ago, in part because investors believe the (biotech) industry's R&D machine is again producing the goods.”
Thomas noted, however, that only one in 10 potential therapies entering clinical trials reaches the marketplace. “The risks are substantial,” she said. Those same risks apply as well to the 10 clinical trials that the California’s $3 billion stem cell agency has been involved in.

Earlier this year, noted bioethicist Art Caplan wrote about what he called the “off-the-rails syndrome” in stem cell research. The STAP article was his starting point.  Stem cell research is a field that has had more than its share of hype. Well-respected scientists routinely refer to its revolutionary potential. Little public attention is paid to the obstacles and the lengthy and often unsuccessful process of developing a truly usable product.  Expectations of desperate patients are raised. Many of them wind up paying for expensive, untested and perhaps unsafe treatments.

The Seragon-Roche deal is also a reflection of the hype that can arise in biotech/stem cell research. It can be so powerful that the supposedly “rational” economic markets are swept up in the exuberance of a nifty research story.  Ultimately the deal may pan out for Roche, although Roche can afford to take a big loss. But stories are stories.

What does all this mean for the California stem cell agency? Good reasons exist to manage expectations so that the public and potential sources of funding are surprised by successes rather than being surprised by the agency’s failures.  No one wants to see a story like the Solyndra scandal emerge from the California stem cell agency.

Friday, April 05, 2013

StemCells, Inc., Rejects $20 Million from California Stem Cell Agency

When does a financially struggling biotech company turn down a $20 million “forgivable loan?”

When it is StemCells, Inc., of Newark, Ca., and the cash is being offered by the $3 billion California stem cell agency. The research program has handed out nearly 600 awards, and it is the first time that a recipient has rejected funding.

That's the latest development in a stem cell saga that began publicly last July and that involved unusual personal lobbying by the former chairman of the Golden State's stem cell research agency. The high point of the saga may have come in September when the agency's governing board finished awarding StemCells, Inc., $40 million in two different awards. But there was a catch. StemCells Inc., had to match that figure with $40 million of its own.

Late last month, StemCells, Inc., threw in the towel on the $20 million awarded on its cervical spinal cord injury application. In comments to analysts March 21, Rodney Young, chief financial officer of the publicly traded company, said:
“The funding would have been in the form of a forgivable loan, however, we have elected not to borrow these funds from CIRM(the stem cell agency).”
According to the Seeking Alpha transcript of the conference call with analysts, Young said,
“You may also recall that last September, CIRM approved a separate application under the same disease team program for Alzheimer's disease, which was also for up to $20 million in the form of a loan. We remain in confidential negotiations with CIRM regarding the terms and conditions that would attach to this loan.”
The company provided no explanation for rejecting the cash, either in the conference call transcript or in its press release.

During the conference call, StemCells, Inc., reported continuing losses. For 2012, net losses totaled $28.5 million compared to $21.3 million in 2011. Revenue for 2012 was $1.4 million compared to $1.2 million in the previous year.

The awards last year to StemCells, Inc., founded by Stanford's eminent researcher Irv Weissman, stirred up a bit of a ruckus. The spinal injury award was handed out routinely in July. Scientific reviewers gave it a score of 79 and recommended funding. It was another matter on the Alzheimer's application. It was scored at 61. Reviewers said it did not merit funding. But the company publicly appealed to the full board, which sent the application back for more examination. It was rejected again. Nonetheless, in September, the 29-member board approved the award on a 7-5 vote, bypassing a rival Alzheimer's application scored at 63. It was the first time in the eight-year-history of the agency that its board approved an application that was rejected twice by reviewers.

Approval came only after strong lobbying by Robert Klein, former chairman of the board. Klein was also chairman of the ballot campaign that created the agency, and Weissman, who holds stock in StemCells, Inc., and sits on its board, was a major fundraiser for the campaign. 

The Los Angeles Times' Pulitzer Prize-winning columnist, Michael Hiltzik, wrote in October that  the process was “redolent of cronyism.” He said a “charmed relationship” existed among StemCells, Inc., its “powerful friends” and the stem cell agency.

As for the remaining $20 million award, Martin McGlynn, CEO of StemCells, Inc., expects “quick” action on finally securing the cash.

Here is an exchange that came during the March conference call between McGlynn and analyst Kaey Nakae of Ascendiant Capital Markets.
Nakae: “Okay. Just 2 more questions. I guess the first one, as it relates to CIRM. In deciding to decline the funding for spinal cord yet continuing to pursue the funding for Alzheimer's, is there a difference in what you're getting from them in terms of potential terms and conditions that allow you to proceed on one and not the other, or is it the fact that you're already in human with -- in spine, and still very preclinical with Alzheimer's?”
McGlynn: :”I think you're very definitely -- you're getting at some important criteria when one considers how to fund programs whether you use debt or equity, etcetera. So I wouldn't disagree with anything that you've outlined or surmised. But I just would pray your indulgence until we're finished, the negotiations with CIRM, which are coming to a close and we expect those to resolve pretty quickly with regards to the Alzheimer's program. And then quite frankly, we can be way more forthcoming and way more disclosive with regards not only to our decisions, but to our thinking.”
StemCells, Inc., was trading at about $1.65 at the time of this writing, down slightly from the previous day. Its 52-week high is $2.67 and its 52-week low $0.59.

Wednesday, October 10, 2012

Yamanaka and the Frailty of Peer Review

More than one back story exists on Shinya Yamanaka and his Nobel Prize, but one that has received little attention this week also raises questions about hoary practice of peer review and publication of research – not to mention the awarding of billions of dollars in taxpayer dollars.

The Yamanaka tale goes back to a 2010 article in the New Scientist magazine by Peter Aldous in which the publication examined more than 200 stem cell papers published from “2006 onwards.” The study showed an apparent favoritism towards U.S. scientists. Also specifically reported were long delays in publication of Yamanaka's papers, including in one case 295 days.

Here is part of what Aldous wrote,
“All's fair in love and war, they say, but science is supposed to obey more noble ideals. New findings are submitted for publication, the studies are farmed out to experts for objective 'peer review' and the best research appears promptly in the most prestigious journals. 
“Some stem cell biologists are crying foul, however. Last year(2009), 14 researchers in this notoriously competitive field wrote to leading journals complaining of "unreasonable or obstructive reviews". The result, they claimed, is that 'publication of truly original findings may be delayed or rejected.' 
“Triggered by this protest, New Scientist scrutinised the dynamics of publication in the most exciting and competitive area of stem cell research, in which cells are 'reprogrammed' to acquire the versatility of those of an early-stage embryo. In this fast-moving field, where a Nobel prize is arguably at stake, biologists are racing feverishly to publish their findings in top journals. 
“Our analysis of more than 200 research papers from 2006 onwards reveals that US-based scientists are enjoying a significant advantage, getting their papers published faster and in more prominent journals (find our data, methods and analyses here). 
“More mysterious, given his standing in the field, is why two of Yamanaka's papers were among the 10 with the longest lags. In the most delayed of all, Yamanaka reported that the tumour-suppressing gene p53 inhibits the formation of iPS cells. The paper took 295 days to be accepted. It was eventually published by Nature in August 2009 alongside four similar studies. 'Yamanaka's paper was submitted months before any of the others,' complains Austin Smith at the University of Cambridge, UK, who coordinated the letter sent to leading journals. 
“Yamanaka suggests that editors may be less excited by papers from non-US scientists, but may change their minds when they receive similar work from leading labs in the US. In this case, Hochedlinger submitted a paper similar to Yamanaka's, but nearly six months after him. Ritu Dhand, Nature's chief biology editor, says that each paper is assessed on its own merits. Hochedlinger says he was unaware of Yamanaka's research on p53 before publication.”
Last week, Paul Knoepfler of UC Davis wrote of other issues dealing with peer review, but coincidentally also dealing with iPS cells. What New Scientist and Knoepfler are discussing is not an isolated situation. It is part of a continuum of complaints, both serious and self-interested but exceedingly pervasive. A Google search today on the term “problems with peer review” turned up 10.1 million references.  Writing on Ars Technica last year, Jonathan Gitlin, science policy analyst at the National Human Genome Research Institute,  summarized many of the issues, citing a “published” (our quotation marks) study that said peer review doesn't work “any better than chance.” Gitlin said,
“A common criticism is that peer review is biased towards well-established research groups and the scientific status quo. Reviewers are unwilling to reject papers from big names in their fields out of fear, and they can be hostile to ideas that challenge their own, even if the supporting data is good. Unscrupulous reviewers can reject papers and then quickly publish similar work themselves.” 
At the $3 billion California stem cell agency, peer review is undergoing some modest, indirect examination nowadays. The agency is moving towards tighter scrutiny of budgets proposed by applicants. And, following a record wave of appeals this summer by disgruntled applicants rejected during peer review, it is also moving to bring the appeal process under more control.

As the agency tries to move faster and more successfully towards development of commercial therapies, it may do well to consider also the frailties of its peer review process and the perils of scientific orthodoxy.   

Wednesday, July 25, 2012

The Harsh Message at the California Stem Cell Agency

Grant reviewers have delivered a harsh message in the latest $243 million research round at the California stem cell agency – at least that is one way to look at it.

In effect, they told the governing board of the $3 billion enterprise that the overwhelming majority of applicants in its signature disease team round do not measure up, despite the fact that CIRM had early on partially vetted their efforts. Indeed, the reviewers said that the researchers deserve only $113 million instead of the full $243 million that was budgeted.

Obviously the results of the review can be interpreted in other ways as well. But the review outcome should raise some flags within the stem cell agency and its 29-member board, which meets tomorrow in Burlingame. It may not auger well for future rounds that also involve CIRM's newly energized drive to push research into the clinic.   

One interpretation of the review results could well be that CIRM's goals are unrealistic, that the agency is trying to move too fast for the normally glacial pace of research and development. Another interpretation is that the science is not good enough in California to accomplish what the agency is seeking to do, a view expressed by some in the early days of the nearly 9-year-old program. Another is that the reviewers themselves don't know enough or have failed to do their homework, which some of the rejected applicants have argued in their appeals. Yet another is that the CIRM review process is inadequate to the task of meeting CIRM's goals. And still another interpretation is that the normal peer review process on which CIRM's procedures are based is mightily flawed, a general contention argued by some(See here, here and here.)

Or quite possibly the result of the disease team reviews could reflect a combination of all of the above, to one degree or another.

Little is known about the substance of what goes on during the grant review process, aside from the staff-written review summaries. Even CIRM board members, who see only the summaries, have complained from time to time about not having enough information to make a good judgment on an application. Reviews are conducted behind closed doors. Information about the economic and professional interests of reviewers is withheld from the public by the stem cell agency.  

Here is a look, however, at what we do know. Initially the universe of applicants in this round totalled 36. That was the number that applied for planning grants for this round. Without a planning grant, they could not apply for a full $20 million award, with some exceptions. The exception process was controlled by CIRM President Alan Trounson, not reviewers. CIRM used the planning grants and the exception process not only to assist applicants but to winnow out weak applications.

Nineteen researchers won planning awards. With exceptions included and minus dropouts, 22 applied later for the big money. Out of the 22, only six were recommended for funding by reviewers, who are known more or less formally as the Grants Working Group. (See the four items at the end of this piece for a list of reviewers involved.)

In the past, reviewers have sometimes not approved sufficient applications to consume the entire amount budgeted for a round. But they have never produced a shortfall as great as in this case. It is all the more dramatic since this round carries a lot of weight for CIRM, which is pushing hard to commercialize research and fulfill at least part of the promises that were made to California voters in 2004 to win approval of creation of the stem cell agency.

One reflection of the unusual nature of the round is the record pace of researchers' appeals of negative decisions by reviewers. At least nine of the 15 rejected scientists are willing to say publicly that something is is not quite right in the review process, ranging from missing facts to inconsistencies in CIRM's endorsement of particular paths of research.

It is safe to say that CIRM directors tomorrow will pluck some applications out of the reject bin and increase the total awarded. But they should also examine the process to determine what generated this particular outcome. The Institute of Medicine, which is currently engaged in a $700,000 examination of CIRM, also might scrutinize this round with some care, given its size and importance to the California stem cell research effort.

Wednesday, August 10, 2011

Faulty Research, Fraud Attract Page One Attention in Wall Street Journal

The Wall Street Journal today reported a sharp upturn in retractions of scientific studies and a trend in research fraud that one editor of an influential scientific journal called a "scar on the moral body of science."

The journal's page one piece was headlined, "Mistakes in Scientific Studies Surge."

The article documented instances in which hundreds of thousands of persons were treated on the basis of faulty research.

Reporter Gautam Naik wrote,
"Science is based on trust, and most researchers accept findings published in peer-reviewed journals. The studies spur others to embark on related avenues of research, so if one paper is later found to be tainted, an entire edifice of work comes into doubt. Millions of dollars' worth of private and government funding may go to waste, and, in the case of medical science, patients can be put at risk."
The Journal piece has implications for the California stem cell agency, which ballyhoos the number of papers its grant recipients publish -- some 600 or so from $1.3 billion in grants and loans. The agency also has established critical bench marks for performance that must be verified on some big-ticket grants if state money is to continue to flow to recipients. And CIRM grants are awarded on the basis of a peer review process.

Naik wrote,
"Since 2001, while the number of papers published in research journals has risen 44%, the number retracted has leapt more than 15-fold, data compiled for The Wall Street Journal by Thomson Reuters reveal.

"Just 22 retraction notices appeared in 2001, but 139 in 2006 and 339 last year. Through seven months of this year, there have been 210, according to Thomson Reuters Web of Science, an index of 11,600 peer-reviewed journals world-wide."
The article said that retractions related to fraud increased more than sevenfold between 2004 and 2009, according to an analysis in the Journal of Medical Ethics, a finding also supported by another researcher.
Naik wrote,
"Why the backpedaling on more and more scientific research? Some scientific journals argue that the increase could indicate the journals have become better at detecting errors. They point to how software has made it easier to uncover plagiarism.

"Others claim to find the cause in a more competitive landscape, both for the growing numbers of working scientific researchers who want to publish to advance their careers, and for research journals themselves.

"'The stakes are so high,' said the Lancet's editor, Richard Horton. 'A single paper in Lancet and you get your chair and you get your money. It's your passport to success.'"
Naik continued, "The apparent rise in scientific fraud, said Dr. Horton 'is a scar on the moral body of science.'"

By about 9 a.m. PDT, the article had attracted more than 200 comments on the Journal's web site. Some readers minimized the problem. Another reader said that to be a credible scientst,
"One must jump (through) many hoops in conforming to proper authorities."
Another commented about problems created by agenda-driven results. Another criticized the peer review process as simply an inexpensive way for scientific journals to fill their pages. Reader Daniel Viel said, "Bottom line: this is big business. A tremendous amount of money involved every time a study 'pushes' drugs."

Saturday, June 18, 2011

CIRM Posts Details of Change in Big Ticket Grant Reviews

The California stem cell agency has added substantial amounts of additional information to its agenda for its board meeting next week, including a proposal that will make significant changes in the reviews for its important clinical trials and disease team rounds.

The additional material was posted late yesterday, only two business days before the meeting that begins next week in San Diego. As we noted yesterday, the dilatory postings do not serve the public, the California stem cell community, the agency or its directors well. One CIRM director, Joan Samuelson,  says laggard postings at CIRM are "crazy" and unprofessional.

"It bugs me," she told other directors earlier this month.

The new review procedures would involve grants or loans in the roughly $20 million and up range. They would provide an opportunity for applicants to comment on reviewer questions in advance of the formal peer review meeting. They would also allow reviewers to question applicants by telephone on the day of the review. The procedures grew out dissatisfaction by some board members with the process in which Geron was loaned $25 million in May.

The Geron funding represents CIRM's first entry into clinical trials. The agency is expected to be involved in more clinical trials as it tries to push stem cell therapies into the marketplace. The agency is seeking to produce results that would persuade voters to approve another bond measure – perhaps as large as $5 billion – to continue CIRM's efforts.

Ellen Feigal, CIRM's new vice president for research and development, told a panel of CIRM directors earlier this month that the new procedure is aimed at "making sure we have complete information in real time about the research grant." She said the changes are designed "to make sure that the investigators, the applicant, actually feel that they have an opportunity to address some of the difficult questions that could arise during the review process."

Her comments were made at the directors' Science Subcommittee meeting June 6. The transcript of the session, which is worthy of review by all potential CIRM applicants, was released two days ago with little notice, which is standard procedure for CIRM.

The review changes call for scientific reviewers to complete their initial review of applications 14 days in advance of a review meeting along with a list of key questions. The questions, plus any additional issues that CIRM staff has, would be go to applicants 10 days ahead of the review to give them time to respond. Applicants would be asked to be available by phone on the day of the review for any additional questions.

The discussion at the committee meeting largely involved details in execution of the plan and whether reviewers would pose questions directly to applicants. The answer was no, that some other person would pose the questions to avoid possible identification of scientific reviewers, whose identities are kept secret by CIRM.

Also discussed was the nature of the questions – whether they would involve nuanced matters or matters that could be addressed with a yes or no. From the committee's discussion, it was clear that this initial effort will be a testing ground to work out exactly how the new procedure will be implemented.

Other information freshly available on the agenda for next week's meeting involves:

  • An $80 million grant program, beginning next year, for physician scientists to conduct research that will translate into possible applications.
  • A $30 million "opportunity fund," controlled initially by the CIRM president, to accelerate development of therapies and implement a recommendation of last fall's blue-ribbon external review panel and assist industry.
  • A $27 million, three-year extension of a training program involving California state colleges(but not the University of California) and community colleges.
  • A job description for CIRM's new director of public communications, who apparently would run the agency's PR efforts from the chair's office as opposed to the president's. The post is part of a reorganization of CIRM management.
(Editor's note: An earlier version of this item did not contain the comments from Joan Samuelson.)

Search This Blog