I read an interesting article recently that stated that life science venture capital agencies (such as AmorChem!) consider essentially half of all published research to be unverifiable. While that may initially appear to be a radical (if not outright controversial) point of view, it gained further traction after the announcement by Amgen that they could only reproduce 6 out of 53 supposedly landmark studies on cancer. This further underlined the outcome of an earlier analysis by Bayer, wherein they were only able to repeat the results published in about 25% of some 67 "hot" papers.
One can hear not only venture capitalists asking what the hell is going on, but the entire scientific community, if not a cynical general public also. That public is subject to the rigorousness and validity (or lack thereof) of major new discoveries, ultimately in clinical trials and at the pharmacy, after that. The fact that an estimated 80,000 patients (2000-2010) were involved in clinical trials based on research papers that were later retracted is a worrying testament to how the public can be impacted by shoddy science.
So, what the hell is going on? Irreproducibility is the thorn in the side of modern science and medicine, and clearly for such complex matters involving various layers of personnel and scrutiny, there are many possible explanations. At the extreme end there are examples of outright fraud; this does not only mean playing with the data, but has gone as far as egregious manipulation of the peer review process itself such that in certain exposed cases the senior author got to review their own paper - as inconceivable as that seems to a typical scientist. I am not going to try to define what causes a scientist to commit any form of fraud, but you can be pretty sure that "publish or perish", tenure, fierce competition for funds and positions, the clear need to be first, and mere survival are all in the mix somewhere.
Not to be confused with the satirical take on the emerging reproducibility problem in science, "The Journal of Irreproducible Results", a certain Dr. Ivan Oransky has founded an entity known as "Retraction Watch" whose goal it is to monitor the retraction of scientific publications very closely, and to dig into the dirt and expose the darker side and slimy underbelly of contemporary scientific research. The fact that they recently received a substantial grant from the MacArthur Foundation emphasises that this issue is being taken seriously and support is growing for a closer monitoring of the scientific process.
Reproducibility is in fact a bit of a dirty word, because often one does not want a competitor being able to repeat something one has done, and then racing ahead like an express train. In case it needs to be said, scientists don't publish to enable anyone else to do what they have done, they publish because they have to, to survive, and because they usually want to be (more) famous. At the risk of bursting anyone's bubble, academic scientists as a pack are more interested in being at the front, leading the pack, and getting the fame that comes with that, than they are in curing a disease. For those trying to get into the club, it's all about getting the papers you need to get a job; rarely anything deeper. But it can become darker, when that need becomes more of a priority than the science itself - even if that's just the reality of life for many trainee scientists who are trying to "make it".
Reproducibility is in fact a bit of a dirty word, because often one does not want a competitor being able to repeat something one has done, and then racing ahead like an express train. In case it needs to be said, scientists don't publish to enable anyone else to do what they have done, they publish because they have to, to survive, and because they usually want to be (more) famous. At the risk of bursting anyone's bubble, academic scientists as a pack are more interested in being at the front, leading the pack, and getting the fame that comes with that, than they are in curing a disease. For those trying to get into the club, it's all about getting the papers you need to get a job; rarely anything deeper. But it can become darker, when that need becomes more of a priority than the science itself - even if that's just the reality of life for many trainee scientists who are trying to "make it".
There are a lot of moving parts in a typical research paper, often involving a combination of technicians, research assistants, graduate students, postdocs, and of course, the lab director as the obergruppenfuhrer-in-chief. While it is largely believed that the head of the lab oversees all aspects of the process from lab bench to published paper, anyone who has ever worked in even a medium-sized lab knows just how distant that oversight can be on a weekly or even monthly basis. A senior postdoc charged with "keeping an eye" on the kids is often too busy grinding out their own experiments to monitor the activities of anyone, and are often daily-obsessed with their own urgent need to get a job to get the hell out of there.
In a typical laboratory, one is by and large left to one's own devices to get the desired data, present it at lab meetings, then compile it into a nice little story - which is where the lab head usually steps in and polishes that story into his/her version of acceptable-for-publication. The entire process only works if the trust that cements it all together is solid and does not have cracks in it due to any one piece of the puzzle not being trustworthy. As precarious as this all may be, I feel in general it works surprisingly well, if we were trained correctly as young scientists in the first place.
Having spent the greater part of my adult life in life science laboratories in four different countries, it has always been apparent where the majority (i.e. those not derived from fraud) of irreproducible results come from - cultured cells. They are the very workhorse of the modern biology laboratory, and are used by essentially everyone as little models of the human condition. There are also very many variables in the use of the array of cell types at our disposal as biologists, such that I am pretty certain that HeLa cells or 293 cells or COS cells in two labs even in the same institute are unlikely to be identical.
It's the things that don't go into published papers that make all the difference. Not only do the sources and grades of various chemicals often play major roles in cultured cells' response to them, but the precise practices of handling cultured cells is rarely as published. Typically, when one is publishing a paper involving the use of given cell types commonly used in that lab, and one gets to the dreaded "Materials and Methods" section of the paper draft, what most seem to do is go to the last paper published (by someone else) and simply copy/paste the cell culture details as published. If it's published, then it has to be acceptable, right?!
In this case, it is a "right!" because that culture protocol is what is written in the cell culture bible. However, that is rarely what is practised. Busy people in busy labs rarely seem to rigorously pass their cells at the same confluency each time, will continue to use them even when they overgrew, because if no one else has them in culture it means starting from scratch again and the boss won't be happy with the delay. Ditto when someone leaves trypsin on them for too long, while chatting on the phone. Additionally, when cells are "behaving" nicely, and cranking out the data you want/like, why worry that they have reached an upper passage number that dictates scrapping them and thawing out a new stock?
Cells are "abused" very regularly, and even differences in serum batches and media preparation can play havoc with the true phenotype of a cell referred to by some generic name. It's a huge problem, and one that is almost impossible to police, or at least impossible in the current scientific system. The issue was discussed recently in a new panel put together by the Global Biological Standards Institute in November that focused on cell line authentication - a central factor in a 2013 white paper by the Institute that underlined the need for new standards in the wide arena of preclinical research. Missidentification of cells prior to or during use is rife and therein lies the root cause of irreproducibility issues in modern research.
Cells are "abused" very regularly, and even differences in serum batches and media preparation can play havoc with the true phenotype of a cell referred to by some generic name. It's a huge problem, and one that is almost impossible to police, or at least impossible in the current scientific system. The issue was discussed recently in a new panel put together by the Global Biological Standards Institute in November that focused on cell line authentication - a central factor in a 2013 white paper by the Institute that underlined the need for new standards in the wide arena of preclinical research. Missidentification of cells prior to or during use is rife and therein lies the root cause of irreproducibility issues in modern research.
While the panel all agreed on that, there was less alignment on who should police the issues. One cannot really ask the journals to do it, not least given the fact that the majority of their editors and reviewers (think, peer reveiew) have day jobs, and their in-house editors are swamped with just handling the volume of papers hitting their desks. One can ask the universities and thus their lab directors to do it, but too much policing will definitely impact the freedom that is academia, and established, famous scientists are not going to take close scrutiny by their own university lying down. There will be a definite resistance if the checking gets too intrusive or burdensome.
I think a variety of stakeholders will have to come together and get new policies and standards introduced into the protocols and practices of using cultured cells, and the means to assess they were adhered to in work that is to be published. The future of scientific research probably depends on it. But it's worth pointing out that a paper containing cellular data that cannot be reproduced is not fraudulent when (as is usually the case) the data was obtained honestly. It may be bad science, or misleading science, but it is not fraud. Fraud is much more insidious and harder to identify/correct, but bad science can be bettered by introducing new policies and standards to which one and all must adhere. Stakeholders will have to introduce an "amnesty" for work previously published, and only investigate work that occurred following the introduction of any such new standards.
All this fuss over cells and cell culture has been simmering for a long time, and it has been placed on the backburner for many years, but it's beginning to boil over now, and we are all going to have to face it in one capacity or another. We simply must get to a point where F9 cells in a lab here in Montreal are the same F9 cells genotypically/phenotypically as those in a lab in Hamburg. Suddenly, reproducibility will be more the norm, rather than today's irreproducibility. There will be initial resistance and some big hurdles to get over, but if we can get cultured cells to fall into line (i.e. cell lines!) all over the world, then the culture of science is going to experience a major advancement as a result!
No comments:
Post a Comment