Just because it is published, doesn’t mean it is reproducible…

February 1, 2011

This lecture, by Keith Baggerly from MD Anderson Cancer Center, is probably the most interesting lecture on statistics that I have ever seen. Like CSI but with research data, uncovering incredible errors in analysis that were actually being used to select treatments for patients in a clinical trial.  Watching it gave me something to think about when we look at new technologies to use as a basis for a startup and how important it is that researchers (and we) are willing to double-check their intuition with rigorous, objective analysis. In the case presented in the lecture, it was fraud or incompetence or both but even with the best of intentions, we tend to accept published information as fact too easily (especially if it suits our intuition).

Usually, reproducibility problems that we stumble on are honest manifestations of the differences between expectations in the lab and the outside, commercial world.  For instance, having been a graduate student, I know that the student may need to make an experiment work only one out of ten times in order to get enough data to graduate and that might still be great science.  However, when you translate the science from the lab to commercialization, things need to work all the time.  One startup of mine spent 2-3 years (unplanned) trying to replicate the grad-student manufacturing techniques.  We thought the students were able to make the material correctly every time, but it turns out that, if a batch didn’t work, it got tossed – no need to report it – still good science because their research was not on repeatable manufacturing techniques but on the properties of the material.  It may be unfair to mention that error in the same post as this lecture because, in my case, the students had done exactly what they should have done.  The error was mine because I followed my intuition that the process was easy to reproduce.  Wrong.  What I should have done is run a rigorous experiment early in the life of the company to determine reproducibility and then used that milestone to decide how to proceed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: