Again for the little its worth, as far as I’ve heard the OSI has been quite well run lately. At the time of the Baltimore/Imanishi-Kari case it seemed that the OSI was heavy-handed, highly politicized, and frankly incompetent; but since then I’ve heard very little criticism of it (and I believe, but can’t quickly find eviudence that, the OSI was reorganized after and becaase of the Baltimore case).

That would be a really good thing to have in curricula. We learn some pretty strange things in high school. Why did I learn about conic sections, catenaries, and cardioids in the standard math sequence instead of statistics, which was an elective I didn’t take? Stats is as enlightening and mind-strengthening as finding volumes of rotated curves – plus extremely useful, whereas many of those other things are almost completely academic.

Likewise, why so many obscure integration techniques like integration by parts and VU integration. They didn’t do that much for the mind since we didn’t learn the proofs which connect them to the rest of the mathematical world – just used them to grind out problems. Much better would be a little real analysis, which is an epitome of a very navigable and repletely interconnected intellectual edifice – it’s much like geometry in that way, but it is a more potent body of thought.

It’s hard to imagine how this could be done more simply than just making people’s lab notebooks available, provided that that would in fact amount to a fairly effective deterrent. I would think it would be a pretty large pain to do phony stuff in a way that made it look like you were gradually and progressively “discovering” it, refining your experiments, etc.

Holding in mind real empirical human nature rather than ideals, even this simple scheme is a recipe for mire IMO. The overseers have to justify their existence, and also compete with one another for importance. They are going to end up always hassling a certain more or less set percentage of the overseen – if there’s no fraud to be found, they will hassle X percent of people about (relatively) poor transparency in their lab notes, more or less regardless of whether anyone is poorly transparent absoluter. Result = a huge pain. Transparency standards will be “by committee” and will always be at least slightly oblique to the goals of lab notes – recording stuff in so it will very be clear to you and other people in the lab, with the method of clear expression highly dependent on what the content is and what your way of thinking is. Bearing in mind also that, IME at least, notes are often worked on in a short piece of downtime while you monitor one or two time-sensitive processes going on on the bench, which you are simultaneously contemplating how to clean up super fast in time for some seminar. Biomed is no serene business.

The problem, as I said, has been more about the non-scientific people who are taken in by fraud, and who force scientists to repeatedly disprove what they already know is false; and I don’t think an audit procedure would help there.

Where audits might help is if they could catch the high-profile frauds before they reach the public consciousness and start to poison the well. But how audits could quickly focus on problematic research, without themselves poisoning innovation … that I don’t know.

The ideal situation would be for people in the street to have a better awareness of how science works — the simple fact that the vast majority of bold new hypotheses are wrong, that the lone voice refuting dogma is almost always mistaken, that most new findings disappear into the rubbish heap of irreproducibility — and then they might not be so eager to see conspiracies so quickly.

But I think a simple non-bureaucratic audit might be easier to implement than that.

I agree with the contention that the amount of untrue science will always be large and fraud will always be an insignificant contributor. For a radical perspective see Ioannidis’ much-read “Why Most Published Research Findings Are False” in PLoS Medicine.

I’m not saying we should not be outraged by fraud at all. We should continue to punish it harshly if it comes to light.

On the other hand, I don”t think I would like the idea of scientific auditing. Most likely, the most immediate effect would be to make scientific research much more bureaucratic.

I think that science can live well with the way things are currently organized – as long as people don’t put too much trust in results that have been found only by a single group. Nothing beats independent validation by a competing group.

The most immediate analogue i can think of is the Bernie Madoff scandal. His auditor noted that there is very little you can do if fake books (or in this case fake data) are being used. Granted, that’s the auditor’s job, to crunch through fake books, but I suppose I’m curious what would fall under the audit.

To the point about faulty controls — at what point is that “scientific editorializing” (what constitutes as a sufficient control, or a sufficiently good study) and at what point is the audit not going far enough?

Would like to get your thoughts, as I have no clear answers in my mind.