

The problem with science

Science is a great system. You examine reality, come up with ideas how it might work, test those ideas, keep the good ones, discard the bad ones, and move on. It’s got one big flaw, though, and that is that science is done by scientists, and scientists are people.
I have a whole slew of scientists mad at me this week – and I will admit that I am pretty irritated back – because none of us cool rational analytical scientists can truly separate our emotions and our egos from the reality-based science that we do. In this current dispute, I get to claim the scientific high ground, at least. My scientific paper that just came out this week unarguably demonstrates that their scientific paper has some rather embarrassing errors. But, in the end, I suspect that even with that seemingly unassailable high ground, I lose the war.
The papers in question are both on the mundane side. They both are catalogs of where the Cassini spacecraft has and hasn’t seen clouds on Titan over the past 4 years. Papers like these, though not going to make headlines anywhere, are nonetheless important contributions to understanding what is going on (at least I think so, or I wouldn’t have taken the time to write one!). Without complete and accurate catalogs of things like where there are clouds on Titan, we cannot begin to understand the more profound questions of why there are clouds on Titan and what does this tell us about the hydrological cycle on the moon. These papers don’t try to answer these questions, but they are necessary pieces of the puzzle.
You would think that two papers that examine the same set of pictures from the Cassini spacecraft to map clouds on Titan would come up with the same answers, but they don’t. And therein lies the root of the problem. When the main topic of a paper is where there are and aren’t clouds on Titan and you sometimes say there are clouds when there aren’t and there aren’t clouds when there are, well, then you have a problem. They have a problem, since theirs is the paper that makes the mistakes. So why are they mad at me? I think perhaps I know the answer, and, perhaps I even think they might have some justification. Let me see if I can sort it out with a little of the convoluted history.
I started writing my paper about 18 months ago. A few months later I realized the other team was writing the exact same paper. Rather than write two identical papers, I joined there team and the two papers merged. The problem was that as I worked with their team through the summer, it became clear that their analysis was not very reliable. I spent hours going over pictures in details showing them spots where there were or were not clouds in contradiction to their analysis. Finally I came to the conclusion that their method of finding clouds and thus their overall paper was unsalvageable. I politely withdrew my name from their paper and explained my reasons why in detail to the senior members of the team overseeing the paper. I then invited them to join me in my analysis done in a demonstratively more accurate way. The senior member of the team agreed that it seemed unlikely that their method was going to work and he said they would discuss and get back to me.
I felt pretty good about this. I had saved a team of people who I genuinely liked from writing a paper which would be an embarrassment to them, and I had done it – apparently – without alienating anyone. I remember at the end of the summer being proud of how adeptly I had navigated a potentially thorny field and come out with good science and good colleagues intact. Scientists are usually not so good at this sort of thing, so I was extra pleased.
I never did hear back from them about joining with me, so when I wanted to present the results of the analysis at a conference in December, I contacted the team again and asked them if they would like to be co-authors on my presentation in preparation for writing up the paper. I was told, no, they had decided to do the paper on their own. Oh oh. I though. Maybe things won’t end up so rosy after all.
Their paper came out first, in June of this year, in the prestigious journal Nature of all places (it’s not hard to figure out the reason for the catty comment often heard in the hallways “Just because it’s in Nature doesn’t necessarily mean that it is wrong.”). I was a bit shocked to see it; I think I had really not believed they would go ahead with such a flawed analysis after they had been shown so clearly how flawed it was (and don’t get me started about refereeing at this point). Our paper came out only this week, but, since their paper was already published, one of the referees asked us to compare and comment on their paper. I had avoided reading their paper until then, I will admit, because I didn’t want to bias our own paper by knowing what their conclusions were and because – I will also admit – I was pretty shocked that they had, to my mind, rushed out a paper that they knew to be wrong simply to beat me to publishing something. I hoped that perhaps they had figured out a way to correct their analysis, but when I read their paper and found most of the erroneous cloud detections and non-detections still there, I realized it was simply the same paper as before, known flaws and all.
So what did I do? In my paper I wrote one of the most direct statements you will ever read that someone else’s paper contains errors. Often things like that are said in couched terms to soften the blow, but, feeling like they had published something that they knew to be wrong, I felt a more direct statement in order.
And now they’re mad.
Reading all of that I certainly hope you come to the conclusion that I am 100% right and they are 100% wrong. You’re supposed to come to that conclusion because I wrote the whole thing from my own biased perspective. And I have my emotions and my ego in there. And I feel wronged.
I’m going to try an experiment from their point of view and see if I can see where I went wrong and irritated them.
Last summer they kindly invited me to be part of their paper, and they shared their non-publicly released data with me (though neither analysis made use of it). They fixed many of the errors that I identified that summer and honestly believed the paper was now good enough. They knew that the analysis wasn’t perfect, but felt like they had invested significant resources in the analysis and that the overall conclusions were correct. So they submitted the paper, and it got accepted in Nature, and they were pretty proud of the effort. Then, out of the blue, my paper is published that says in unusually direct words that their paper is not to be trusted.
Here are some reactions I can guess that they might have had:
(1) Mike Brown’s complaints about are paper are simply sour grapes because our paper came out first and in a more prestigious journal. He is trying to attack our paper so that his paper, which lost the race, somehow seems relevant.
(2) Mike Brown is a nit picker. If you look carefully you will find that while the details of the cloud maps are different between the two papers, the overall conclusions are largely the same. In the end, the conclusions matter, not the details like this.
(3) Mike Brown is a betrayer. He learned about our analysis last summer and then tried to use what he learned against us.
(4) Mike Brown is an impolitic ass, and even if he had concerns about the paper he aired them in an unkind way and now we detest him.
And now I must in the end admit that one of those is actually true. I plead guilty to (4). (1) and (3) are factually incorrect. (2) is bad science (yes: the details matter, not just the conclusions). But (4)? Yeah. OK. Probably. That’s the problem with science. All of those scientists. And few scientists are renowned for their social skills. Even me.
So there are some things that we can all agree with, and some things that we might disagree with. Reality admits little room for differences of opinion. Interpretation of reality, though, is always more subjective.
Everyone should agree: The paper that was published in Nature this June is at times incorrect about where there are and are not clouds. This is simply reality and not open to much discussion (which doesn’t mean there won’t be much discussion).
In my opinion: These errors are fatal for a paper purporting to be about where there are and are not clouds. In their opinion: These errors are not significant and don’t affect the conclusion of the paper. In my opinion my opinion is correct, but I am sure that in their opinion their opinion is correct. Unlikely we’ll come to a conclusion on this one, as this is not about reality, but about interpretation of reality. No analysis is 100% correct and everyone has their own opinion about when an analysis crosses the threshold from mostly correct to fatally incorrect. We have differences of opinions on where this threshold sits, obviously.
In their opinion: The statements in my paper discussing the problems with their paper are disproportionately harsh. In my opinion: The statements in my paper discussing the problems with their paper are harsh, but proportionate to the flaws in the paper. But I will admit that this is the part I am the most uncomfortable with. The statements in my paper are harsh. Maybe too harsh. Did I let too much emotion and pride come in to play as I wrote them? Probably. But as I wrote those statements I was fairly appalled at what seemed to me a lack of concern with reality on the part of their paper. Everyone makes mistakes in scientific papers. Sometimes even big ones. But I had never come across a paper where the mistakes were pointed out before the paper was submitted for publication and the authors had not fixed them. Again, though, my opinion is colored by the fact that I find their analysis fatally flawed. Their desire to go ahead is colored by the fact that they find their analysis good enough.
In their opinion: Mike Brown is a detestable ass. In my opinion: They are shooting the messenger for delivering a message that they already knew. But perhaps both opinions are correct.
Sadly, for me at least, I tried really really really hard to make this work. And to me, “make this work” meant make sure that any papers published which described clouds on Titan were factually correct while at the same time not alienating my colleagues. I failed at both.
So I think we end with this:
The other team will probably always think I crossed a line by writing so harshly of their paper. I will probably always think that they crossed a line by publishing a paper they knew to have factual errors.
Who is right? Probably both. I suspect they let their egos and emotions allow them to care more about publishing a paper in Nature than whether or not that paper was correct. I suspect my ego and emotions caused me to write more harshly than I needed to. That’s the problem with science. It’s done by scientists. Scientists have all of those egos and emotions just like everyone else and no one has figured out a way to leave them at the door when you walk in your lab or your telescope or wherever you sit down to write papers.
In the end though, the only losers in this process are the scientists themselves. While all of us are sitting around feeling wronged, reality marches on. If you would like to know where clouds are or are not you can go read an accurate account. But that’s probably the last paper you will read from me in this field, for I am bowing out. The study of Titan was always just my hobby. A hobby that causes this much anguish is not a very good hobby. Time for a new one. I’ll miss Titan and trying to finally figure out what is going on with all of those clouds, but there are many other interesting things out there in the universe. Time to start exploring once again.


  1. Mike - Wow, what a train wreck. Is there a simple explanation why it's difficult to do this analysis?

  2. Sadly, it's NOT difficult! the big mistake the other guys made was to try to do it in some intelligent automated computerized way with no sanity checks. We went for the dumb look-at-every-picture-Cassini-has-taken approach and simply found the clouds. Dumber is sometimes better.

  3. I have a colleague whose motto regarding publications is that "It is better to be first - and to be wrong." That way you get more citations, and we all know that citations are the standard by which a paper is judged. I'm willing to bet you find that philosophy somewhat... distasteful?

    Looking on the bright side, I doubt that you'll upset too many school children about this particular icy body.

  4. Aww crap I waited too long to email -- I was going to ask if you need/wanted a volunteer, I'm in LA and found your posts and papers(at least the ones you put up here and on arxiv) fascinating and I was hoping there was some way that I could help. Well may be there's some other thing I can help you explore that isn't named Titan, but still has interesting weather. Expect an email/letter soon.

    -Brent Markus

  5. Aah, so THAT's what the drama was all about - and I had feared, following your tweets over the weekend, that you had now proven that Mercury wasn't a planet either ... :-)

    Where is your new paper published? I couldn't find it via either ADS or arXiv (and your homepage lacks a list of publications, it seems).

    Even for those not exactly interested in details of Titanian meteorology the pair of paper might make quite interesting reading at some meta-level, given the different approaches.

  6. I think you did the right thing by talking frankly about it in your paper. Particularly if you'd discussed your concerns with them before! It's the least ass-like approach IMO. This is how science works isn't it?


    Do you live with a bag over your head?

    That shit happens all the time. And it can't be your environment, because I know at least one person who has done that before going on to a job in your department.

    If you don't correct these mistakes, who will? A grad student who puts their life on hold and lives in poverty for four years only to find the database on which their research is based is flawed? If you flee Titan for the Kupier belt, who is gonna keep the bastards honest?

  8. Thanks for this.

    I wonder, though, how you would advise a person in your position who has not achieved the career status that you have. In a way, you are free to be bold like this and defend the purity of science, perhaps burning a few bridges along the way. But what would happen if a postdoc did something like this and pissed off a prominent scientist in an important group. Wouldn't that be career suicide? At the very least it would shrink the pool of future potential collaborators and jobs.
    What advice, if any, do you have for more junior scientists who are faced with the dilemma of the integrity of science versus the urgency to publish?

  9. Eliat --
    You're absolutely right. I can get away with burning bridges much more than I could 15 years ago, for example. My advice to a student or postdoc: don't do it. In the end you will be punished for pointing out major mistakes of others and it could take years (or a field change) to recover. Sad, cynical advice, yes, but that, again, is the problem with science.

  10. My recommendation is just the opposite. How can you justify doing something yourself and then tell others, just because they may have less career recognition than you, not to do it? Grad students and post docs shouldn't just go along with nonsense that passes as science. I say, burn the damn bridges. If you know you're right, stick with your position and you will be vindicated with time and more data. It would not be career suicide. What it would be is courage. As a grad student, I refuse to live by some double standard where you can get away with something while I can't. You're asking people to give in to intimidation and that's just plain wrong.

  11. Raven --
    I'm with you. But I would also say be sure to choose your fights. Retreating one day to win the larger war is not necessarily unwise. That doesn't mean ignoring the problem. There are other ways of dealing with erroneous papers, such as simply never citing them and not considering the erroneous results as correct.

    Myself, I've always been a bridge burner, but I consider that a personality flaw rather than much of a virtue!

  12. I kind of agree with Raven. I'd advise anyone junior to talk it though with a postdoc or advisor first, to see if they at least agree with you. But if you know you're in the right, and otehrs agree, then why shut up? While some might disagree with you, or think you're an arrogant arse, most people will respect you for it. As long as you're polite and professional about it you'll be fine.

  13. Mike,
    Could it be that you have let your heart overrule your head here? Retinr to scinece, and see if the different lists of Titanic cloud sites allow different conclusions to be drawn. Do testable predictions arise from those, that can discredit the other list, or dare I say it, yours?

  14. JohnD --
    That is my point exactly; of course my heart is overruling my head (or at least overruling my sense of politeness). In this particular case, though, it is not a question of making predictions to see which list is correct. It's simply a matter of looking at the actual data from the past and seeing which is correct. There, at least, there is no dispute. Their list contains errors. The question then comes: how much does that matter to the paper? That's where the difference in opinion lies.

  15. Unfortunately, going back to the Kuiper Belt may not end the headaches. Latest word is there is are several upstart "planet hunters" looking for objects out there and they're no easier to deal with than the Titan people. In fact, they may be even more petty since some of them are hard core Plutophiles. Looks like going from the frying pan into the fire.

  16. Hurray for upstart planet hunters! That could only be a good thing!

  17. Thanks for sharing this story. I had a couple of similar experiences with yours regarding science publications, though I work in a different sub-field of astronomy (extrasolar planets). I once refereed a paper for ApJ, for which a "detailed" mistake result in totally flawed conclusion. The authors withdrew the paper by actually agreeing with my report. Then they published their paper essentially unchanged in another journal!

  18. Nice piece! Seems to me you put your finger on a most important point. One of the problems with science as a way of knowing is a frailty in the human spirit—old testament mixed with modern empiricism.

  19. Nice article, and in my view you did the right thing. Science shouldn't be about avoiding conflict or smoothing egos. There's too much real work to do.

    A (partial) solution to these issues would be to have improved standards of publishing. The internet makes it very feasible to publish source data, analysis scripts, simulation code, etc. as part of the final paper. It seems we're still stuck in the pre-computer age, where all of the data and analysis are hidden from view. You can't really blame referees/editors from not catching these errors, since they have limited information to work with. So much is taken on faith. Yes, eventually the truth will out -- but it's an inefficient process.

    If both teams published all of their imagery and tagged cloud locations, it would (a) put this disagreement on a more factual footing, (b) improve the quality of reviewing, and (c) dissuade people from publishing known-shaky analysis in the first place.

    What we need is an arxiv on steroids. (And the end of proprietary publishing of publicly-funded work, but that's another topic.) Is anybody working on this?

  20. I guess I'm confused: why isn't what the competing group did a clear case of academic misconduct? Why aren't lots and lots of committees from the various institutions, publications, etc. looking into this? And why was my previous comment on these matters not published here? Surely you're not afraid of controversy or you wouldn't have mentioned this in the fist place?

  21. sorry, anon; my in box doesn't show any other comments from you. are you sure it got transmitted?