Wednesday 20 April 2011

Update on exercise and memory story

A few months ago, I wrote a blog post in response to a “pre-arranged” submission by Kirk Erickson and colleagues to Proceedings of the National Academy of Sciences (PNAS), which purported to find evidence that moderate exercise leads to substantial improvements in memory. The article in question received a great deal of media attention, with big claims being made that older adults, who tend to be worried about declining memory abilities, might be able to hold off the effects of old age on memory with a simple exercise regime.

Unfortunately, when the data were looked at more closely, it was clear that the picture was more complicated. For one thing, although the experimental group that performed exercise for one year did show a 2.3% increase in memory score, a control group who did not perform the exercise showed a 3.7% increase over the same period. In other words, the exercise group did not show any increase in memory performance relative to the control group.

When I published this blog post, there was quite a bit of correspondence over Twitter, and Susan Krauss Whitbourne wrote a follow-up article that included an interesting interview with one of the authors of the experiment, defending the findings.

Now, Robert Coen and colleagues have published a letter in the same journal as the original article, PNAS, arguing strongly that the Erickson memory results are flawed. As they say:

Contrary to both the title and abstract, there is virtually no evidence in this article that exercise improved memory. After 1 y there were no differences between the exercise and control groups.

They go on to argue that “both the title and abstract are misleading and a major overstatement of the findings.”

For those not used to the usually fairly genteel nature of published academic debate (as opposed, perhaps, to the occasionally more robust discussions during conferences), the wording used by Coen and colleagues in their letter is very strong and represents quite a rebuke.

Erickson and colleagues have written a reply to Coen’s letter in which, to all intents and purposes, they acknowledge the charges brought. As they say: “We agree that the title and abstract could have been clearer on the lack of a difference between the groups in terms of spatial memory performance.”

They go on to suggest the possibility that brain regions other than the one they focused on in the original article, the hippocampus, might have contributed to the effects reported.

Erickson et al. also make the extraordinary point that “our study was not conducted in a vacuum, and our results are consistent with other research on the effects of exercise on memory.”

In other words, they appear to be saying that it’s ok to overstate your results if the effects you erroneously claim to have found are consistent with previous research.

This is simply not good enough. Scientists often moan about how their cautiously-phrased, carefully-caveated journal article has been over-simplified and their findings misrepresented and sensationalized by journalists and the media. In this instance though, as I wrote in my previous blog post, the obscuring of the true nature of the findings seems to have been attributable to the way the scientists concerned chose to write them up in their paper.

As Coen and colleagues conclude in their letter, "it behooves us all to ensure rigor in our scientific reporting."

I couldn’t agree more.

(thanks to @markgbaxter for bringing the PNAS letters to my attention)

4 comments:

  1. Charlie Wilson @crewilson20 April 2011 at 12:59

    Here here...

    ReplyDelete
  2. I would be embarrassed to have my name within 20 miles of that reply. "Our data contradict the hypothesis that exercise helps memory and that changes in hippocampus volume will correlate with performance on a memory task, but it's ok because other people have found exercise helps memory and we just know the hippocampus is involved, ok."

    I'm going to use this paper to teach students to 'spot the problem' and 'read the Results, dammit'.

    ReplyDelete
  3. I think these are all great points. Many media science writers overstate the findings or make classic errors of generalizing from college students to "people" and mice to humans (but rarely vice versa).I would still maintain that the nuanced findings are important to consider (i.e. the correlational as opposed to the mean differences). The article also demonstrates the messiness of real world research. The "control" group also exercised, albeit not to the same extent. There was no way they could restrain the subjects from engaging in any exercise, and considering that the exercise group's aerobic activity was only moderate, that undoubtedly confounded the results even further. I do like Wilson's point about comparing the title with the results and looking for alternative explanations as a teaching method. However, I still plan to continue my own "moderate" if not strenuous exercise just to ensure my hippocampus (or whatever region is affected) stays healthy!

    ReplyDelete
  4. Thanks all for the interesting comments.

    Andrew - I entirely agree that the original article makes a good example of why it's important to read the whole paper, and not just the title and abstract.

    Susan - You're right that the control group didn't do nothing, and that complicates things. I actually thought it was a strength of the original study that they had been careful to select a "realistic" control group. However, it does mean that the results indicate that exercise won't improve an older adult's memory any further than whatever regular physical activity they might do.

    ReplyDelete