Reading a study without being fooled by it

Fitness content runs on studies. A paper appears, an infographic follows, and within a week a training practice has been declared obsolete on the strength of a single experiment nobody has read.

You do not need a research degree to filter most of this. You need a handful of questions.

Who were the participants?

The most common reason a finding does not apply to you.

An enormous proportion of exercise research uses untrained young people, frequently university students, because they are available. Untrained participants improve on essentially any stimulus, which means studies on them are poor at distinguishing between training methods — everything works.

A finding in untrained participants tells you very little about what an experienced lifter should do. Look for the training status of the sample and treat "resistance-trained" as meaningful and "recreationally active" as not.

Also note age, sex and health status. Studies in older adults, in clinical populations, or exclusively in young men may not generalise.

How many, and for how long?

Small samples produce unstable results. A study with ten people per group can easily produce a large apparent effect by chance, and the published literature is biased toward those results because striking findings get published and null ones often do not.

Duration matters as much. Six-week studies capture mostly neural adaptation and are poor predictors of what happens over a year. Very few training studies run long enough to answer the questions lifters actually care about, which is a genuine limitation of the field rather than a criticism of any one paper.

What was actually measured?

Surrogate outcomes are everywhere. A study might measure muscle protein synthesis over a few hours and conclude something about muscle growth — but acute synthesis responses correlate poorly with long-term hypertrophy. Hormonal spikes after training were once considered central to growth and are now understood to be largely irrelevant to it.

Ask whether the thing measured is the thing you care about. Strength, muscle size measured properly, and performance are outcomes. Blood markers and acute responses are mechanisms, and mechanisms are frequently wrong about outcomes.

What was the comparison?

Effects are always relative to something. A study showing that a method produced gains means little if the comparison group did nothing. Compared with nothing, almost everything works.

The useful comparisons are against a reasonable alternative practice, matched for total volume and effort. Many studies claiming superiority for a novel method are actually comparing more work with less work.

Statistical significance is not importance

A statistically significant result means the observed difference was unlikely to arise by chance under the assumptions of the test. It says nothing about whether the difference is large enough to matter.

Look for effect sizes and, ideally, confidence intervals. A significant difference of half a kilogram of muscle over ten weeks is a real finding and a practically trivial one.

Conversely, a non-significant result in a small study does not mean there is no effect. It frequently means the study lacked the power to detect one.

Who funded it, and what did they want?

Industry funding does not automatically invalidate research, and a great deal of good work is industry-funded. But it is a known source of bias, particularly in supplement research, and it is worth knowing about.

Check the conflicts of interest statement. Check whether the authors sell something related to the finding.

Prefer syntheses to single papers

The most useful documents in the field are systematic reviews and meta-analyses, which pool multiple studies and weigh them. They smooth out the noise of individual small trials and they show you where the evidence is consistent and where it is a mess.

A good meta-analysis will also tell you the quality of the studies it included, which is often the most informative part.

When a single new study contradicts a well-established body of work, the prior probability strongly favours the body of work. Extraordinary claims require more than one paper.

The most useful habit

Read the methods section before the abstract's conclusion. The abstract is written to be quotable; the methods tell you what was actually done.

And be sceptical of anyone in fitness who cites a study without noting its limitations. The people worth listening to are the ones who tell you how confident they are and why, and who change their position when the evidence moves.