Commentary on A Decision Tree for Controlled Trials
Sportscience 9, 40-41
With their "Decision tree for controlled trials", Alan and Will have provided an excellent resource for experimenters. They communicate precisely why there is no panacea for experimental design, but that choice depends on the feasibility of including control participants, the length of treatment washout time, as well as the reliability of the outcome measure.
To aid the communication of the various types of experimental design, Alan and Will used a notation system presented in their Figure 1. Such schematics have been attempted before, but I think the notation used by Alan and Will has the great advantage that the arrows show exactly which time-point is compared to other time-points for generation of the change or difference scores. This favourable aspect of the notation system communicates the precise link between experimental design and analysis of data.
Alan and Will included the time-series or quasi-experimental design in their paper at my suggestion. Researchers might wonder why such a design would be adopted at all, given its obvious lack of a control group. One example might be situations in which time itself is the intervention. Such is the case in studies on circadian variation in performance. Any readers who have tried to research elite athletes might also find it difficult or maybe even unethical to include a control group. In the future, Alan and Will might like to extend their statistical expertise to this situation in particular, since the analysis of time-series data might involve complicated covariate analysis (to control for intervening variables also changing over time) of correlated data-sets.
In the paper, two important issues were also mentioned. First, even in a fully controlled trial, it was pointed out that there may be reactive effects due to the participants knowing they have been allocated to either the treatment or control group. Another so-called threat to validity in a controlled trial is the potential for change in participants' behaviour if they receive feedback about their pre-treatment scores before the treatment or post-test. Even performing the pre-treatment test can in principle affect the control and experimental treatments differently. Although any physiological responses to exercise might not be due to such reactive effects, this threat to experimental validity might influence an outcome measure of human performance. One design that is supposed to estimate reactive effects due to the pre-treatment measurements is called the Solomon 4-group. Using the new notation, the design is as follows:
The Solomon 4-group is a complicated design and demands a large sample size (due to the inclusion of four groups). Nevertheless, I have seen it employed in some large scale studies on physical activity interventions, for example.
Secondly, the important issue of lack of retention of research participants (often called subject mortality or attrition) was mentioned by Alan and Will. If a treatment has been so badly received by participants that they decide to vote with their feet, a researcher can hardly label the treatment a success, even if the data analysed on the remaining "selected sample" suggests that this is so! The CONSORT statement cited by Alan and Will deals with this important issue by advising researchers to distinguish between two types of analysis: intention-to-treat, where you include all participants in the analysis, regardless of how well they complied with the treatment, and as-treated, where you include only those who did everything properly (Altman et al., 2001). A good reference on the Web is the intention-to-treat page at Gerard Dallal's statistics site. Intention-to-treat analyses are an issue where the outcome is mortality or morbidity that can be quantified without a post-test, but exercise and sport-science analyses are mainly as-treated, because participants have to get through the treatments and perform the post-test before they can be included. Regardless, it is important to document what happens to all the participants, and to justify the approach you have taken in the analysis.
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang L (2001). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine 134, 663-694
Published Dec 2005