When discussing whether or not experiments can detect differences that matter, it is crucial to look at the results of the experiment and think about them critically. In a statistics class I once took, my professor presented data that about commute times by route taken. The data showed that there was a statistically significant difference in the time between each route, but when further analyzing it, the data actually showed that the time difference was less than twenty seconds. He cautioned us that while data may be significant, it also may just be pointless.
When considering whether the study shows correlation or causation, you must consider how the data was collected and whether other variables were controlled for. If they were not, it is not accurate to say that it is causation.
When discussing the mechanism of relationship, Hanage proposes returning to a reductionist approach. Often, it is necessary to look at the big picture, but here it is important to pinpoint little details in order to determined the true nature of the relationship between findings.
When thinking about how experiments reflect reality, it is important to consider the findings within the bigger picture. What are the real consequences of the work, and how could they potentially be expanded upon to further the field in which they were presented and beyond. I think that this is the most important of the five questions, because without real world application, many findings are not useful.
When considering whether anything else could explain the results, it is necessary to look at the experimental design and think about whether or not any confounding variables exist.
