Interpreting and reporting clinical trials with results of borderline significance

This week Amy Kirkwood gave a useful talk on how to report clinical trial results which have p-values just above or below the commonly adopted 0.05 threshold. Amy outlined her research showing the the inconsistencies in reporting these types of results and emphasised the importance of using standardised language when interpreting p-values. Her paper in the BMJ can be found here and her slides from the talk are below.

Borderline Significance

Miranda Armstrong (Cancer Epidemiology Unit, University of Oxford):Reported frequency of physical activity in a large epidemiological study

Thank you to Miranda for giving an excellent talk about the methods adopted in the Million Women Study (a nationwide study investigating how reproductive and lifestyle factors affect women’s health)to assess physical activity over time via self-administered questionnaires. One of the most fascinating outcomes was that some participants actually underestimated their physical activity levels by failing to take account of common day-to-day tasks such as housework, and instead only report strenuous activity.

Miranda has kindly given us her slides from the talk, and a link to Miranda's paper on this topic which can be found here.

Armstrong 2011 Methodology Seminar

Clare Relton: ‘Rethinking Pragmatic Randomised Controlled Trials’

Thanks to those who attended Clare Relton’s interesting talk on ‘rethinking pragmatic randomised controlled trials’ last Thursday. Clare has kindly given us the slides from her talk which can be found below. Clare has also written a paper on this topic in the BMJ, a link to which can be found here.

Stats and Epi 210911

Design, conduct and evaluation of complex interventions

Thanks to all who came to Dr. Gill Lancaster's talk yesterday on the design, conduct and evaluation of complex interventions. Gill's slides can be found at the end of this post. Most of what Gill spoke about is based around a recent paper on the same topic in Statistical Methods in Medical research (linked here).

Gill also mentioned that many of the discussions from the RSS Primary Health Care Study Group are archived on a weblink on JISCMail..so if you're interested in looking at past meeting summaries or presentations these can be found here.

Oxford Talk SMMR Paper July 2011 Final

Routine use of PROMS in health care settings

Thanks to Jill Dawson and everyone who came to last weeks meeting. Jill's slides can be found below, and if you are interested in this area you may want to read her two recent papers on the subject:

Routine use of PROMS in health care settings
BMJ. 2010 Jan 18;340:c186. doi: 10.1136/bmj.c186
Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ

Using PROMs - an example using elective shoulder surgery
The Open Epidemiology Journal, 2010, 3, 42-52

Routine Use of PROMs in Health Care Settings_150611

Ben Cairns talk - Agreeing to disagree

Thanks to all who came to Ben's talk yesterday. Here's a link to the full-text of the paper which formed the basis of the discussion:

Lifetime body size and reproductive factors: comparisons of data recorded prospectively with self reports in middle age
BMC Medical Research Methodology 2011, 11:7

Summary of journal club discussion - Coste and Pouchot, 'A grey zone for quantitative diagnostic and screening tests'

Thanks to those who contributed to today's journal club. Here is a short summary for anyone who couldn't make it.

We started the discussion by considering what clinicians do when they conduct a quantitative test. In general practice, it is rare for GPs to make a diagnosis based on one test result; often clinicians use multiple sources of information like signs, symptoms or other tests. In some cases, such as PSA testing, where there is a large grey zone, the doctor may sometimes ask what the patient wants to do based on a test reading (or series of readings).

One of the ideas we had was the the grey zone depicts the unacceptable levels of false positives or false negatives. This needs to be balanced against the harms occuring from each test/diagnosis based on the false positives or negatives. This led to the idea of the grey zone as giving us an idea of the value of a test; i.e. the proportion of people for whom you can get a conclusive result or not.

We thought a bit about how you would construct the grey zone for a specific diagnostic or screening test, and wondered if this could be achieved through clinical Delphi consensus, a systematic review of the literature, etc. One member said that she was involved in a study where patients were asked to set the grey zone, and to trade off how many false negatives were acceptable in order to gain one true positive. We agreed that a patient-set grey zone would be different than a clinician's, as patients tend to be more tolerant of false positives.

Beth Shinkins kindly prepared some discussion questions to go with this paper which are posted below:

Discussion Points
1) Coste and Pouchot extend the standard binary positive/negative test result framework to include an intermediate range of values where the diagnostic test is unable to determine disease status with any certainty. This is based on the argument made by Feinstein (1990) that the use of a single threshold is not representative of the reality of clinical decision making.
a.Statement by Battaglia and Pewsner (2003): “In practice clinicians hardly ever interpret results of continuous tests as being only ‘normal’ or ‘pathological’. They always take into consideration ‘how positive’ or ‘how negative’ the result is.” Battaglia and Pewsner (2003) – Do you agree with this statement? If so, why do we insist on using a binary framework if these interpretations do not translate into day-to-day practice?
b.Do you agree with the concept of classifying people as diseased or not diseased? Or are we talking more about people who require treatment/monitoring/therapy etc. as opposed to those who don’t?

2)“Pre-test probabilities may vary according to the epidemiological context, the care facility, information already gathered about diagnostic risk factors, and other factors; furthermore ‘subjective probabilities’ produced by clinicians or experts may be unreliable”
a.Does this completely undermine the clinical usefulness of the methodology?
b.How comfortable do you think clinicians are in using measures such as likelihood ratios to describe the accuracy of a test?

3)Excluding those in the ‘grey’ zone from diagnostic accuracy measures such as sensitivity and specificity are meaningless in isolation and potentially very misleading.
a.Do you agree? How could this be overcome?
b.How can we determine what proportion of the sample falling into the grey zone is acceptable?

4)Coste and Pouchot spend very little time explaining how a doctor should proceed in the face of an intermediate test result.
a.What do you think a doctor should do/is currently being done? Repeat the test? Use an alternative test?
b.Lemoine (2009) questions the assumption that a patient is either ‘diseased’ or ‘not diseased’ claiming that “the uncertainty is therefore always attributed to the measurement or to the test procedure itself” rather than “the qualitative complexity of a biological situation”. Coste and Pouchot do not discuss the possibility that an intermediate test result may be an indicator of a subclinical or early stage of disease – is this an important oversight?

5)“The proposal to enrich the interpretation of test results by measuring continuous parameters in shades of grey is an important step in the right direction. The introduction of different shades of grey may help to improve the interpretation of diagnostic test results and, more importantly, improve clinical outcomes.” Battaglia and Pewsner (2003)
a.Do you agree? How would this translate into clinical practice?

New schedule for March to September 2011

I'm happy to announce a new schedule of paper discussions, seminars and talks for the statistics and epidemiology methodology sessions.


16 March at 1pm
Journal club
A grey zone for quantitative diagnostic and screening tests
Joël Coste and Jacques Pouchot
International Journal of Epidemiology 2003;32:304-313
Location: Teaching Room A, Rosemary Rue Building

20 April, 1pm
Talk by Ben Cairns, Cancer Epidemiology Unit, University of Oxford
Agreeing to disagree: self-report vs. measurement of body size in epidemiological studies
Location: CTSU Main Meeting Room, 1st floor, Richard Doll Building

25 May, 1pm
Talk by Jim Lewsey, Department of Public Health, University of Glasgow
The utility of advanced survival analysis methods for epidemiological and health economic modelling
Location: Seminar Room 1, Rosemary Rue Building

15 June, 1pm
Talk by Jill Dawson, Department of Public Health, University of Oxford
Use of PROMs in healthcare settings
Location: MSc Teaching Room, 1st floor, Rosemary Rue Building

13 July, 1pm
Talk by Gill Lancaster, Department of Maths and Statistics, Lancaster University
Design, conduct and evaluation of complex interventions
Location: Department of Primary Care, Hythe Bridge Street

21 September, 1pm
Talk by Clare Relton, ScHARR, University of Sheffield
The cohort multiple randomised controlled trial: a new study design
Location: TBC

The talks by Jill Dawson, Gill Lancaster and Clare Relton will be based on recent papers by the speakers which I'll circulate closer to the seminar date.

Please note the location of each seminar...apologies for the lack of a dedicated meeting space but things have been slightly complicated by our department moving to a different location!

Radio Silence

Apologies again for the lack of updates - we have got a schedule of speakers/paper discussions from March 2011 onwards, so watch this space!

Why most research findings are false - Paper discussion on 20 October

The winning paper for the paper discussion on 20 October was 'Why most published research findings are false' by John Ioannidis. Please do join us at 1pm in Teaching Room A (Rosemary Rue Building) for an informal discussion of a controversial and thought-provoking paper.