Stats and epidemiology methodology seminar/journal club

Interpreting and reporting clinical trials with results of borderline significance

2012-01-23T04:17:00.000-08:00

This week Amy Kirkwood gave a useful talk on how to report clinical trial results which have p-values just above or below the commonly adopted 0.05 threshold. Amy outlined her research showing the the inconsistencies in reporting these types of results and emphasised the importance of using standardised language when interpreting p-values. Her paper in the BMJ can be found here and her slides from the talk are below.

Borderline Significance

Miranda Armstrong (Cancer Epidemiology Unit, University of Oxford):Reported frequency of physical activity in a large epidemiological study

2011-11-18T06:18:00.000-08:00

Thank you to Miranda for giving an excellent talk about the methods adopted in the Million Women Study (a nationwide study investigating how reproductive and lifestyle factors affect women’s health)to assess physical activity over time via self-administered questionnaires. One of the most fascinating outcomes was that some participants actually underestimated their physical activity levels by failing to take account of common day-to-day tasks such as housework, and instead only report strenuous activity.

Miranda has kindly given us her slides from the talk, and a link to Miranda's paper on this topic which can be found here.

Armstrong 2011 Methodology Seminar

Clare Relton: ‘Rethinking Pragmatic Randomised Controlled Trials’

2011-10-06T09:58:00.000-07:00

Thanks to those who attended Clare Relton’s interesting talk on ‘rethinking pragmatic randomised controlled trials’ last Thursday. Clare has kindly given us the slides from her talk which can be found below. Clare has also written a paper on this topic in the BMJ, a link to which can be found here.

Stats and Epi 210911

Design, conduct and evaluation of complex interventions

2011-07-14T05:34:00.001-07:00

Thanks to all who came to Dr. Gill Lancaster's talk yesterday on the design, conduct and evaluation of complex interventions. Gill's slides can be found at the end of this post. Most of what Gill spoke about is based around a recent paper on the same topic in Statistical Methods in Medical research (linked here).

Gill also mentioned that many of the discussions from the RSS Primary Health Care Study Group are archived on a weblink on JISCMail..so if you're interested in looking at past meeting summaries or presentations these can be found here.

Oxford Talk SMMR Paper July 2011 Final

Routine use of PROMS in health care settings

2011-06-20T01:11:00.000-07:00

Thanks to Jill Dawson and everyone who came to last weeks meeting. Jill's slides can be found below, and if you are interested in this area you may want to read her two recent papers on the subject:

Routine use of PROMS in health care settings
BMJ. 2010 Jan 18;340:c186. doi: 10.1136/bmj.c186
Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ

Using PROMs - an example using elective shoulder surgery
The Open Epidemiology Journal, 2010, 3, 42-52

Routine Use of PROMs in Health Care Settings_150611

Ben Cairns talk - Agreeing to disagree

2011-04-21T02:12:00.001-07:00

Thanks to all who came to Ben's talk yesterday. Here's a link to the full-text of the paper which formed the basis of the discussion:

Lifetime body size and reproductive factors: comparisons of data recorded prospectively with self reports in middle age
BMC Medical Research Methodology 2011, 11:7

Summary of journal club discussion - Coste and Pouchot, 'A grey zone for quantitative diagnostic and screening tests'

2011-03-16T07:15:00.001-07:00

Thanks to those who contributed to today's journal club. Here is a short summary for anyone who couldn't make it.

We started the discussion by considering what clinicians do when they conduct a quantitative test. In general practice, it is rare for GPs to make a diagnosis based on one test result; often clinicians use multiple sources of information like signs, symptoms or other tests. In some cases, such as PSA testing, where there is a large grey zone, the doctor may sometimes ask what the patient wants to do based on a test reading (or series of readings).

One of the ideas we had was the the grey zone depicts the unacceptable levels of false positives or false negatives. This needs to be balanced against the harms occuring from each test/diagnosis based on the false positives or negatives. This led to the idea of the grey zone as giving us an idea of the value of a test; i.e. the proportion of people for whom you can get a conclusive result or not.

We thought a bit about how you would construct the grey zone for a specific diagnostic or screening test, and wondered if this could be achieved through clinical Delphi consensus, a systematic review of the literature, etc. One member said that she was involved in a study where patients were asked to set the grey zone, and to trade off how many false negatives were acceptable in order to gain one true positive. We agreed that a patient-set grey zone would be different than a clinician's, as patients tend to be more tolerant of false positives.

Beth Shinkins kindly prepared some discussion questions to go with this paper which are posted below:

Discussion Points
1) Coste and Pouchot extend the standard binary positive/negative test result framework to include an intermediate range of values where the diagnostic test is unable to determine disease status with any certainty. This is based on the argument made by Feinstein (1990) that the use of a single threshold is not representative of the reality of clinical decision making.
a.Statement by Battaglia and Pewsner (2003): “In practice clinicians hardly ever interpret results of continuous tests as being only ‘normal’ or ‘pathological’. They always take into consideration ‘how positive’ or ‘how negative’ the result is.” Battaglia and Pewsner (2003) – Do you agree with this statement? If so, why do we insist on using a binary framework if these interpretations do not translate into day-to-day practice?
b.Do you agree with the concept of classifying people as diseased or not diseased? Or are we talking more about people who require treatment/monitoring/therapy etc. as opposed to those who don’t?

2)“Pre-test probabilities may vary according to the epidemiological context, the care facility, information already gathered about diagnostic risk factors, and other factors; furthermore ‘subjective probabilities’ produced by clinicians or experts may be unreliable”
a.Does this completely undermine the clinical usefulness of the methodology?
b.How comfortable do you think clinicians are in using measures such as likelihood ratios to describe the accuracy of a test?

3)Excluding those in the ‘grey’ zone from diagnostic accuracy measures such as sensitivity and specificity are meaningless in isolation and potentially very misleading.
a.Do you agree? How could this be overcome?
b.How can we determine what proportion of the sample falling into the grey zone is acceptable?

4)Coste and Pouchot spend very little time explaining how a doctor should proceed in the face of an intermediate test result.
a.What do you think a doctor should do/is currently being done? Repeat the test? Use an alternative test?
b.Lemoine (2009) questions the assumption that a patient is either ‘diseased’ or ‘not diseased’ claiming that “the uncertainty is therefore always attributed to the measurement or to the test procedure itself” rather than “the qualitative complexity of a biological situation”. Coste and Pouchot do not discuss the possibility that an intermediate test result may be an indicator of a subclinical or early stage of disease – is this an important oversight?

5)“The proposal to enrich the interpretation of test results by measuring continuous parameters in shades of grey is an important step in the right direction. The introduction of different shades of grey may help to improve the interpretation of diagnostic test results and, more importantly, improve clinical outcomes.” Battaglia and Pewsner (2003)
a.Do you agree? How would this translate into clinical practice?

New schedule for March to September 2011

2011-02-18T05:51:00.001-08:00

I'm happy to announce a new schedule of paper discussions, seminars and talks for the statistics and epidemiology methodology sessions.

16 March at 1pm
Journal club
A grey zone for quantitative diagnostic and screening tests
Joël Coste and Jacques Pouchot
International Journal of Epidemiology 2003;32:304-313
Location: Teaching Room A, Rosemary Rue Building

20 April, 1pm
Talk by Ben Cairns, Cancer Epidemiology Unit, University of Oxford
Agreeing to disagree: self-report vs. measurement of body size in epidemiological studies
Location: CTSU Main Meeting Room, 1st floor, Richard Doll Building

25 May, 1pm
Talk by Jim Lewsey, Department of Public Health, University of Glasgow
The utility of advanced survival analysis methods for epidemiological and health economic modelling
Location: Seminar Room 1, Rosemary Rue Building

15 June, 1pm
Talk by Jill Dawson, Department of Public Health, University of Oxford
Use of PROMs in healthcare settings
Location: MSc Teaching Room, 1st floor, Rosemary Rue Building

13 July, 1pm
Talk by Gill Lancaster, Department of Maths and Statistics, Lancaster University
Design, conduct and evaluation of complex interventions
Location: Department of Primary Care, Hythe Bridge Street

21 September, 1pm
Talk by Clare Relton, ScHARR, University of Sheffield
The cohort multiple randomised controlled trial: a new study design
Location: TBC

The talks by Jill Dawson, Gill Lancaster and Clare Relton will be based on recent papers by the speakers which I'll circulate closer to the seminar date.

Please note the location of each seminar...apologies for the lack of a dedicated meeting space but things have been slightly complicated by our department moving to a different location!

Radio Silence

2011-02-10T03:43:00.001-08:00

Apologies again for the lack of updates - we have got a schedule of speakers/paper discussions from March 2011 onwards, so watch this space!

Why most research findings are false - Paper discussion on 20 October

2010-10-19T06:03:00.001-07:00

The winning paper for the paper discussion on 20 October was 'Why most published research findings are false' by John Ioannidis. Please do join us at 1pm in Teaching Room A (Rosemary Rue Building) for an informal discussion of a controversial and thought-provoking paper.

Stats and epidemiology paper discussion - Wednesday October 20

2010-10-18T02:45:00.000-07:00

Dear all,

This Wednesday, we plan to hold a paper discussion for the stats and
epidemiology methods seminar. If you would like to come, please take
a look at the following papers and vote here:

Doodle poll link

to choose a paper to discuss. The most popular paper will be chosen
for discussion - I will announce the results tomorrow.

1. Research Methods & Reporting: Is a subgroup effect believable?
Updating criteria to evaluate the credibility of subgroup analyses
Xin Sun et al

2. Bias in identifying and recruiting participants in cluster
randomised trials: what can be done?
S Eldridge, S Kerry, DJ Torgerson - BMJ, 200

3. Why most research findings are false
John P. A. Ioannidis

We plan to meet in Teaching Room A in the Rosemary Rue Building at 1pm
this Wednesday October 20. Everyone is welcome!

Summary of talk by Sue Mallett

2010-10-01T03:00:00.001-07:00

Thanks to all who came for Sue's informative talk on quality in reporting for prognostic models. Her talk was based on two publications, and if you would like any further information please see the following papers:

1. Reporting methods in studies developing prognostic models in cancer: a review.
Sue Mallett, Patrick Royston, Sue Dutton, Rachel Waters, Douglas G Altman. BMC Medicine, 2010

2. Reporting performance of prognostic models in cancer: a review
Susan Mallett, Patrick Royston, Rachel Waters, Susan Dutton and Douglas G Altman. BMC Medicine 2010, 8:21

Our next session will be a paper discussion on Wednesday 20 October.

New seminar series for Michaelmas term

2010-09-06T02:37:00.000-07:00

With summer over, we are re-starting the statistics and epidemiology methods seminar series. Please find a schedule below for the next few months.

The sessions are a chance to discuss methodology in epidemiological research, and are intended as a platform for learning and discussion. Everyone is welcome, and we would welcome any suggestions for future sessions. I would also be happy to hear from you if you are conducting research using a new or under-utilized approach and would like to lead an upcoming meeting.

Summaries for all sessions will be up on this website.

Sue Mallet, What makes a prognostic model more reliable for use in clinical practice?
22 September 2010
1-2pm
Rosemary Rue Building, Teaching room A

Paper discussion, paper to be decided
20 October 2010
1-2pm
Rosemary Rue Building, Teaching room A

Richard Peto, Rubbishing random effects
17 November 2010
12:30-1:30pm
CTSU Main Meeting Room 1st floor Richard Doll Building

Ly-mee Yu, How to handle missing data in trials
15 December
1-2pm
Rosemary Rue Building, Teaching room A

Update on seminar series

2010-06-10T10:12:00.001-07:00

If you are here looking for the next scheduled seminar, sorry for being so quiet!

We have teamed up with the Clinical Trial Service Unit (CTSU) and aim to bring a new schedule of statistical primers, seminars and paper discussions starting from September 2010. I will circulate more details in September.

In the meantime, if you have an idea for a seminar or would like to present some of your own work, please do email me and let me know!

Nada

The power of graphs in meta analysis

2010-04-27T01:21:00.001-07:00

We met today (April 26) to discuss the paper 'More than numbers: The power of graphs in meta-analysis' by Leon Bax and colleagues.

We went through the different plots types (short powerpoint presentation below). A few of the points I picked up on included:

- Limitations of the funnel plot, including the need for at least 25 studies or more to determine whether any studies are 'missing'.
- The need for accurate and complete trial registers to estimate publication bias
- Reporting the funnel plot statistic instead of including the plot in a paper
- The usefulness of L'Abbe plot, which can be extended to other studies with continuous variables. It was also pointed out that L'Abbe plot demonstrates whether the different studies report a constant risk reduction/increase which may be useful. There is a 'bubble plot' in Excel which can be used to draw a L'Abbe plot.

Many of the attendees said that they generally didn't report any plots other than a forest plot in a meta-analysis. Paul G. said that he sometimes tries a L'Abbe plot, and has also used 'Rosenthal's file drawer N' method to estimate publication bias by estimating how many studies of no effect would be needed to change the summary estimate.

We had some concerns about the simulation studies in this paper. The researchers who scored the graphs' ability to demonstrate hetereogeneity or bias may have benefitted from a training period, and may have preferred the forest plot due to familiarity. We thought that it may have been better to have more raters scoring fewer graphs, and agreed that most of the different plots were poor at assessing publication bias (Table 4).

Our next meeting is scheduled for May 19.

Meta Analysis Plots

April 21 meeting options

2010-04-16T03:52:00.000-07:00

Dear all,

I hope that some of you will be able to join us next week for the
stats and epidemiology journal club on Wednesday April 21. The
meeting will be held at 1pm in the 2nd floor meeting room in the
Rosemary Rue Building.

I have attached three possible papers for discussion. If you are
interested in coming, please send me back your vote for which paper
you'd like to discuss. The options are:

1. Bias in identifying and recruiting participants in cluster
randomised trials: what can be done? Sandra Eldridge and colleagues,
BMJ. 2009 Oct 9;339:b4006.
A discussion of how to design cluster randomized trials to minimize
selection bias

2. More Than Numbers: The Power of Graphs in Meta-Analysis. Leon Bax
and colleagues, Am J Epidemiol. 2009 Jan 15;169(2):249-55
What graphs should we use to report the results from a meta analysis?
A comparison for some of the popular options and limitations in each.

3. Power for studies with random group sizes. Walter T. Ambrosius,
Stat Med. 2010 Mar 11.
How to appropriately conduct a power calculation for observational studies

Let me know by next Monday and I will circulate the winning paper! Just email me at nada.khan@dphpc.ox.ac.uk.

Meeting on 17 March 2010 - Capture/Recapture

2010-03-19T02:27:00.000-07:00

Thanks to Geraldine Surman and Matthias Pierce from NPEU for a great talk on capture/recapture methods and applications in their own projects. I've uploaded their talk rfom Wednesday, which includes some Stata code and may be of interest to other people who wish to apply this method in their own research. Both Geraldine and Matthias are happy for people to contact them to discuss capture/recapture in further detail. Next meeting is scheduled for 21 April.

Capture Recapture Mar 10

Meeting on 20 January - Missing data and Mendelian randomization

2010-01-21T09:02:00.000-08:00

Dear all,

Hope you enjoyed Nicola Fitz- Simon's discussion on missing data and Mendelian randomization. I've uploaded her slides (see below) so do take a look if you missed the meeting. Our next meeting is scheduled for Wednesday February 17.

Missing Data and Mendelian Random is at Ion

December 16 discussion - Am J Epi paper by Kurth et al

2009-12-16T06:20:00.000-08:00

Today we discussed the paper by Kurth et al in the Am J Epi . Some slides on our discussion can be found just below this post. Our next meeting is on Wednesday, January 20. Nicola Fitz Simons will be leading a discussion on missing data and Mendelian randomization. Happy Christmas!

December Meeting

Next meeting - 16 December, 1pm

2009-12-14T07:57:00.000-08:00

Our next meeting is on 16 December 2009 (Wednesday) at 1pm in the 2nd floor meeting room. The paper up for discussion is:

Kurth et al
American Journal of Epidemiology, 2006; 163: 262-270
Results of Multivariable Logistic Regression, Propensity Matching, Propensity Adjustment and Propensity-based Weighting under Conditions of Nonuniform Effect

I've put together a few slides to get our discussion going - I thought that some points for discussion could include the benefits and disadvantages of different adjustment methods, and a consideration of what question we're asking of the data when we conduct analysis.

Hope to see you there! I'll bring some Christmas treats for us to snack on during the meeting. Let me know if you can't get a hold of the paper full text.

Summary of our second meeting on Oct 26

2009-10-29T04:02:00.001-07:00

Many thanks to Karen Smith for her clear and informative seminar on the use and application of interrupted time series analysis. Karen has kindly allowed me to publish her Powerpoint slides online, so if you are interested in seeing her talk, it's just below this post. Our next meeting is on Wednesday, November 18 at 1pm. Nicola Fitz-Simons, who is based at NPEU, will be leading a seminar based on some of her methodological research.
Interrupted Time Series

Seminar by Karen Smith

2009-10-12T03:38:00.000-07:00

The next journal club will be led by Karen Smith, senior medican statistician at the Centre for Statistics in Medicine here in Oxford. Karen will be leading a discussion on interrupted time series analysis based on a paper she worked on, which was recently published in the BMJ. The paper can be found here:

Effect of withdrawal of co-proxamol on prescribing and deaths from drug poisoning in England and Wales: time series analysis

Please do join us for this talk and discussion - just to note that Karen will be giving her talk on Monday, October 26, NOT October 21 as we had previously scheduled. All seminars will take place from 1-2pm, 2nd floor meeting room, Rosemary Rue building.

Testing for baseline balance in clinical trials

2009-09-09T09:37:00.000-07:00

There are a number of interesting points in this paper. In particular, section 4: Some misconceptions about balance.

To paraphrase, consider a two arm trial in which 200 patients (100 male and female) are allocated at random to one of two treatments. The resulting proportions of male and females in each arm after randomization are different (this is far more likely to be the case than being exactly equal (50 males and females in each arm) regardless of how well the randomization was performed. Exact balance in the covariates is probably more indicative of a non-randomzied trial.

Does this matter, yes if the covariate in question (gender in this case) has an effect on the outcome,

Suppose a model for the outcome y is;

y(1= treatment) = mu + beta1*x + theta + error

y(0 = control) = mu + beta1*x + error

where theta is the true treatment effect and beta1 is coefficient for the confounding covariate and x is the covariate (gender). A naive estimate of theta is the difference of means between the two groups.

theta.hat = ybar(1) - ybar(0)

which will be biased by the value of

beta1(xbar(1) - xbar(0))

The magnitude of which is dependent on the distribution of x across the treatment groups and the value of beta1.

The problems with testing for imbalance:

It is common for people to judge balance via some kind of significance test of the group means. There are problems with this as is described in Senn's paper. Firstly, all that matters is the observed data not the wider population distribution over repeated identical trials. So the question of "is that a real difference" is meaningless and arises from the confusion between identifying a sample and population. Secondly, is one of why should a difference of two standard errors define imbalance. As described earlier any difference in the distribution of x is potentially relevant when beta1 is not zero and does not become nullified if p = 0.1, and as the p-value is dependent on the sample size it is possible for a difference in means to be balanced in terms of the p value in a small trial whereas in a larger trial the exact same difference would be identified as imbalanced.

The biggest problem with the significance test is not conditoning on an important covariate just because the p-value is > 0.05. If the effect of the covariate on the outcome is substantial the bias will be important. However, if balance is obtained in the important covariates it does not follow that you can ignore it and perform an unconditional analysis. Although, the estimate of the means will be unbiased, the standard errors will not be. The effect of the covariate on the variance is in fact maximised with the unconditional analysis.

Conclusion

This is from Senn's paper directly;

Thus, to sum up, a conditional analysis of an unbalanced experiment produces a valid inference; an unconditional analysis of a balanced experiment does not. Question; what is the value of balance as regards validity of an inference? Answer: none.

First meeting - 8 September

2009-09-09T01:55:00.001-07:00

We held our first meeting yesterday to discuss an old, but relevant paper by Stephen Senn. Here's a link to the abstract:

Testing for baseline balance - Stephen Senn

Our discussion revolved around a number of issues highlighted in the paper. We wondered whether or not it's necessary to add a 'Table 1' to a publication, and although some of the members thought it was useful to add a table describing patient characteristics, other felt it's generally unnecessary to conduct significance tests to highlight differences between two groups in clinical trials or case control studies. Although baseline tests may demonstrate that the randomization hasn't worked, it's very easy to manipulate randomization and maintain baseline balance in a clinical trial.

We also talked about how to decide what covariates to add to an analysis. The paper specifically advises readers to choose covariates based on previous studies, and fit those covariates in multivariable regression techniques whatever the degree of imbalance in the baseline tests. This approach was generally well received, and we talked about some of the issues around covariate selection. Some of the members were experienced in using causal diagrams and directed acyclic graphs, so we thought that this might be an interesting area to discuss in a future seminar.

Since this was the first meeting, I thought it would be a good idea to think forward and discuss ideas for future meetings. Areas for future discussions could include Mendelian randomization and approaches to missing data. I'd like to also invite speakers to discuss their own work, so if you have any ideas or would like to discuss your own work in progress, email me at nada.khan@dphpc.ox.ac.uk.

Next meeting is on October 21!