Dr. Eric Lim's Weblog

I reviewed a manuscript recently. Researchers were reporting a new test to diagnose `Christmas pudding embolism’ (details altered to preserve confidentiality).

Using a `mistletoe test’ (well, the baseline investigation they chose was not that well known either!) you achieved a sensitivity of 15% (95% CI 10 to 23%), and the new `red-nose reindeer’ test achieved a sensitivity of 49% (95% CI 45 to 55%). They concluded that the `red-nose reindeer’ test was at least 3 times more sensitive and should considered for routine diagnosis of `Christmas pudding embolism’.

Problem? Yes… a misconception that invalidates their conclusion.

Unlike most tests, sensitivity (and specificity) does not begin at 0%. A test is completely useless when the probability of getting a positive or negative result is no better than chance. If chance alone were in place, the probability of getting either a positive or negative result is … 50%. So, that’s where sensitivities and specificities start. Ergo, the `red-nose reindeer’ test is much worse than the `mistletoe test’ (it is much closer to 50%).

So how do you interpret a test result that has a sensitivity of 15% and a relatively narrow confidence interval?

Here comes the `science’ bits….

Unlike the usual interpretation, where in a test with high sensitivity, a negative result effectively rules out disease a `low’ sensitivity (i.e. way below 50%) has the interpretation that a negative result effectively rules in disease (ie a negative mistletoe test is a great way to diagnose `Christmas pudding embolism’). In practice, it is never used as such, but more likely to reflect the inappropriateness of the `mistletoe test’.

I guess the bottom line is no matter how confident you are about your research, it’s always a good idea to get a statistician to look through your work prior to submission.

Posted by Eric Lim at 09:41 AM | Permalink

I love being in an era where new technology is being evaluated. I sat in a lecture by a famous liver surgeon (Alan Johnson) in Sheffield (UK, where they filmed the `Full Monty’) prior to his retirement. He said that he had seen it all with regards to surgical research. It always follows the same pattern. First everyone loves a new surgical advancement, then everyone hates it, then we find indications for its appropriate use.

I think off-pump surgery is between phase 1 and 2 of the `Johnson chronology’. What really amuses me is how evangelical surgeons can be when defending their preferred technique. The sentiments are best reflected in this great quote:

`To cut things short, it is extremely important to understand that there is abundant clinically relevant though not necessarily statistically significant, scientific evidence to validate the midterm safety and efficacy of OPCAB.’ Anesthesiology 2005; 103: 902.

Frankly, surgical research can never be 100% replicable and reporting can never be 100% free from bias. The variances in patient response to a tablet (due to pharmacokinetics, pharmacodynamics differences) are usually balanced out if a trial is large enough, but no two surgeons will operate with the same skill. A new procedure that is shown to be superior compared to the standard in the hands of gifted surgeon A, may be inferior to the standard in two-left-hands surgeon B.

The guardians of information usually lies in the hands of the people most interested in the new technique. To give you an unbiased opinion about a new technique, would you employ a team of people who hate it? No, but you wouldn’t object if we recruited a team of people who already love it? A balanced panel is difficult, simply because people who are not interested in a technique do not usually want to get involved in its evaluation.

Therefore, how much of what can we believe about anything has been reported about off-pump surgery, except for our own intrepretation on the applicability of the raw results of the (randomized) clinical trials?

Posted by Eric Lim at 02:42 PM | Permalink

Statistics, I never understood any of it, but strangely the formidable aura generated by the knowledge and interpretation of the numbers gave an authority to textbooks and anyone who possess knowledge of it (a bit like Frodo in Lord of the rings).

Cardiothoracic surgery must be the single specialty where the use of statistics is most advanced. Just open any page in Kirklin's and you will be accosted by the most impressive series of numbers, equations and diagrams. Have you ever seen a comparable book in any other medical specialty? To be honest, I’ve always thought that you need a Masters in Statistics to fully comprehend it. It’s exactly what I obtained, and today I think it’s truer than ever.

What hope is there for the standard reader to understand the masses of information presented? The disability leads to a major vulnerability - when you cannot interpret the raw data, you become critically reliant other peoples interpretations.

Have you thought about it? It happens all the time! Not just on the podium in plenary sessions of prestigious meetings but anytime anyone tells you anything about cardiothoracic surgery, it will be based on a subjective interpretation of data. Unfortunately, it’ll usually be someone senior to you, and questioning the validity (no matter how nicely) risks a down turn to your career progression.

I always knew there was a good reason to be critically reliant other peoples interpretations.

Posted by Eric Lim at 10:13 PM | Permalink