See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/321101080

Risk, probability and decisions in emergency medicine v2

Book · November 2017

CITATIONS READS

0 4,862

2 authors:

Simon D Carley Natalie May

Manchester Metropolitan University Ambulance Service New South Wales
226 PUBLICATIONS   2,942 CITATIONS    14 PUBLICATIONS   54 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

evidence based medicine View project

#FOAMed View project

All content following this page was uploaded by Simon D Carley on 16 November 2017.

The user has requested enhancement of the downloaded file.

Risk, probability and
decisions in
emergency medicine

St.Emlyn’s virtual hospital, blog and podcast

 Think

SMACC Gold
Foreword Thanks for downloading and reading our first St.Emlyn’s e-book. In it, we
focus on decision making, arguably one of the most important core skills of
medicine. In the acute specialities such as Emergency Medicine, Acute
Medicine, Critical Care and Prehospital Care the complexities of clinical
decisions are amplified by the time critical nature of our practice. We are
often required to make decisions quickly, and also at a point in the patients
journey when there may be significant uncertainty as to what the underlying
cause of the patient’s injury or illness is.

Those of us who operate in the time critical, information light world of the
resuscitationist know that we are judged by the quality of these complex
decisions. This is both an immense challenge, but also a huge privilege as
those decisions can transform a patient’s clinical course. If you follow the
St.Emlyn’s blog and podcast then you will already know this, but I would
challenge you to stop and consider how much of your formal time in training
has been spent on understanding risk, uncertainty, decisions and dilemmas?
I suspect relatively little, which is arguably surprising, considering its
importance. How do we learn these skills, and how did others develop
them? A useful exercise is to stop and think about a colleague or teacher
who exhibits that elusive quality of ‘great clinical judgement’. How do you
think they developed those skills? It’s likely a combination of practice,
reflection, feedback, review and time, but undoubtably, it did not come
without effort or study. At St.Emlyn’s we’ve always believed, that by thinking
about thinking (meta-cognition), we can improve our understanding of
clinical practice and thus become better clinicians. This book is an
introduction into how we do this.

ii
We hope to inspire you to think more deeply about the risks,
uncertainties and complexities that underpin decision making in
acute specialities.

We could not have put this work together without the help and
guidance of all who have contributed to the St.Emlyn’s blog and
podcast. Clearly, the authors of the following chapters, blogs and
podcasts have contributed, but the wider #FOAMed community
have moulded and guided us to think harder and deeper about
decisions in a way that we could never have achieved on our
own. See Chapter 3 for links and ideas on others in the #FOAMed
world who think the same way that we do, and who love to share
what they know freely, openly and without deed or covenant.

As this is our first venture into e-books, we’d love to hear what
you think. Get in touch via the website, twitter, Instagram or
Facebook. We’re interested in what you have to say and would
love you to join our conversation.
Note: There are no patient images in this book. Clinical pictures
Simon Carley are of simulated patients and mannequins.

Professor of Emergency Medicine

• Subscribe to the http://www.stemlyns

• Subscribe to our PODCAST on iTunes
• Follow us on twitter @stemlyns
• See our best pics and photos on Instagram
• PLEASE Like us on Facebook
• Find out more about the St.Emlyn’s team

iii
Chapter 1

Emergency
Medicine
A risky business

This chapter explores some of the fundamental

statistical principles that underpin diagnostics. All
emergency physicians should understand these, as
they are key to using diagnostic tests wisely.

There are a few stats in here, but don’t be

frightened, it’s pretty straightforward stuff.
Section 1: Written by Simon Carley

You are not a diagnostician

KEY POINTS

1. You are not (really) a diagnostician.

2. Most diagnoses you make are probabilities.

3. Maybe you’re a problastician.

5
Let’s start by thinking about whether you are really an emergency So, I’ll bet that you said inferior MI didn’t you. You did, I know you
medicine diagnostician. Are you a diagnostician? Of course you did because I would ahve said the same thing too, but we would
are! Why do I even need to ask? both be wrong, well possibly….

Well, to be honest I’m no longer very sure to be honest. When I The first reason, is that inferior MI is a politically incorrect
speak to friends and colleagues in emergency medicine and ask diagnosis, according to the politically correct guide to cardiology.
them to describe themselves, ‘diagnostician’ is a word that is
Secondly, and leaving the PC brigade behind, when we look at
commonly used. I know what they mean and so do they, but is it
this ECG, we come to a conclusion that it is, indeed, an MI yet we
the right word? Making a diagnosis is perhaps not quite as
also know that there are many ST elevation mimics, such as LV
straightforward as you might think, and if diagnosis is as
aneurysm. In this case, we know that we should still start therapy
straightforward as we think, then perhaps we are not
and get the patient off to cath lab but there is a possibility that we
diagnosticians.
will be wrong (about 5% of the time in fact). What we are doing is
This can be a challenging topic so it may be simpler to attaching a label (in this case MI) to a probability. So it’s probably
understand if we consider an example. an MI, so let’s treat it as an MI, and overall patients (as a
population) will be better off.
Let’s think about something really easy. Have a look at the ECG
below and tell me what the diagnosis is. The patient is a 54 year Great, but that’s all a bit picky isn’t it? Does this really matter?
old man with chest pain. Well in the example above, perhaps not, as we are going to treat
on the basis of a high risk, so we have decided a diagnosis and
we are going to treat. We know that this presents problems for
some patients as the therapy for things like MI (and stroke) are
risky in themselves. Every so often, we will harm a patient through
a known complication of a therapyfor a disease they do not have.
Worth stopping and thinking about that one for a moment. It
means that if we assign therapies on the basis of a probability
harm will happen, sorry, harm WILL happen.

6
Difficulties also arise when we consider what we might do in the that a sensitivity of 98% means that we miss 2% of patients with
exclusion of disease because the same problems arise. Let’s disease. That’s 1 in 50 folks, with that implication, you will be
think of another example where we are seeking to exclude a missing a lot of serious disease in your career……or does it?
diagnosis. If we are talking about probabilities when ‘ruling in’ a (more of this later)
diagnosis, then do we get similar problems when we ‘rule out’?
Now that you are worried that you’re not a diagnostician at all,
The diagnosis of the moment seems to be PE, with loads of posts just a doc playing the odds and gambling with your patients life,
and debates pinging around the blogosphere. I’m sure that you it’s time to think about the implications of being a probablestician
will see a number of rule out strategies for the management of PE rather than a diagnostician, and perhaps why as clinicians we like
patients. I’ll also bet, that in the majority of cases, the implication more certainty than we typically justify. That’s for part 2.
of the rule out strategy is that, if the patient is negative for
investigation, then the patient does not have a PE. This is not
true. Just as it’s not true for most rule out MI, appendicitis, UTI, Simon Carley
renal stone and subarachnoid bleed protocols. What we are
 
actually doing is moving patients from a pretest probability where
we are worried, down to one where the risks of pursuing further PS. Ever since I made the word probablastician up I’m still not
investigation, outweigh the benefits. This is one of the reasons quite sure how to spell it ;-)
why I like the way Scott Weingart wrote the rule out PE pathway
at EMCRIT. The pathway from Scott Weingart is not typical in that
respect, as the end point is ‘stop work up’, as opposed to PE
ruled out – because it’s not. In reality, the sensitivity of most PE
rule out strategies is in the 90’s but they are certainly not 100%.

The obvious result of this, is that if you use a typical rule out
protocol, you are really using a probability protocol and therefore
you must be missing something. Let’s take a really good rule out
protocol for PE that has 98% sensitivity, – that’s fab and 98% is
regarded by most people as a SnOut, but what we now know is
7
Section 2: Written by Simon Carley

Almost every diagnoses is a mix of truth and error

KEY POINTS

1. Whether you get a positive or a negative

result, it’s probably a mix of truth and error.

2. As the clinician interpreting the test you

won’t know which it is!

3. Uncertainty is greatest in the earliest

stages of clinical assessment (in other
words in the emergency department).

8
 probabilistic and in the groups labelled ‘diagnosis made’ or diagnosis
Now that you have begun to understand the probabilistic nature of excluded, we are really mixing up two groups of patients.
medicine, you will hopefully be wondering about probability, as it
• The diagnosis made group includes, those who really have the
applies to diagnosis. I hope that I have convinced you that when we
disease (True positives) and also those of who do not have the
label a patient with a diagnosis, we are usually assigning that label on
disease (false positives).
the basis that the likelihood of them having it is high….., but how
• Similarly, the diagnosis excluded group, again, have a mixed
high?
group of those who do not have the disease (true negatives) and
those that we have missed (false negatives).
Well, to some extent that tipping point for labelling should depend
•
upon the diagnosis we are looking at. At one extreme I often use the
generic term for all non-specific paediatric illness (it’s a bit of a virus –
and no antibiotics won’t work); at the other extreme I want to be pretty
certain that my diagnosis of myocardial infarction is correct as, the
therapy for the condition (thrombolysis ot PCI) has a risk in itself. So
setting the tipping point for labelling is very much disease specific, but
as diagnosticians (let’s go back to using that word from now on) we
need to have a good feel for the consequences of diagnosis.

The rather simplistic view of

diagnostics could be
summarised in the following
flow chart.
I know that you know this, as it will be familiar from med school stats
The reality of course cannot lectures. In fact, we are really just talking about the traditional 2×2
be this simple though and diagnostic table that will be familiar to you all. Let’s just review that
there are two fundamental 2×2 table before we recognise that it’s bloomin’ useless to clinicians.
problems in this process,
t h e fir s t i s t h a t t h e
diagnostic process is

9
“Useless you say”, is that not heresy?” The 2×2 table is the We have yet another problem, and we’ve not even got onto therapy
foundation of diagnostic studies, and to be honest, I’ve produced yet! We must accept that our diagnostic prowess is based on putting
plenty in my time as a researcher. The difficulty, is that all 2×2 tables patients into categories that are inevitably mixed.
are created in the knowledge of the results of the gold standard
Simon Carley
….which is not what we have, when using the new test. So from our
perspective when we are using a clinical test, we only get the results
of A+B or C+D, we cannot differentiate without doing more tests.

This is a situation that has been described as ‘researchers having their

head in the clouds’, whereas we clinicians, have our feet firmly on the
ground. Time to put on your glasses and go 3D!

As the poor clinician, we are left scratching our heads, not really
knowing whether our patient does or does not have disease.

In (micro) summary, when we look back at the simplistic flow diagram

of diagnosis above, we know that the first stage, the stage of making a
diagnosis, is flawed.

10
Section 3: Written by Simon Carley

Is it OK to miss a diagnosis?

KEY POINTS

1. If we know we miss diagnoses, we need to

ask ourselves how many it is acceptable to
miss.

2. The hardest thing to do is to accept that we

always work with a degree of risk.

3. The level of diagnostic risk we accept

depends on many factors relating to the
patient, the condition and the clinician.

4. We can improve our understanding of risk

by explaining using natural frequencies.

11
Emergency medicine is a risky business. In this continuing series, we certainty of disease (or lack of disease). You may remember this
are going to spend a bit of time looking at how we feel about risk and diagram that explains how the poor clinician is looking at a
what we consider to be acceptable in practice. If nothing else, it’s a combination of either true positives + false positives OR true negatives
good excuse to look at a really risky video – watch this link and think + false negatives.
about whether this is acceptable.
So, having accepted that we are probablasticians, we must then face
the fact that we are wrong some of the time. If you are keeping up to
I’m guessing that you think not (unless you are crazy IMHO), but the
speed then you will already be getting to the next question which is….
Isle Of Man is where I did some of my med student training and was
one of the placements that convinced me that EM was for me (as I How often is it acceptable for me to be wrong?
was there for the TT races in 1991). As for the question, that’s
OK, if you think never, then you’re just not getting this, go back to the
somewhat the point, as risk is a personal choice, TT riders accept a
beginning and start again! We have to be wrong some of the time
level of risk, which is pretty much at the highest level I can think of.
when we are investigating for things like PE or ACS. The reason is that
Personally, I would not be caught dead on the back of a motorbike,
we cannot pursue these diagnoses to the absolute max as we
but put me on a road bike descending in little more than lycra, at
eventually end up causing more harm than good, by exposing patients
50mph and I am seriously happy. Doesn’t really make sense does it?
to diagnostic strategies (radiation usually), which are harmful in
Anyway, enough of bikes for now. What about risk in the ED? Well, if themselves.
you read the last post, I think I proved that our typical process of
So, are you happy with a sensitivity of 98%?

I reckon that you said yes. Not all of you would have said yes, but I
reckon that you would be reasonably happy with 98%, as that’s a level
that is commonly considered to be a SnOUT: so sensitive that if it’s not
positive it rules out the diagnosis. Great, but let me ask you the
question a different way.

diagnosis leads to a probability of disease rather than an absolute

12
How many patients are you prepared to send home with a missed The risk is…
PE?
• 1 person in your house
• 1:5
• 1 person in a school class
• 1:10
• 1:25 • 1 person on your road
• 1:50
• 1 person in a village
• 1:100
• 1:500 • 1 person in your town
• 1:1000
• 1 person in your city etc.
Not so easy now, I suspect. The answer you are looking for is, of
course, 1:50 as that is the same as a 98% sensitivity. If you are Now, PE is a particularly good example for us to think of, as the 98%
normal, the expression of this risk as a natural frequency, will not fit as sensitivity point is considered to be the point of clinical equipoise
comfortably as the more palatable figure of 98% sensitivity. Don’t where pursuing the diagnosis further is just not worth it. I think that’s
worry if you are uncomfortable with this. It’s normal for clinicians to be probably true for PE, but what other factors affect the clinical
more wary about accepting risk as a natural frequency. When I’ve equipoise point for other diagnoses? I would consider the following.
done this test at conferences, I get completely different answers,
• Consequence of a missed diagnosis
depending on whether I ask people for an opinion on; a test with 98%
sensitivity (they love it), or a test with an ability to spot 49/50 PE’s (not • The success of therapy for a confirmed diagnosis
as keen), or a test which misses 1 in 50 PEs (really not very keen at
all). • The risks of further testing

There is a whole world of literature out there on the subject of risk and • The temporal proximity of further risk

risk perception. I personally like to explain risk in terms of natural • The likelihood of getting sued (sad but true folks)
frequencies (I think this came from Ken Calman).
• Who’s risk is it anyway?

You may be thinking at this point that it’s just about whether you miss
it or not, isn’t it?

13
Not really. know about it really soon! Over-investigate with excess radiation, and
your patients might get more cancer in 10-50 years time. There is no
Consequence: There is clearly a difference in missing something like
doubt, that the former risk weighs much more heavily on the mind,
a subarachnoid bleed, versus missing a fracture of the lateral
and on the medico-legal insurance.
malleolus. One causes pain and delayed therapy, the other may result
in death. Clearly, it may be more acceptable to have a lower sensitivity Medico-legal factors: Many of us must face the fact that our practice
for less severe conditions. is influenced by the consequences to us as emergency physicians. We
are actually part of the decision making when we accept a diagnostic
Success of therapy: If we are to fret and worry about making a
process, that we know leads to the missing of diagnoses, exposes us
diagnosis, then it must be to some purpose. If there is no effective
to a risk of medico-legal action. This is particularly important to realise
therapy for the condition we are seeking to diagnose, then the process
in view of the temporal proximity of risk. You will get sued for missing
of diagnosis is somewhat useless. For example, I might choose to
a diagnosis this week, you won’t for giving someone cancer in 20
define the exact virus causing that nasty sore throat – but since my
years time. This is a real problem, as it skews your practice towards
treatment is going to be exactly the same, what’s the point? Similarly,
something that might be wrong, for the population you are
in conditions that are going to be fatal regardless of a diagnosis,
investigating.
should we pursue it?
Who’s risk are we talking about?: Thus far, we have really been
Risk of further treatment: EM physicians do not use gold standard
talking about risk from a physicians perspective, but if we leave the
tests. This is either because there are risks associated (angiography
medico-legal stuff to one side, it is clear that it is the patient that
for PE for example), or because we cannot achieve it in the time
experiences the risk. It is really about them at the end of the day, and
period available to us. Once you reach the point that further exposure
although we think about miss rates in terms of percentages, patients
to testing, results in more risk to the patient than just letting the patient
either experience the process of diagnosis working correctly or not.
go, you should stop (a major problem in litigious systems where errors
So, whilst we think about a % chance of getting it right, the patient
of omission are considered to be worse than those of commission).
only experiences a dichotomous outcome.
Temporal proximity: I’m sure that you are really interested in reducing
the harms from unnecessary investigations, or at least, if you are like
me, you are at this particular moment, whilst we are thinking about
populations and risks. In reality, the risks of investigation are not
linked in time to th patient in front of us. Miss a PE, and we might

14
So what?

So, in summary, risk perception and acceptance is really quite

complex depending on what you are looking at and who you are, and
even how it is expressed. Bear this in mind, as we move to the next
section to discuss the consequences of missing a diagnosis.

Simon Carley

15
Section 4: Written by Simon Carley

How many diagnoses do I really miss in the ED?

KEY POINTS

1. Prevalence has a huge impact on the

performance of diagnostic tests.

2. Prevalence often allows us to practice with

acceptable levels of risk.

3. We should practice great medicine, and not

medico-legal medicine.

16
This section looks at understanding risk and diagnosis in Emergency ambulatory end of the department who has some respirophasic
Medicine. In part three, we looked at how tests with apparently high (pleuritic) chest pain, some mild shortness of breath, but no major risk
sensitivities (e.g. 98% sensitivities) actually mean that you miss 1 in 50 factors, a normal chest X-ray and a desire to go home (which you
patients with disease. Now to be honest, if you just look at that figure share).
in the context of a disease like PE or Acute Myocardial Infarction, then
The exact prevalence of disease will depend on a number of things,
it makes for pretty uncomfortable reading. I know of few emergency
but we can use data from the literature to help us. Let’s take the
physicians who are immediately uncomfortable with that kind of figure,
incidence from Kline’s paper on the investigation of PE (6.9%), and
and to be honest even as I write it down it makes me stop and think
express that as a diagram based on 100 patients who might turf up in
again (and I know the answer to the question that follows).
your ED.
So, how can it be that we apparently miss 1 in 50 PEs in the ED, yet
The white squares represent patients who do not have PE, and the red
we are not in court every other week, or worse, in the coroner’s court
are those that do. Now of course, at
as a result of a patient’s death? To explain this we need to start
this point, you have no idea who has,
thinking about other factors that eventually affect the performance of a
and who does not have the disease,
test in the real world that we work in. The best way to do this, will be
so you are going to start
for us to start with an example. For ease of convenience, let’s stick
investigating them. You would not
with Pulmonary embolus. I’ve chosen PE as it is a topical subject and
want to send 6.9% of your patients
also the PERC rule to help illustrate the issues around risk. This post is
home with an undiagnosed PE!
not really about PE, more about how diagnostic test work in practice.
It’s just a convenient example….., oh and I hope my math is correct as
we go along. If its not, I’m sure you will let me know. 🙂
Right then!! Let’s get testing.
How likely is PE in the patients I see in the ED?
So, there are a number of things we
You should probably have a feel for this from your own experience. can do with our group of patients, but it’s pretty clear we need to do
What we are referring to is the prevalence of disease in the population something. Since we don’t want to do any invasive tests at the
of patients that investigate for PE. Probably best if we take out those moment we could start with the PERC rule as a way of starting the
in resus with no blood pressure. They are not the sort of patients that diagnostic process. (You might also want to whizz over to Life in the
concern us from a discharge perspective, as you are just not going to Fast Lane for a reminder as well). What happens if we apply PERC
send them home. Rather, we are thinking about the patient in the (lets take the original data at a sensitivity of 97.4%, a specificity of
17
21.9% and a prevalence of 6.9%) to the 100 patients who turn up in Key question that. We really need to get an idea of where those
the ED? How many of them end up PERC positive and how many original 6.9% of patients have ended up after the application of the
PERC negative? Again we can express this as in diagram form as PERC rule, and again we can get an idea by mapping them onto the
shown below. post-PERC diagram as shown below.

  So after applying PERC, the patients who do have a PE remain

Patients represented by the yellow circles (roughly 79% % of patients)

predominantly in the group who need further investigation and that’s
are those that are PERC positive and will need investigating further.
fine. However, there is a little bit of red left in the green zone, i.e. in the
Those in the green circles (roughly 21% of patients) are classified as
group that we are considering sending home with no further
PERC negative and are the ones that you might consider sending
investigation.
home with no further testing. [/DDET]

But what about the patients who DO have a PE?

So how many am I actually missing?

18
If you work the maths out (if you must, you can), it works out that Well, the key question is really to get an idea about the prevalence of
6.7% of your original 6.9% of patients are in the yellow zone and will disease in your population. Because of the way PERC rules, it will
hopefully be picked up by further investigation. In the green zone, that probably be similar, but that’s not the case in other diseases where
leaves just 0.2% of patients who had a PE. Why? Well basically, your prevalence can vary a great deal. Clearly, as the prevalence rises you
high 90s sensitivity only applies to patients who actually HAVE end up missing more patients per group of patients investigated.
disease. The sensitivity part of the performance of a diagnostic test is
In our case study around the PERC rule, if we were to increase the
only really relevant to patients with the target condition (in this case
prevalence to 30% then we would end up missing more patients with
PE). So basically, we capture the majority of patients with PE in the
PE (about 5 per 500 investigated) and at some point you would need
PERC positive group.
to say that this is not an acceptable strategy.
Great 🙂
If you want to play around with how prevalence changes how your
diagnostic test will work, according to the number of people
That’s just 1 patient missed for every 500 investigated as a low risk PE
investigated, then download this excel file and play around with the
patient. This is clearly more reassuring. At Virchester, we investigate
numbers on a hypothetical population of 1000 patients in your ED.
about 700 patients for PE every year, so we would expect to be
Just input the sensitivity, specificity and prevalence and you can see
missing 1-2 patients per year if we applied a strategy such as the one
how it will perform.
described above.
DROPBOX DOWNLOAD HERE
We think that’s pretty good and we will argue that it is an acceptable
miss rate. It means that there is a group of patients who do not need In Virchester, we really quite like this calculator, as to be honest, the
to go onto more invasive testing at this stage (so long as you are sums make our heads hurt and it’s quite useful to see how the
comfortable with that miss rate). Although we are still missing one in performance of a test changes with a change in prevalence. We prefer
50 PEs the number of PEs missed per person investigated is actually to see the effect as a natural frequency rather than percentages and
quite small. statistics. It makes it feel more real to us as clinicians and you can also
use it with patients if you are so inclined. If you want to work things
round the other way by filling in the event rates to see sensitivity and
specificity (plus a whole bunch of other stuff as well), then I would
suggest using this fantastic online calculator.
So can I use this in my population?

19
This is one of the many reasons why it’s so important to know about
the characteristics of your own patients when assessing new
diagnostic tests.

What if you miss the diagnosis in a lawyer?

A very cynical question if you don’t mind me saying so! However, it is

a fair question, as we have accepted that we are going to miss a
proportion of patients. We are going to have to balance the
convenience of our investigation protocol, which does not include
blood tests and radiology, against the knowledge that some people
will be missed. Can we justify this or do we have to accept that we are
doing the wrong thing some of the time? I put it to you that the
question is ‘what is the wrong thing?’ Have a think and then revisit this
in the next section.

Simon Carley

20
Section 5: Written by Simon Carley

Does a correct diagnosis mean therapeutic success?

KEY POINTS

1. A correct diagnosis does not mean the

treatment will work.

2. In fact patients can be harmed or helped by

a correct or false diagnosis.

3. Diagnosis and therapy are rarely studied in

the same trials (but they should be).

4. As we improve or change the performance

of a clinical test it may subsequently affect
the risk:benefit of therapy.

21
In the previous section we looked at how the performance of most of It’s what I was taught at medical school, and I’ve probably
the clinical tests we use in practice, mean that we inevitably have to promulgated it for many years. Why? Because it feels good to think
miss some patients who have the target condition we are looking for. like this. It gives the illusion of certainty and effectiveness.
We looked at PE and showed that a test with 98% sensitivity means
However, we now understand a lot more about the process and realise
that one in 50 people with the target disorder will slip through the net.
that the diagnostic process does not confirm or refute a diagnosis,
BUT because only a minority of patients that we investigate actually
rather it moves probabilities around so that the true picture is more
have PE, we get away with it most of the time.
complicated.
I reckon that you’re still worried about those misses though….
OK. So this is fine and dandy, but let’s challenge the next assumption
If you are then I think that’s fair enough. Few clinicians find this about the outcomes from diagnosis. In talking to clinicians, patients
reassuring, but let’s explore what we mean by a ‘miss’. and lawyers the next assumption is that the correct diagnoses lead to
benefit and false diagnoses lead to harm. Graphically we can draw
Is missing a diagnosis always a terrible thing to do?
this out as follows.
Clinicians generally start off with a fairly simplistic view about
diagnosis and therapy and it looks a bit like this.

This too is still far too simplistic though. If you have been following any
of the recent debates around the use of thrombolysis in stroke you will
be all too aware that therapy itself has inherent risks, particularly when
we consider therapies such as thrombolysis, operation (e.g.
appendectomy), cardiac catheterisation etc.

22
As an ED clinician this is an area where we just need to stop and think The patient had the opportunity to experience each of these outcomes
about how we, as clinicians view outcomes from therapy differently with therapy, but interestingly there are also three potential options for
from how patients view outcomes. Our tendency is to look at those patients who declined therapy.
populations of patients but patients don’t really care that much about
• They can experience an adverse outcome from their MI which
other patients (Ed – harsh but I think I know what you mean). Rather,
might have been avoided by thrombolytic therapy.
patients want to know what’s in it for them. As an example, we can
• There would have been no difference either way. Treatment or no
return to the good old days of thrombolysis (still being used in remote
treatment would have made no difference.
settings of course) when I used to have a rehearsed and regular
• They can avoid the complication of therapy by declining the
conversation with patients before starting thrombolytic therapy. The
treatment.
aim was to to explain that there were three possible outcomes from
the proposed treatment.

• Thrombolysis could improve their outcome both in terms of

survival and in terms of longer term cardiac function.
• It might make no difference at all.
• It might cause harm in terms of death, stroke or bleeding.

23
In other words, there were potential risks and benefits regardless of Well, yes of course. Diagnostics are important but it should now be
whether a patient did or did not have therapy. As clinicians we do not clear that simply getting a result from a test is only the start of the
typically consider all of these potential outcomes (well I don’t anyway). process for patients. If we are to understand if patients are to benefit
from the diagnostic process then we will be better served by taking a
Now….stop and think back to those conversations….did you, or did
more utilitarian approach to diagnostics. Many diagnostic studies in
you ever overhear, someone (it could have been you), say to a
the literature are designed to answer the simple question regarding
patient…
whether or not the patient has the target disorder. The diagnostic
“If you don’t have the clot buster you’ll die from your heart attack” cohort is the typical model and produces familiar data such as
sensitivity and specificity. This is fine, but it does not answer the
I heard this many times, and still do, for a variety of different
question about whether a diagnostic test actually benefits patients.
treatments, but the fact is that this is a lie. It gives an illusion of
certainty that simply does not exist, as everyone who has ever visited If you have never read the paper by Foex and Body from the EMJ on
the NNT will know (it may also interest you to know that at the time we the philosophy of diagnosis then click on the abstract now.
were doing this the NNT for thrombolysis of an inferior MI was over
100).

So how does this pan out graphically? We can put the various
outcomes into a diagram that shows the complexity of the potential
outcomes that are possible when we consider diagnoses where the
diagnostic process is imperfect and where the therapy and/or the
diagnostic process contains inherent harm. Examples in emergency
But surely making a diagnosis is always good isn’t it?
medicine practice are very common with the investigation of pleuritic
or cardiac sounding chest pain being perfect examples. All the Most of the time, that is certainly the case. A diagnosis is usually a
potential outcomes described below are possible for your patients good thing to know, but perhaps not always, as testing becomes ever
going through a diagnostic process for the investigation of PE or acute more accurate and our ability to pick up sub clinical disease rises. A
coronary syndrome. good example here would be PE where we are increasingly able to
pick up tiny PEs in the lungs. The clinical significance of this is being
Seriously?
questioned for PE and in many other areas such as high sensitivity
Where does this leave us now? Do diagnostic tests work at all? Troponin in chest pain (though I am convinced to be honest).

24
If we recognise and understand the link between diagnosis and
therapy, and if we are able to balance the potential benefits and harms
from the diagnostic process then we will be able to achieve better
outcomes for patients.

But, surely making the diagnosis is enough.

Aren’t RCTs nice but unnecessary?

This is a common complaint/theme/anxiety amongst diagnostic

researchers who argue that diagnostic tests can be evaluated in
simple trials, which are cheaper and easier to perform. Whilst that is
true I increasingly believe that diagnostic cohort studies can only take
us so far, and there are increasingly calls for RCTs to be used more
frequently in the evaluation of diagnostic testing. There’s a good open
access review here from the Ottawa thrombosis program (who know a
fair bit about this sort of thing).  You can also have a look at the
excellent paper from the UK Sheffield group on the evaluation of rapid
assessment cardiac panels in the RATPAC trial as an example of an
ED based study.

In summary, we really need to think carefully when considering how

we use diagnostic tests, how we communicate information to patients
and how we reassure ourselves that the diagnostic process actually
leads to patient benefit.

Simon Carley

25
Section 6: Written by Simon Carley

How many steps to disaster

KEY POINTS

1. We all miss diagnoses.

2. The consequences of a miss are not always

a disaster.

3. Some misses are good.

4. We need to find a balance between over

and under diagnosis in clinical practice,
there is no perfect point for many
diagnostic tests, but there is a ‘best’ point.

26
Let’s accept that most diagnostic processes ‘miss’ some patients, but clinicians is that harm will then happen. A missed diagnosis surely
what do we mean by a miss exactly? Is it always a terrible thing and means that we have lost the opportunity to make the patient better,
why is it that if we constantly miss diagnoses (we do you know) we are but hang on a minute. In the last section we talked about how many
not in court every week? Hopefully, if you have read the previous therapies (such as thrombolysis) have an inherent harm within them,
sections, then you are now absolutely convinced that error is an so it’s not so clear cut as we might have thought. Diagnosis is,
integral part of the diagnostic process. Arguably, it is so much a part arguably, no guarantee of success.
of the diagnostic process that I don’t really consider it error any more.
Let’s stop and think about what needs to happen for true patient harm
It is inevitable that some patients will slip through the diagnostic net
to take place. Perhaps we can think of this as a series of steps. For
when you see them. You will not diagnose them, they will appear to
serious harm to take place, a number of things need to happen.
you, to be free of disease and you will, no doubt, reassure them and
yourself that all is fine and dandy. Both you and your patient will
hopefully sleep well that night.

Just occasionally though, just once every so often, you will wake
about 3am and wonder…., ‘I wonder if that patient was the one that
slipped through the net?’, or ‘I wonder if I might call them tomorrow to
check they are OK?’ I’ll give a pound to any emergency physician
who has never woken with such thoughts, and I’m pretty certain that
the pounds will stay in my pocket.

But a miss is a miss isn’t it?

Is a miss, like missing a penalty kick in the World Cup final (1994,
 
Roberto Baggio) and handing the title to Brazil? You an watch the
video here. I think everyone would agree that this was a huge miss Step 1. The patient needs to get worse.
with immediate and long lasting consequences.
Pretty obvious if you think about it, but not intuitive. Many conditions
What does this football analogy tell us as clinician in the ED? that we see in the ED are self limiting, even potentially serious ones
such as DVT/PE or even some acute coronary syndromes (depending
If we don’t ‘make’ a diagnosis, what are the potential outcomes for the
on your definition) can resolve spontaneously with no long term
patient who was missed? The natural assumption amongst most
27
sequelae. Infections such as pneumonia could go either way but a is an admit/senior review until proven otherwise. Returns are high risk
significant number of patients will get better spontaneously from a patients.
whole range of infectious diseases, cardiovascular conditions and
 
trauma.
Step 4. You have no treatment on return
 
So, when they do come back, for things to get really bad, there has to
Step 2. The patient needs to not come back.
be nothing that we can do to make it better. A patient with a missed
Most conditions get worse over a period of hours/days and the patient MI might come back in cardiac arrest and not survive, which would be
will develop new or worsening symptoms. Patients with infective awful, but those cases are rare. More commonly a patient will return
disease are classic for this, even when they have significant disease. with a worsening of disease. A missed chest infection may turn to
Almost all other conditions will ‘usually’ worsen, but sadly, not all. pneumonia, a wound infection to an abscess. Whilst it would have
There is a proportion of patients with conditions such as ACS, PE, been better for your patient to have been treated at the first
SAH where sudden, rapid and fatal deterioration may take place. opportunity, there are still therapeutic options and in the vast majority
There is little that can be done in these circumstances, but believe me, of cases they will get better.
the incidence of this is rare in comparison to the number of patients
In other words, even if a patient comes to the ED and you do not
we see. So basically, most ‘missed’ patients who get worse will come
identify their underlying condition (and if you have read the other posts
back.
in this series you will know that this HAS to happen) then it does not
  mean that disaster will ensue. Most of the time there are either no
consequences at all, or, the patient will deteriorate and return within a
Step 3. You need to miss it again.
time frame that gives you opportunity to intervene and treat the
Possible. It is possible to make the same mistake twice. Indeed there condition.
is something about our pride as physicians which is challenged by a
The odds then are very much in your favour, even with the necessity of
patient returning with the same problem that we have already ‘ruled
accepting the fact that we miss diagnoses we can find some solace in
out’. Experience has taught me that pride is not a good feature for an
the odds that such misses do not lead to disaster. Most patients will
emergency physician. Any returning patient should be considered a
safely traverse a number of steps to safely reach the other side of their
‘red flag’. In general terms, I teach our juniors that a returning patient
illness. Does this relax you? Does this make you complacent? Well
perhaps. It makes me a little more relaxed about the whole uncertainty
28
of the diagnostic process but I’m not sure that my patients see it that
way. Thus far we’ve not really considered the patient, but they must
surely feature somewhere and they do…..in Part 7.

Simon Carley

29
Section 7: Written by Simon Carley

Risk Proximity

KEY POINTS

1. If the outcome from a risky event is likely to

happen soon, then we value it more highly
than if it were to happen in the future.

2. Clinicians may make poor decisions if they

think that they may be blamed for the
inevitability of probabalistic errors.

3. Over investigation may protect the

reputation of the clinician at the expense of
harm to the patient.

30
Risk proximity is an important concept. In simple terms, it just that last week the department sent home a patient who then returned a
describes the time from an event happening and the risk potentially week later with a skull fracture on CT. Apparently, an incident form was
occuring. It’s not something that we often think about in the submitted and it’s all being investigated.
emergency department, but it’s important and influences our decision
Both these examples are arguably violations of routine practice but I’ll
making processes. In this post we begin to delve into why risk
bet a fine English pound that we’ve all experienced similar episodes.
proximity may make some of our ED decisions a little less rational than
There are, of course, many reasons why this might happen, ignorance,
we might hope.
error, bad medicine or laziness but another factor that I see in the ED
Let’s think of a few examples. relates to perceptions around risk proximity.

Mr X is a 35 year old postman. He goes to the gym regularly and

competes in Triathlons. He has been training quite hard recently. He
presents to the ED with left sided respirophasic chest pain. One of the
junior docs asks you to talk through the case. You listen to the case
and he appears to be low risk, in fact he is PERC rule negative. You
suggest that he can safely discharge the patient without further testing
when you hear…..’but I’ve just sent a d-dimer, do you think we should
wait, just to be sure?’

Little Jimmy is an 8 year old boy who has been hit by a car. The
description from the scene is that the car was moving slowly, maybe
10-15mph and that Jimmy was ‘clipped’ rather than fell onto the curb.
He was not knocked out, has no external head injury and although he Risk proximity simply means how close we are, in terms of time, to a
vomited soon afterwards and had a sleep in the car on the way in to risk occurring. In project management this is used to ensure that focus
hospital, he does not meet criteria for a head CT. You reassure the on risks is balanced with a greater emphasis on those risks that are
parents that no further imaging is required and suggest that he just likely to occur in the short term. In medical decision making we see
needs observation at home. They look at you oddly and ask what the similar behaviours with a focus on those risks that are likely to occur
CT scan shows. You realise that a colleague ordered a head CT from sooner rather than later, but before we explore that further it’s worth
triage. After a rather embarrassing conversation with the parents about stopping and thinking about who’s risks we are managing..
the normal CT you ask why it was requested. Your colleague tells you
31
Who is really at risk when we make a decision in the ED? Dr A may appear to be a great clinician and his patients may love him.
If we assume that there are good mechanisms for reading those
Obviously, it’s the patient. In the examples given above, the first
X-rays then no injuries will be missed and no patient will return after a
patient may have a PE, the second might have a significant head
week with a quinsy or pneumonia, complaining that they were not
injury. These risks are low but they are not zero and so the patient is at
given antibiotics at an early stage of their illness.
risk if they do not get an investigation. However, whilst we might frame
the risk in terms of the patient, it is not the patient who typically makes Dr A will have few complaints and a file full of thank you cards. Their
the decision. It’s the clinician who decides whether to proceed with a practice will of course be expensive and much harm will come to their
test and their decision is also influenced by a personal risk. patients but that harm will be hidden from view.

None of us wants to be Dr A, but let’s face it, there are temptations. If

you’ve ever missed a diagnosis, or rather when you become aware
that you’ve missed a diagnosis, you will know that your behaviour and
investigation thresholds change for the following days, months, even
years.

As a thought experiment we can consider Dr A who always sends

blood tests, gets X-rays for everyone, and if a CT might be indicated
he orders it. If you’ve got a sore throat then have some antibiotics, bit The bottom line is that overinvestigation protects the reputation of the
of a cough then more antibiotics, and if you have some pain this clinician. In other words it’s much easier to get ‘pinged’ for failing to
Codeine should sort you out perfectly. Is Dr A a great clinician? Well investigate than for over-ordering tests which are not really indicated.
no, if you’ve read the rest of this series then you will know that there As a result, the decision to order an investigation is determined by a
are major harms associated with overinvestigation. The paradox is that combination of the risk to the patient (of having the disease) and the
risk to the clinician (of missing an important diagnosis).
32
What about patient harm? and departments, but in the UK where staff are often transient and
junior, it may be very difficult to see patterns in time to control them.
Patients can be harmed through the diagnostic process in many
different ways as other articles in this series have described. In terms So what does this all this mean for the ED clinician?
of risk proximity, time has a significant impact in how we understand
It’s complex. Emergency medicine is a speciality where the conduct
and perceive risk within the diagnostics process.
and understanding of the diagnostic process is at the very core of
In the short term, there are clearly the risks of missing an important what we do. We need to understand how the concepts of risk
diagnosis. In the examples at the beginning of this post we might miss proximity may affect our own judgement and the judgement of others.
a PE with our patient returning within weeks suffering from a big PE or It should be a factor when we consider how to structure our diagnostic
even death (although this is really unlikely). Similarly, missing a skull strategies (clinical decision support guidelines are an obvious solution
fracture or small intracranial haematoma in a child may be perceived which can be audited) and can aid us when we investigate critical
as a terrible miss (with the blame squad arriving mob handed to find incidents and our reactions to them.
out who is to blame).
The bottom line is that risk is divided between clinician and patient.
The other risks of overdiagnosis are, rarely if ever, linked by patient or Risks are not always realised at the same time and many of the harms
clinician. If I decide to CT all children with head injuries from this day that can result from over-investigation (and subsequent unnecessary
forth I will no doubt cause many cancers and cataracts in my patients. therapy) will not be temporally linked by either.
However, these complications will take place many years in the future.
It’s might be worth reflecting on this the next time you see an
I will never see them and it’s highly unlikely that the patient will ever
unnecessary test ordered in the ED.
associate an unwarranted investigation with a future disease.
Simon Carley
Similarly, overinvestigation can lead to an increasing number of false
positives. In past posts we have discussed the problems of
overdiagnosis in PE leading to an increased number of patients taking
anticoagulants. If they then have a bleeding complication it is almost
impossible to link that complication back to the original poor decision
to overinvestigate the ED patient.

In managed health organisations, or where there is close oversight of

clinical practice, it may be possible to see trends between clinicians
33
Chapter 2

Metacognition
in EM

Metacognition is simply thinking about thinking.

Arguably this is the core skill of emergency
medicine: the ability to think, consider evidence and
make decisions. In this chapter we explore some of
the more complex elements of metacognition in
emergency care.
Section 1: Written by Simon Carley

What is Gestalt?

KEY POINTS

1. Gestalt means shape or form.

2. The term originated in the psychological

literature concerning our perception of the
world.

3. In medicine the term is used beyond it’s

original derivation to encompass elements
of practice such as clinical judgement.

4. Gestalt features in a number of supposedly

objective scoring systems used in EM
practice.

35
 Gestalt originated in the psychological literature towards the beginning
of the last century. It was a counter to the structuralist arguments
If you follow #FOAMed conversations on twitter then you will have
promoted towards the end of the 19th Century. Structuralist
come across the term Gestalt a lot. Many conversations about clinical
approaches suggest that everything can be broken down into
decisions hinge around this elusive, slippery and somewhat obscure
individual components, thus by knowing the nature of constituent
term. Perhaps it is in a rather circular manner something that is difficult
parts we can understand the whole. For example, a structuralist
to define, yet we know it when we see it….., or perhaps we are merely
approach would suggest that by understanding all the aspects of
using the wrong term.
automotive engineering, we can understand what a car is. Gestalt
This section is an introduction to thinking about perception, would argue that we perceive a car as a whole and not as
interpretation, judgement and clinical decision making. components. Gestalt states that our minds organise information to a
global perception rather than by assessing each individual element, in
So for starters, do you agree with Linda?
other words ‘The whole is greater than the sum of its parts’.

Properties in Gestalt systems

Much of the literature around gestalt focuses on visual perceptions

and there are several examples around that you may be familiar with.
There are four properties in gestalt systems, emergence, reification,
invariance and multistability.

It’s time to find out.

What does Gestalt mean?

Originally Gestalt means ‘shape or form’. It’s a German term that has
come to be associated with a theory of perception and understanding.

Where does the term come from?

36
Emergence Reification

Emergence is the phenomena where a complex pattern emerges from Reification is the process by where we construct spacial relationships
simpler forms. Perhaps the best known example of this is the following from elements outwith those presented. Several examples are shown
picture of a Dalmatian dog. There is clearly no picture of dog here, but below where geometrical objects are perceived by the relationship
our minds are able to form the final image from the component parts. with the components presented, yet they do not actually exist.
None of the components in itself is dog like, yet the sum of the image
gives a dog sniffing the ground

37
 Invariance

Multistability Invariance is the property whereby objects are perceived as the same
despite differences in relative shape, size, rotation scale or aspect.
Multistability takes place when we can perceive an object in a variety
Similarly we perceive the same shape regardless of environmental
of different ways; it’s a common visual puzzle that you will be familiar
effects such as setting, time of day, location etc.
with. Examples below include the Necker cube and the Rubin vase,
but there are many others.

38
Most work in Gestalt has focused on the visual aspects of perception  that all look very similar of fist assessment. Few patients are precisely
but it can be recognised in other senses, such as through the the same yet, somehow we manage to make a similar diagnosis
interpretation of sound and music. (invariance).

Do such visual elements of Gestalt exist in medicine? How do we use it medicine?

Arguably, they do. We build towards a diagnosis from elements of its We seem to use it a lot. Gestalt is often thought to be similar to
form rather than an entire picture or a consistent form. Diagnoses ‘clinical judgement’, though we need to think that through a little more
deeply later. It is referred to in conversations between clinicians and
also appears in a number of clinical decision rules.

Several clinical decision rules appear to encompass Gestalt. For

example, the  PERC rule is designed to be used in a population that
the clinician believes to be low risk; it then goes on to list a number of
other tangible features that also determine low risk. So, if the objective
measures are evidence of low risk, what then is the original perception
of the treating clinician that this is a low risk patient? Similarly, in the
Wells DVT score, we assign a score of -2 for patients in whom we feel
an alternative diagnosis is likely. This soft perception on the part of the
clinician is again arguably embracing Gestalt within a scoring system.

If we are looking for a simple definition of Gestalt, then we can simply

consider it to be a sensory interpretation that is greater than the sum
of its parts (a concept which predates Gestalt and which dates back
to Aristotle). In clinical terms, we can define it thus: ‘A structure,
emerge from individual components that we bring together to ‘form’ configuration, or pattern of physiological, biological or psychological
that which we recognise and subsequently label (emergence/ phenomena so integrated as to constitute a functional unit with
reification). We are often faced with constellations of symptoms and properties not derivable by summation of its parts.’
signs that may represent one condition or another. For example, you
have probably met conditions like COPD, Heart Failure or Pneumonia,

39
So what might this mean in real terms? Well, I often use the silly walks http://commons.wikimedia.org/wiki/File:Silly_Walk_Gait.jpg
example. You are walking through resus and you walk past bed 1, bed
experts resort to systematic enquiry. There are many, many other
2, bed 3, then suddenly turn on your heels and return to bed 2.
examples within our practice that both help and hinder us as
Something is up…, but you’re not sure what. Something about the
clinicians, but what is not in doubt is the importance to clinical
noise, sight, smell, atmosphere tells you that something is not right.
decision making inherent within the realms of perception,
The observations/monitors are either irrelevant or contradictory. You
interpretation and judgement.
just know…, the outcome is greater than the sum of the perception.
Gestalt and the immeasurably measurable.
We also use Gestalt daily in the interpretation of ECGs in the ED.
Whilst we may teach an atomistic approach to our medical students; If Gestalt is truly greater than the sum of its constituent parts, then this
calculate the rate, rhythm, size, relationship etc. As experienced can only be the case where all parts are perceived, considered and
clinicians though, we do not do this at all. A simple observation of a valued. That is not, however, a characteristic of medical education. In
cardiologist or emergency physician reading an ECG is an example of our patients and clinical studies we have a propensity to measure and
heuristics, gestalt and judgement over a constructivist approach to value what is measurable. Blood pressure, heart rate, respiratory rate
are quantative values which we can define and share. However, other
elements of assessment which are arguably highly valued by clinicians
such as agitation, sweating, distractability, attentiveness are difficult to
measure, but are perceived by clinicians. Ask an experienced clinician
to look at a patient in the resus room who has an abnormal respiratory
pattern, sweating, agitation. They will comment and place value on
these findings, however, they will not commonly feature in clinical
scores (which favour objective data). This is referred to as Gestalt, but
it is not. The signs are there, they are perceived, and indeed,
articulated at handover between the resus teams, but they might not
make it onto the nurses observation chart.
data interpretation. Only when pattern recognition and gestalt fails, do
Beyond that, are more subtle clinical signs such as distractability and
Jazeen Hollings other responses that may be difficult to explicitly perceive and share,
but a clinician may sense and use in formulation without perception.

40
  Summary.

So Gestalt in its purest sense, is somewhat contrary to the positivistic

pseudo-scientific view that many clinicians hold onto as a model for
their practice. Gestalt is one element and pathway that links the
acquisition of data to processing and ultimately to clinical judgement
and decision making. In its simplest sense, it is an assessment that is
greater than its parts, but in the world of emergency medicine the term
is often confused with the wider realm of clinical judgement.

 Simon Carley

R e f e r e n c e s / F u r t h e r R e a d i n g


41
• Wikipedia Gestalt Psychology predict disposition of psychiatric patients based on a brief
• Cervellin G1, Borghi L, Lippi G.  Do clinicians decide relying medical evaluation?
primarily on Bayesians principles or on Gestalt perception? • Gerd Gigerenzer and Wolfgang Gaissmaier. Heuristic Decision
Some pearls and pitfalls of Gestalt perception in medicine.Intern Making Annual Review of Psychology Vol. 62: 451-482 (Volume
Emerg Med. 2014 Aug;9(5):513-9. doi: publication date January 2011) DOI:
10.1007/s11739-014-1049-8. Epub 2014 Mar 8. 10.1146/annurev-psych-120709-145346
• Giovanni Casazza • Giorgio Costantino • Piergiorgio Duca • STEPHEN G. HENRY  RECOGNIZING TACIT KNOWLEDGE IN
Clinical decision making: an introduction. Intern Emerg Med MEDICAL EPISTEMOLOGY Theoretical Medicine and Bioethics
(2010) 5:547–552 (2006) 27:187–213
• Gunver S Kienle MD and Helmut Kiene MD Clinical judgement • Anderson RC, Fagan MJ, Sebastian J. Teaching Students the Art
and the medical profession. Journal of Evaluation in Clinical and Science of Physical Diagnosis Am J Med. 2001 Apr
Practice 1;110(5):419-23.
• Andrea Penaloza, MD; Franck Verschuren, MD, PhD; Guy Meyer, • Christopher A. Feddock, MD, MS  The Lost Art of Clinical Skills
MD; Sybille Quentin-Georget, MD; Caroline Soulie, MD; Frédéric T h e A m e r i c a n J o u r n a l o f M e d i c i n e

Thys, MD, PhD; Pierre-Marie Roy, MD, PhD Comparison of the Volume 120, Issue 4, Pages 374–378, April 2007 
Unstructured Clinician Gestalt, the Wells Score, and the Revised
Geneva Score to Estimate Pretest Probability for Suspected
Pulmonary Embolism Annals of Emergency Medicine.
http://dx.doi.org/10.1016/j.annemergmed.2012.11.002
• Julian N. Marewski, PhD; Gerd Gigerenzer, PhD Heuristic
decision making in medicine. Dialogues Clin Neurosci. Mar 2012;
14(1): 77–89.
• Body R, Cook G, Burrows G, et al. Can emergency physicians
‘rule in’ and ‘rule out’ acute myocardial infarction with clinical
judgement? Emerg Med J Published Online First: doi:10.1136/
emermed-2014-203832
• Michael J. Dreschera, Frances M. Russelld, Maryanne Pappasb
and David A. Pepperc  Can emergency medicine practitioners

42
Section 2: Written by Iain Beardsell

When we are the diagnostic test

KEY POINTS

1. All Doctors are Jackasses.

2. Our personal state of mind influences our

diagnostic performance.

3. Learn to focus your attention when faced

with a difficult cognitive task.

4. Beware the patient you don’t like.

5. Learn to recognise and adapt at times

when you might not be performing at your
best.

43
Simon’s recent post about Gestalt, has, like many of you, got me
thinking. What is that “end of the bed-o-gram” that we place so much
value on? How do we know if it is accurate? Can we teach and learn
these skills? What are the characteristics of this diagnostic test and
can it be relied upon?

Perhaps, even more important for me, is how we know when our
gestalt sensor is slipping, when we are not at our best and when other
outside forces are subconsciously altering our own sensitivity and
specificity. I, and others, have written at length about the
circumstances we all have to work in: the never ending patient load;
the overcrowding; the constant worry about making a mistake.

On the amazing trip to Fiji I shared with Nat May, Nick Jenkins and
Anne Creaton in 2015, I accidentally came out with something more
profound than my usual witterings (it must have been the Fiji Gold) [Ed
– one of my great twitter skills is making you sound smart! – Nat]
Chris Nickson Doctors are Jackasses
Over my comparatively short career as an Emergency Physician, I can Social Media and Critical Care
recollect many occasions when I haven’t quite been at my best as a Play
diagnostic clinician. Usually, these pass without incident and In a busy ED there are many times when I rely on my gut instinct to tell
unnoticed to me as well as others, but I know there have been times me when something doesn’t fit. It is that hard to define mix, that
when the extraneous stresses have influenced my decision making Simon describes so well:
and the accuracy of my gestalt. In the #FOAMed world there have
been many excellent talks about these “biases” and how we, as “Something about the noise, sight, smell, atmosphere tells you that

clinicians, think. One of the very best comes from Chris Nickson at the something is not right. The observations/monitors are either irrelevant

first SMACC conference – if you haven’t watched this video yet, you or contradictory. You just know…, the outcome is greater than the sum

really must. And if you’ve seen it before, you should probably watch it of the perception.”

again. Click on the link below to watch the video on the SMACC site.

44
What about when I am tired, hungry and distracted by external were told your blood gas machine wasn’t calibrated properly and was
stresses: the worry about a family member’s illness; a concern about giving less accurate results you would stop using it until it was back to
your child’s school performance; an upcoming birthday where you’ve the expected standard and we must regard ourselves as the same. We
forgotten to buy a present. The daily burdens of everyday life are are a diagnostic instrument that needs to be cared for with internal
sometimes hard to ignore. Add in the large numbers of patients in the checks and balances to recognise when it is not working before it is
ED all vying for your attention and the managerial pressure of a too late and a fatal error has been made.
government target and you have a heady mix that inevitably alters
There are a few techniques I have developed over the years to try to
your ability not only to sense when the pattern doesn’t fit, but also to
combat this cognitive fatigue when I recognise it and I would be
find the strength to do something about it.
interested to hear others’ ideas:

• To avoid distraction when listening to a colleague present a

patient I’ll walk with them around corridors of the department, or
even go outside into the ambulance bay. This helps me focus my
thoughts on what I’m being told and give them my full attention.

• There are some patients who we instinctively don’t like and I

believe our gestalt is affected accordingly. As taught by a senior
colleague, Mike Clancy, I spend twice as much time with them
and deliberately remove emotion from the consultation (and
gestalt is surely affected by emotion), perhaps depending more
on objective testing than would normally be my practice.

• Beware of any patient whom you see near the end of a shift.
Reducing the patient load for your colleagues may seem to be a
It is vitally important that we recognise when this is happening to
help, but you are not going to be as thorough when the shift
ourselves and those around us. Time constraints demand that we
clock is against you and handovers of partially completed
make quick decisions using efficient processes and for many of us
assessments present a danger of their own. Use the last minutes
that includes an element of gestalt. We must be vigilant and only use
of your shift wisely.
this diagnostic test when it is functioning at its best. After all, if you
45
 • Festive holidays, especially Christmas, can alter our judgement.
All our patients would like to be with their loved ones at these
times, but serious disease is no respecter of the urge to open
presents and eat turkey.

• Admit to others when you are not at your best. This could be
due to a poor nights sleep or a difficult situation outside work,
but asking a trusted colleague to keep an eye on you and make
sure you decision making is sound can be very reassuring.

Undoubtedly, gestalt is a valuable tool in the Emergency Physician’s

armoury, but like all diagnostic tests we must only use it when it is
functioning at an acceptable level and we have a duty to all to
recognise when it is not.

Iain Beardsell

46
Section 3: Written by Simon Carley

Do risk factors really factor?

KEY POINTS

1. Risk factors do not predict pathology in the

way that we think they do.

2. Acute coronary syndrome diagnosis in the

ED is inversely proportionate to the number
of risk factors the patient has.

3. The importance of many risk factors may

not be as significant as we think.

4. Use this section to reflect on how you use

risk factors in your diagnostic processes.

48
We’ve been involved with the SMACC conference from it’s inception in and reflect on why we make the decisions that we do in the resus
2013. We believe it’s the best conference in the world for anyone room and emergency department.
involved in resuscitation, and it’s not just about the latest and greatest
Click on the link below or picture above to visit the SMACC website
bits of kit, or the latest airway technique. There is plenty of science,
and watch the video.Simon - Carley Do Risk Factors Really Factor?
encompassing cutting edge clinicians in emergency medicine, critical
Social Media and Critical Care
care, prehospital care and education. At St.Emlyn’s we sometimes
take a more philosophical view, taking pride in thinking deeply about
what we do and why. That’s why I was asked to speak in 2015 on the Hopefully this presentation will make you stop and think about how we
Gold Coast on risk factors and how we use them in emergency use risk factor information in the ED. The bottom line is that the
medicine. strength of risk factors in the ED is perhaps not as powerful as we
think. That’s a problem if they have too much influence on our
decisions. In practice it’s more risky for us when we ignore clinical
symptoms because risk factors are absent. In cardiac disease it’s
pretty clear that this would not be a great plan.

Simon Carley

This section links to a talk about thinking about our thinking,

meta-cognition if you prefer. As emergency physicians our practice is
based on our ideas, interpretations and beliefs and in this presentation
I hope that by talking through the importance of chronic health disease
risk factors in emergency medicine it might give you pause to think

49
Section 4: Written by Rick Body

How accurate is clinical judgement for acute coronary syndromes?

KEY POINTS

1. Lots of risk factors do not rule in ACS.

2. The character of pain is a useless test in

ACS.

3. Doctors are a bit rubbish when it comes to

the Gestalt of ACS.

Anyone who’s worked in Emergency Medicine for any length of time will appreciate that
an acute coronary syndrome (ACS) is one of the most difficult diagnoses to rule in or rule
out in the Emergency Department. We’ve probably all seen the old stat that 2% of all
acute myocardial infractions (AMIs) are missed in the ED. You might also have seen the
stat that up to 7% of patients discharged from the ED have prognostically important
myocardial damage, which by today’s standards would be considered as acute

50
myocardial infarction. Perhaps that’s why missed ACS is one of the should have been able to know that if your gestalt had been a bit
leading causes of medical litigation. better or if you’d have been just a bit bolder.

What William Osler, King of Gestalt, had to say about clinical However, the literature is quite clear about this. If the patient’s
judgement for acute coronary syndromes symptoms consist of acute discomfort or pain in the chest,
epigastrium, jaw, neck, throat or arms and you haven’t otherwise
explained it, you’re right to consider ACS – even if the symptoms may
not seem so convincing. This is based on good evidence that patients
with ACS often have atypical symptoms. Taking individual symptoms,
for example, the character of the pain, the radiation of the pain and the
number of risk factors a patient has don’t affect the probability of ACS
to any significant extent, as you can see from these slides taken from
my talk at SMACC Gold (based on our own research – which you can
find here).

Why is this such a hard diagnosis to make in the ED?

Maybe we struggle because, as doctors, we like to use our clinical

judgement. We like to think that we can use the clinical information we
have to make diagnoses without having to rely on tests. Perhaps we
also feel like we lose face if we admit patients for investigation, when
patients don’t actually have the diagnosis we investigated them for.
Don’t you hate it when someone from an admitting team catches you
in the corridor and says something like, “Hey, you remember that
patient you referred to me yesterday with query ACS? We sent him
home – it was just gastritis”? It makes you feel inferior – like you

51
 So what did we do in this research?

We ran a cohort study. The main aim of the study was to validate the
MACS decision rule but we also ran several substudies using the data
collected. In this one, we wanted to know about the diagnostic
accuracy of emergency physicians’ clinical judgement for acute
coronary syndromes – both alone and in combination with the tests
available on arrival – troponin and the ECG. To be honest, we didn’t
expect to find a useful ‘rule out’ strategy – we simply wanted to
quantify the accuracy to better inform our practice and to find out
whether we might be able to refine and improve the MACS rule by
incorporating gestalt.

In this study, we asked emergency physicians to record their ‘gestalt’

Even grouping symptoms together as ‘typical’ or ‘atypical’, the
or overall clinical judgement for ACS using a 5 point Likert scale (from
patients with atypical symptoms are no less likely to have ACS than
‘definitely ACS’ to ‘definitely not’ ACS. We then cross-tabulated this
those with typical symptoms – as shown this great study from Louise
with the patients’ outcomes – and the outcomes we were interested in
Cullen’s group, which I was honoured to be involved with.
were a diagnosis of AMI and the occurrence of major adverse cardiac
events (MACE) within 30 days.

What did we find?

The bottom line is that, for patients in whom we’ve already considered
the possibility of ACS as a diagnosis, our ‘gestalt’ regarding the overall
probability of that diagnosis can’t be relied upon by itself to rule out
that diagnosis. Nor can gestalt be used to ‘rule in’ the diagnosis – only
half of those whom clinicians felt ‘definitely’ had ACS actually had AMI
or developed MACE within 30 days. Here’s another slide from my
SMACC Gold talk to illustrate that.

52
 at the overall diagnostic value of gestalt combined with the ECG and
initial troponin level.

What we found may seem surprising. If we discharged patients who

had a normal ECG, a normal troponin and we felt that the diagnosis
was ‘probably not’ ACS, then we wouldn’t have missed a single case
in this cohort – the sensitivity was 100%.   Almost a quarter of
patients could have been discharged using that strategy.

If we used a high sensitivity troponin assay rather than a standard

contemporary one, we could also have discharged patients in whom
we felt the diagnosis ‘could be’ ACS without missing any cases –
100% sensitivity. Using this strategy, over 40% of patients could
potentially have been discharged – without missing a single AMI
i n t h i s c o h o r t .

And one of the really great things about this strategy, is that it didn’t
seem to matter if it was an experienced consultant or a junior doctor
Is that all there is to it?
giving their gestalt – we still achieved 100% sensitivity.
No, wait, there’s more. And here’s where the story gets really
interesting. Nobody in their right mind would consider the possibility of
What does that mean for our practice?
ACS without doing an ECG, right? So it’s probably unfair to assess
clinical judgement alone without at least taking account of the ECG. Before anyone gets carried away by these promising findings, it’s
We’re unlikely to want to discharge these patients if they have important to recognise that there are some limitations to this work.
ischaemic ECGs, even if we do think that ACS is still unlikely. What’s The 95% confidence intervals for sensitivity extend down to 95.6%
more, we have another amazing test that we’d always run in these and the first reports of new diagnostic technology often overestimate
patients – troponin. We run that test on arrival too, and again we performance (commonly due to reporting and publication bias).
wouldn’t send the patients home if they had a positive troponin on Therefore we need to validate these findings – first in observational
admission – even if we did think that ACS was unlikely. So we looked studies and then, if they still show promise, in interventional trials. Of
course, doctors might be slightly less bold when it comes to stating

53
their gestalt if they knew that they’d have to send patients home
based on their estimates. We can only find that out by evaluating the
strategy in practice.

This means that you shouldn’t

use gestalt to rule out ACS
r i g h t n o w, e v e n i n
combination with the ECG
and troponin. But you can
rest assured that the
probability of ACS in patients
with normal troponin and
ECG on arrival is extremely
low if your clinical judgement
suggests that the diagnosis is
unlikely. This can affect your
practice – because we often
treat patients with possible
ACS on the assumption that
they have that diagnosis, before further tests can confirm or refute it.
Those treatments have risks (especially bleeding). If the diagnosis of
ACS is extremely unlikely, the patients really aren’t going to benefit
from early treatment overall. So hang fire with your prescribing pens –
and rest assured that your judgement is probably right, even though
you do still need to rely on those serial troponins (at least for now!)

Rick Body

54
EXCLUSIVE ADDITIONAL #FOAMed ANALYSES, ADDED ON 20TH The table shows that this approach isn’t sufficient to rule out AMI.  The
AUGUST 2014 bottom line, is that clinical judgement alone isn’t sufficient to rule out
AMI (as reported in the main paper).  Even if you combine that with
We recently had an offline request from Anand Swaminathan
ECG findings, you still can’t rule out AMI (or rule it in).  It’s only when
(@EMSwami) for some extra data from this study.  Swami was keen to
you start to incorporate troponins that you potentially get the rule out
know how much high sensitivity troponin really adds once the
(pending validation, of course).
physician’s gestalt and the ECG findings are taken into account.  He
So, is troponin a friend or a foe in the Emergency Department?  We’ll
be posting more on that (from SMACC Gold) imminently, but here’s
some good evidence to suggest that it may well be a pretty good
friend.

Thanks again for a great request, Swami!

asked if we can also report the sensitivity and specificity of the gestalt
+ ECG – i.e. without high sensitivity troponin.  So here we go, in what I
believe could be a first in the #FOAMed world – new data being
presented as #FOAMed outside of a traditional medical journal. 
Thanks for the request, Swami!

The table below shows the diagnostic accuracy of the combination of

gestalt + the ECG (i.e. the presence or absence of acute ischaemia on
the initial ECG in the treating physician’s opinion)  for  the adjudicated
diagnosis of  AMI (as described in the paper). AMI ‘ruled out’ if no
ECG ischaemia and ‘could be’, ‘probably not’ or ‘definitely not’ ACS.

55
56
Section 5: Written by Richard Carden

I’m all about the Bayes, ’bout the Bayes, no treble

KEY POINTS

1. The Reverand Thomas Bayes described his

theory of probability in the 1700s, and yet
we are still struggling to grasp it’s
importance in clinical medicine.

2. Bayes theorem underpins much of our

thinking around probabilities.

3. Bayes links posterior (post test)

probabilities to prior probability (pre test)

4. Simply put: Initial belief + new objective

data = new and improved belief.

Who knew that the Reverend Thomas Bayes would be such a prominent figure in
pop culture? Not sure what we’re talking about, then follow this link to listen to
Meghan Trainor sing. Ignore the spelling error in the title of the song, we know who
she is really singing about.  I haven’t gone crazy, and hopefully I can explain the

57
inspiration behind this post in due course.  Thomas Bayes was a Prior probability x likelihood  α posterior probability
Presbyterian minister back in the days (1700s) when church and
Or very simply interpreted: Initial belief + new objective data =
science began to butt heads.  Despite writing some mathematical
new and improved belief
papers, he never published his eponymous theory.   It was a
friend, Richard Price, who discovered Bayes’ manuscript after his There are very few things, if any, that we can have 100% certainty
death, rewrote the document and about, and as McGrayne says in her excellent book (The Theory
submitted it for publication.  It was That Would Not Die): ‘probability is the mathematical expression
further elaborated upon by of our ignorance’.  I would really recommend reading the book!
Laplace (of Laplace’s Law fame)
You can watch a great video of McGrayne in her own words
who got wind of the Theorem
following this link
when Price visited France.   Over
the years it has been shunned and
been the subject of intense Why is any of this relevant?
scrutiny but nonetheless has had a
One could argue that a lot of medicine is a practical example of
huge impact, especially during
Bayes’ Theorem, but none more so than Emergency Medicine.
wartime (Turing, in part, used Bayesian techniques to crack
We have a prior probability, which we update, when new
Enigma!).
information is found and develop a posterior probability.   We do
Laplace described his version of Bayes’ Theorem as “inverse this all day every day, albeit without realising it, on a
probability”.  By this he meant that the Theorem looked at effects subconscious level.  I think the real importance is being aware of
to infer the causes.  Bayes’ Theorem uses the terms  how this theorem applies to us and how it can help us to avoid
‘prior’,  ‘likelihood’  and  ‘posterior’  to describe probabilities.   The pitfalls.  I am going to propose that by not using this theorem, or
prior is the probability of an initial belief, the likelihood is the using it incorrectly and incompletely leads to missed diagnoses
probability of other hypotheses and the posterior is probability of and other errors.
the revised belief:
Let’s imagine you see a patient with chest pain.  You take a brief
Prior times likelihood is proportional to the posterior. history and by the end of doing so have applied probabilities to
what you believe the cause to be; let’s say ACS for the sake of
58
argument (essentially your differential diagnosis is a list of update our probabilities once we have our posterior.   I am not
Bayesian priors!).  You then examine the patient and increase or endorsing the over-investigation of every single patient with chest
decrease the probabilities depending on your clinical findings, in pain, but rather challenging whoever is reading this to take an
this case there are none.   You send a Troponin, fortunately it is extra minute or two and re-examine their probabilities and
within an appropriate timeframe for your particular assay, and differentials when receiving a bundle of negative results.
maybe a couple of other bloods and eagerly await the results.
Research
The results all come back normal….bingo! You can discharge the
patient confidently having ruled out your differential diagnosis. Bayesians and Frequentists have a long-lasting feud over the
superior approach to probabilities.  The NEJM recently published
This is is an all too familiar scenario, and if scenarios like the
an article, around breast cancer therapies, but also had a
above are happening often, then it is no wonder that we miss
discussion around the future of clinical trials and the applications,
other pathologies (i.e aortic dissections).   I believe an all too
strengths and weaknesses of using Bayesian and Frequentist
common pitfall in EM, amongst other specialities, is a failure to
study designs.   The I-SPY 2 Trial uses Bayesian statistics to
identify which combination of breast cancer drugs has the highest
probability of being efficacious, and using this outcome will
progress certain drugs to the next round of clinical trialling.  This
is an efficient means of research, in that it reduces the
unnecessary cost and time in investigating all the
potential medications in all stages of clinical trial, and limiting it to
those that are the most likely to be efficacious.  The data to form
the initial probabilities is based on small study populations.

A great clinical application of Bayes’ Theorem is work done by

the Centre for Trauma Sciences that used Bayesian modelling to
predict the risk of individuals developing acute traumatic
coagulopathy (ATC).  You can see and use the model here.  The

59
model combines existing data from studies looking at the causes
of ATC with expert knowledge.

You can also read lots of Bayes related work in the #FOAMed
world – Casey Parker at Broome docs is a great place to start.

This article is intended as a very inadequate primer to Bayes’

Theorem.   I believe it is integral to our practice and is gaining a
position in our research activities. My take home message:
 ‘Always update your priors’, i.e. don’t ignore new information!

Rich

@richcarden

60
Section 6: Written by Chris Gray.

Overconfidence in clinical decisions.

KEY POINTS

1. No one wants to be perceived as over

confident.

2. We are not good at assessing our own

performance

3. Good quality feedback improves your

ability to know how good you really are.

4. Seek out colleagues and trainers who can

give you fair, honest and regular feedback.

5. The Dunning-Kruger effect is not always

what you think it is from the graphs you see
on Twitter.

61
If you have ever worked as or with a doctor, you’ve probably this way. A consultant anaesthetist is unlikely to be fazed by a failed
encountered something called a multi-source feedback form, or a epidural, but a junior trainee may be – particularly when the consultant
“360 appraisal”. It is part of the process for junior doctors to progress then proceeds to make it look easy!
to the next stage of our training, and is often a valuable tool as an
Overconfidence is an excessive belief in something succeeding,
anonymous insight into what our senior doctor, nursing, and allied
without any regard for failure. This can lead to mistakes. An
health professional colleagues think about us, both good and bad.
overconfident person can be so focused on the path that they are
On my most recent feedback form whilst on a rotation outside of the taking, that they are blinded to other possibilities, also making it more
emergency department, there were two comments with mention that I difficult for other team members to challenge decisions. They might
was bordering on overconfidence. The rest of the feedback was very miss clues that point towards a different answer. Confirmation bias
positive, but it was these comments that really stuck with me. I wasn’t can play a role here, as they may be more actively seeking evidence to
sure what I was doing to give the impression of overconfidence. Was I back up their conclusions. Poor follow-up can reinforce behaviour
actually overconfident? How would I know? What could I do to through not knowing whether the right decision was made.
improve? And really, the most important question – was it affecting my
Arrogance takes this a step further, involving active dismissal of
patients?
others’ opinions, and lack of humility. It also differs from
This post brings together my reflection over the last few months. overconfidence in that usually the person is competent, whereas in
overconfidence their confidence is greater than their competence.
Overconfidence
The line between confidence and overconfidence is fine. Too confident
It’s important to be aware that confidence can be a good thing. In an and your team may feel they can’t challenge you and mistakes might
emergency situation, for example, you want to know that the team be made. Important details may be missed, lessons not learnt for the
leader has control. If they appear confident, they instil confidence in future. Not confident enough and you may lose faith in your own
those around them, and this can make the team work more effectively decisions, or not attempt something due to fear of failure. The line is
and efficiently towards a common goal. Confidence is formed with also difficult to define as it is largely based on the perceptions of other
knowledge, and gained with experience. It can fluctuate; you may people. A more experienced person may be able to spot
have had a run of successful, first pass lumbar punctures, but fail one overconfidence in a colleague, where others with less experience just
and your confidence diminishes. This can affect how a situation is see confidence.
managed the next time it presents. The more experience you gain in a
particular setting, the less likely it is that your confidence will shift in

62
Does it exist in medicine? patient needs to be admitted for further management, or discharged
home to benefit from time passing with the potential for follow up at a
Of course. Doctors are no less vulnerable to overconfidence than the later stage. In such instances, the issues surrounding confidence do
rest of the population. Unfortunately, due to the nature of the job, not reflect confidence in a diagnosis, rather on where the patient
errors made by healthcare professionals can have life-changing should go after they leave the department.
consequences for the patients.

A review of the literature by Graber and Berner (pdf) in 2008 found that

diagnostic error exists at a rate of less than five percent in perceptual
specialties such as radiology and pathology, but that this can rise to
15% in others. Overconfidence is a contributing factor; studies
reviewed by the paper found that physicians often don’t use available
information resources to aid diagnosis, and where guidelines or
protocols were in place to manage specific conditions, these were
only followed around 50% of the time.

Physicians know that error exists, but seem to believe that it is less
than it really is. Despite an error rate of around 5-15%, research by
Friedman et al. comparing confidence and accuracy in diagnosis by
medical students, residents and attending physicians gained some
perhaps unsurprising results. The medical students were the least
confident and accurate, the attendings the most, and the residents (See Footnote on this diagram at the end of this section)
were more confident but less accurate than their attendings. We have
It is rare that we get feedback though. The patient we send home
seen this highlighted before in the Dunning-Kruger effect where less
today may well return tomorrow. If something bad happens, we might
skilled individuals overestimate their ability.
hear about it in one of those heart-stopping “do you remember that
In the emergency department, decision making is complex. Time and patient you saw last night?” conversations, but due to the varied shift
resources are limited, the environment is busy, and physicians are nature of the emergency physician, it’s unlikely we will be there to see
often looking after multiple unwell patients. In many cases, a diagnosis them the second time around. If we don’t hear anything, it is easy to
may not be made, and the decision comes down to whether the assume that what we are doing is right. We become more confident in

63
our management, but that doesn’t mean we are right. Often we will had to be honest with the specialty you referred to and say “I don’t
seek advice on unusual presentations, but we can become know what’s going on, but this person is unwell and needs
overconfident in the routine. admission”. See them on the ward the next day or find the ward
doctors and ask what happened to the patient. What treatment did
How can we avoid the dangers of overconfidence?
they give, what could have been done differently? In emergency

Know your limitations.
 medicine, the majority of the time you don’t get closure in the

Being aware of the extent of your own knowledge and skills is department, so it’s important to seek this out later on and compare

valuable. It allows identification of gaps in knowledge, and therefore how confident you were with the eventual outcome.

the potential to fill these. If you do something often enough, and Seek feedback.

nothing bad happens, your confidence is increased. Incompetence is Be it immediate feedback on a case from a senior, or more generalised
also reinforced, leading to overconfidence, and it may be that one bad feedback from the team as a whole in a multi-source appraisal, one of
outcome is viewed as “unlucky” despite the fact that actually you were the most useful determinants of confidence against overconfidence is
lucky the rest of the time. Continue to learn. Minimise the unknown how you are viewed by others. Ask for it often, meet with your
unknowns. supervisor regularly, and utilise anonymous observations from a wide

Ask advice.
 range of colleagues to inform your clinical and non-clinical practice.

A second opinion is useful to either confirm or correct an initial Revisit.


diagnosis. This can be gained from a more senior doctor or a Once you have your feedback, process it, then think about it again a
specialist. The latter is done whenever we refer a patient into hospital, few days later. Some situations are emotional, stressful, and advice
but how many times do we actually find out what the specialist from colleagues can get lost at the time. Going over the event and the
thought? Without knowing the result of the consult, confidence feedback you received a few days later can make it easier to take on
increases in our management of the patient, whether it was right or board, and easier to reflect on. This could be done on your own, or
wrong. For those of you that are the senior opinion – create an with a colleague.
atmosphere which enables advice to be easily sought.

Follow your patients up.


If you don’t know whether your diagnosis was the right one, you can’t \
change your practice for the next patient you see. Make a note of
patients where you were sure, unsure, had to ask for help, where you
64
Final thoughts Footnote on Dunning Kruger.

The feedback process has helped me to be more mindful of how I The graph in this paper was discovered to be a cartoon illustration of
make management decisions for my patients. I’m more conscious of the original work. This was discovered by our good friend Ross Fisher
my ability to make mistakes, and measures I can employ to try to and announced at the Don’t Forget the Bubbles conference in 2017.
minimise overconfidence, without taking it too far and losing
Although the illustration here never appeared in the original ‘IgNobel’
confidence. I know there will still be times when I’m more confident
winning paper it does speak to the principles in the paper.
than I should be, but hopefully will be able to recognise this when it
happens. We have kept it here as it is the one that is seen across social media,

Chris and it makes the point well. If you want to know more about the
Dunning-Kruger effect, and arguably if you want to quote it then please
follow this link for the original paper with more accurate graphs.

65
Section 7 Written by Natalie May

When is a Door Not a Door? Bias, Heuristics & Metacognition

KEY POINTS

1. Clinical decisions are prone to a range of

predictable (and not so predictable) biases.

2. Expert clinical decision makers realise

when clinical information is contradictory
and understand how to deal with this when
it happens.

3. Expert clinical decision makers use a

combination of objective knowledge
combined with an understanding of
uncertainty and risk to come to a clinical
decision.

4. Understanding bias, heuristics and meta-

cognition can make you a better clinician
(and educator).

This is an amended version of a talk on metacognition I gave at the Royal North

Shore Hospital Grand Round. The clinical scenario has changed but the content
remains the same. The purpose of this talk was to help our colleagues in the
hospital specialties appreciate the mindset of the Emergency Physician and the
difficulties intrinsically associated with the job we do.

66
Picture the scene:
It’s early October and the ED is already busy. You’re on the Then you ask him how he got the bruise.
morning shift and one of your first jobs is to clear out the patients
from the short stay ward to make space for the inevitable “Bruise?” he asks, looking for it. “I didn’t know I had a bruise.”
admissions of the day. One patient is a 19-year-old student, male,
who presented the preceding evening with vomiting and fever. Suddenly alarm bells ring in your head – what if it’s not a bruise?
He’s normally well (only taking olanzepine for schizophrenia). The What if it’s purpura? But the blood tests are normal…
overzealous registrar failed to diagnose Freshers’ ‘flu  before  
doing bloods at 0200 (which were reassuringly normal – WCC 5, What can we do when we are faced with contradictions in our
CRP 4). For some reason he’s been on the ward all night for a diagnostic processes?
dose of ondansetron and two litres of Hartmann’s. His fever Well, as emergency physicians we spend a great deal of time in
resolved immediately with IV paracetamol and his observations this state. With the work of eminent EPs like Prof Pat Croskerry
have been normal all night – currently HR 92/min, NIBP (check out these audio and video lectures which cover much of
110/68mmHg, RR 18/min, SpO2 97% on RA. The nurses tell you the first part of this talk), we are increasingly aware of our own
he had some profuse diarrhoea at around 0600 and you dread thinking processes, so called “metacognition”.
having the ward closed if it’s norovirus…
When we make any kind of decisions, our brains rely on the
As you approach the bed to kick him out of your unit before information available to them. We think we tend to process this
infection control arrives, you notice he’s barely visible underneath information in one of two ways, known as dual process theory as
the mountain of sheets and hospital blankets – apart from a left described by Nobel prize winner Daniel Kahneman in his book
shoulder, which sports a cracking bruise. “Thinking Fast And Slow”. System one thinking is fast and
intuitive – these make up around 95% of the decisions we make.
You wake him up, explain to him that he will feel much better System two thinking is slow, deliberate, reflective, measured –
being ill in his own student flat, and reassure him that the blood and this is much less common.
tests were all normal and he’ll feel better in a few days. He nods
sheepishly.

67
To demonstrate, let’s try a small quiz – grab a bit of paper and a We tend to think in one of these two modes: fast, reflexive and
pen. There are no tricks here – just answer the questions. intuitive or slower, reflective, analytical and deliberate. It’s really
useful, if you’ve seen a lot of sick patients, you get good at
• On a standard fire engine, there are two drivers up front, one recognising patterns. Imagine a 55 year old man with chest pain,
at the rear and three additional staff – how many do you he’s grey and sweaty, what are you expecting on the ECG?
need for five standard trucks?
• How many turtle doves did my true love send to me on the Heuristics
2nd day of Christmas? But sometimes, we need to overcome the automaticity of System
• A bat and a ball cost £1.10 in total. The bat costs £1 more I, otherwise we can make errors. Many of our patients can be
than the ball – how much does the ball cost? approached in a very simple way. When we first start in medicine
• It takes 5 machines 5 minutes to make 5 widgets, how long and particularly in EM, everything is new, everything is difficult.
would it take 100 machines to make 100 widgets? Thought processes are conscious. But we naturally reduce the
• In a lake there’s a patch of lily pads and every hour it complexity of our decision-making to simple pathways: if this,
doubles in size. If it takes 48 hours to cover the entire lake, then this – these processes are known as heuristics. Defined as
how many hours would it take to cover half the lake? “a practical problem solving approach not guaranteed to be
perfect but sufficient for the immediate goals”, these are mental
The answers are 30 (5 x 6) and 2. These questions are just to shortcuts which may be generated subconsciously.
warm you up. The others are 5p, 5 minutes and 47 hours. Think
about them for a while and feel free to tweet me if, after a bit of a As doctors,  we like them. It appeals to us to think we are
think, you can’t figure out why those are the correct answers. superhuman and infallible. There’s even a word for “I just know” ,
This is known as a cognitive reflexive test: your intuitive response the word “gestalt”. Sadly, so far attempts to validate clinician
is to get these wrong but if you take the time to reflect you can gestalt alone against other clinical scoring systems have failed to
get yourself to the right answers. It has little to do with show that we are really as brilliant as we think.
intellect/intelligence, instead it tells us about your ability to
suppress the intuitive response, to overcome the System I Dunning & Kruger described this in learners, that as we accrue
thinking and engage System II. knowledge we enter a phase of blissful ignorance. Educationalists

68
call this “unconscious incompetence”; you might hear it referred picture, of a dermatomal painful rash consisting of numerous tiny
to as being “at the top of mount stupid”. We certainly see this in blisters?
doctors who are new to
medicine and especially How long did it take for you to know the diagnosis? Did you even
new to Emergency need to click on the link to know it was shingles? There was a
Medicine. In some places time, hard to believe it, when you saw this for the first time and
it’s the way we teach you didn’t know what it was. But when we undertake deliberate
medicine. Occam’s razor practice regularly, it will become subconscious (we can switch
states that ‘the simplest things over to system I, which is why we use simulation and we
explanation is usually the best’. Consider the patient with  RUQ regularly attend life support courses) so that when we don’t have
pain, fever, jaundice. Acute cholangitis, right? Except that the time to sit and slowly employ type II thinking, we are equipped to
specificity is 93.2% and the sensitivity is 36.3% . This is very take action.
much an imperfect test, probably not good enough for ruling in or
ruling out in isolation.
Other biases
Eventually, we get burned, we make a horrible error and we get There are other factors at play here too. We all have intrinsic
away with it – or we don’t – and we come crashing down the biases that shape our decision-making, many of which we might
other side. And with experience comes wisdom, so we can learn not be aware of. Think about our patient. Is anyone willing to
to plug in cognitive stopgaps. For example, I have a programmed admit that they thought his symptoms might have a psychiatric
cognitive stop to convince myself that any young woman with cause? Bias against obese patients is particularly common and
abdominal pain or syncope does not have an ectopic pregnancy. especially powerful. I strongly recommend that you each
When we shortcut and rely solely on heuristics and system  I undertake the implicit association test online (Harvard’s is freely
thinking, we leave ourselves open to errors in our available). The concept of framing, that is, the context within
decision-making. which information is presented to you, either by the patient
themselves or by the referring clinician, will shape the way you
We have to work hard to deliberately employ system II to ensure receive it. Contextual information matters. We tell ourselves that
we don’t make diagnostic errors. It’s very hard. What’s this the risk factors patients have increase the probability of having
69
certain disease
processes, which is • confirmation bias (giving more weight to information which
true, but it doesn’t seems to align with what we already think)
mean that patients
without any risk factors • search satisfaction (stopping once we have found
at all can’t have those something – the reason we miss the second fracture)
exact same disease
p ro c e s s e s . T h i s i s • premature closure (stopping too early in the diagnostic
Hickham’s dictum, the counterargument to Occam’s razor. process because we think we have an answer)
Patients can have as many diseases as they damn well please
(with or without whichever risk factors they have or don’t have)! • hindsight bias (forming opinions about early
decision-making based on what happened later “it was
There are a host of other biases that play into our obvious… I can’t believe they missed it”).
decision-making processes (see this great summary at First10EM)
things like:

• availability bias (we judge the likelihood of disease by the

ease with which examples come to mind, so we are more
likely to diagnose diseases we’ve had recent experiences of,
or are particularly attuned to [a subtype called significant
case bias])
• base rate neglect (we ignore the prevalence of a disease in
our diagnostic reasoning, such as the “worst first” approach
to diagnostics)

• anchoring (seizing a diagnosis based on early information

and failing to adjust as new information becomes available)
70
I probably also need to mention blindside bias; the tendency to We have to play off our own cognitive processes against an
hear a talk like this, to nod your head in agreement throughout understanding of probability and in order to understand
thinking about all the people this stuff applies to and thanking probability better we have to appreciate Bayesian statistics.
God or the tooth fairy that you’re exempt (you’re not).
Bayes theorem states that
Emergency physicians do a huge amount of decision making and • The tests are not the event
it’s something we aren’t really trained in. When we ask medical • Tests are flawed (false positive and negatives occur)
students the areas they think they need to focus on to be good • Tests give us test probabilities, not real probabilities
doctors, they say things like, “knowledge”, “practical skills”. • False positives skew results (especially for rare diseases)
• People prefer natural numbers
Diagnostic error doesn’t always lead to bad things happening. If • We need to convert test results into real probabilities for our
we diagnose sinusitis and it was a viral URTI, it doesn’t really patients
matter. The majority of our patients will get better in spite of what •
we do to them (we know this through placebo-controlled trials). By working as Bayesian practitioners, we try to add meaningful
The answer must be more tests! information to improve our probability of getting the right answer.
We must order more tests: more CT scans, more bloods, One of the helpful components of information we can add, is
troponins and d-dimers and CRPs for everyone! time. We see patients comparatively, very early in their disease

Well, no, not just because of cost and radiation exposure,

because the results of tests depend on the questions we are
asking. All tests have false negative and false positive rates and
for many, the performance of the test is dependent on the
prevalence of the disease within the tested population, not within
the entire population.

71
trajectory; there are very few diseases that fully manifest in an Two days later, the blood culture taken by the very thorough
unequivocal way immediately. Most develop slowly; the problem registrar overnight at the same time as the WCC of 5 and CRP of
is that patients decide when to come to the ED. 4 grew – N. meningiditis. It’s all so easy in hindsight 🙂

We work in a decision-rich, information-light environment. By

So thanks for indulging me on this journey into metacognition. I
nature, some of our decisions and diagnoses are going to be
thought this was a great case for exploring the way that we as
incorrect as disease processes manifest themselves which might
medical practitioners think and in particular the challenges of
be why our excellent general medical colleagues always seem so
thinking in Emergency Medicine. That’s always in our interest
wise and knowledgeable on the post-take ward round the next
because our job can be pretty tricky. That’s the message I’d like
day.
you to take away from this: Emergency Medicine decision-making
is often decision heavy and information  light. We will get things
Let’s go back to our case
wrong and sometimes we pass some of that decision-making
Having thought about all these things, what should we do?
responsibility on to our inpatient colleagues, because we believe
that time will help us to make sense of what we see.
We don’t have a ready diagnosis to tie things together nicely as
Occam’s razor. We have several concerning features: the fever,
All this does not really answer the original question of course.
the “bruise”, the vomiting and a potentially treatable, potentially
fatal possibility of meningococcal sepsis. We can make a decision
When is a door not a door?
to push the schizophrenia and the Freshers’ ‘flu, common as it is,
When it’s ajar 🙂
out of the picture until we are satisfied there isn’t anything else
going on.
Nat
We decide  to give antibiotics and refer to the inpatient medical @_NMay
team. They repeat his bloods, finding his WCC now more than 20
and CRP in the 200s. We  feel justified, ah, the power of
confirmation bias. Want to know more? Try these

72
All Doctors Are Jackasses

First 10 EM on Cognitive Errors – Part One, Two, Three & Four

Search Satisficing/Satisfaction at LiTFL

Monty Hall problem explained (including simulator)

73
Section 8

Making good decisions in the ED.

KEY POINTS

1. It’s difficult to judge your own decisions

without information and we often lack this.

2. Looking at outcomes is good but not

enough. Judgement is about process not
outcome. Nobody wants the lucky doctor,
they want the good doctor.

3. Normal flight is the best place to develop

and train judgement. Make sure that you
spend time in the middle and not at the
extremes of practice.

This section is based on a talk I gave at the Royal College of Emergency Medicine
conference in Manchester, and a similar talk on Metacognition in the ED delivered
today in Torino at the EuSEM conference.

74
The title of the talk is ‘Making Good Decisions’ and in reality that </div>
could include many things. Life as an emergency physician is The slides for this talk can be seen below.
crammed full of decisions and therefore full of judgement. The
world of the emergency physician is an uncertain one, where we What do we mean by ‘judgement’?
are required to make difficult decisions on a daily basis. If you
want to get the quick version of the talk watch this short summary There is a risk that we can get bogged down with semantics and
from the Royal College youtube channel on making good definitions so we should define what we we are talking about. For
decisions in the ED and then ,if you’re interested in knowing the purposes of this section, we consider judgement as a term
more, read on. that describes a range of techniques, psychological processes
and ideas that we use in
Interactive 2.1 Slides on ED clinical decisions the emergency
department to make
decisions. These include
concepts such as risk,
p r o b a b i l i t y, g e s t a l t ,
reasoning and
uncertainty. These are
topics that we have
looked at over the years in EM and there are a few reasons why
that is particularly relevant to us as emergency physicians. It’s
worth reflecting and reviewing why this is such an important topic
for us.

1. Emergency medicine is often considered to be a risky

speciality, in that we deal with a population of patients who may
go on to have adverse consequences of their disease. We also
operate at the admission/discharge interface and so every time
75
we see a patient (or at least in my experience every shift), we take
a risk. Is discharge the right decision or are we sending someone
home who might have an adverse event?

2. We, arguably, treat populations as much as we do individuals.

This can cause difficult realities in our practice as probability
underpins many of our decision
making processes. If we take
something like pleuritic chest
pain and the risk of PE we have
great evidence that the tools we
returned that diagnoses are refined and that uncertainty falls. We,
use have a sensitivity of 98%
as emergency physicians, operate in this zone, the zone of
which sounds great. That does
uncertainty, the zone of judgement, the zone where it is the mind
of course, mean that we miss
of the emergency physician has primacy.
1:50 patients with PE, and this
is still good practice. In essence, our diagnostic tools have an
acceptable failure rate. Therefore we need to be wise in when to
use clinical decision rules, and also when not to.

3. We frequently see patients at an early stage of the disease

process when clinical information may be unavailable or at least
only minimally manifest. At our initial assessment, in the ED, with
only that information that we can glean at the bedside the level of
uncertainty can be high. As time passes, results come back,
trends in clinical course become apparent and uncertainty reigns.
It is only after information flows, time passes, investigations are
76
4.     I’ve been a proponent of evidence based medicine for many clinician. He has no neurology and only has a mild headache. He
years, but in the last 5-10 years, it has become increasingly is discharged from the ED, as his symptoms have resolved. Three
apparent to me that evidence is not enough. Nearly all months later, he is fine, nothing happened and he came to no
evidence is filtered through something before it reaches a harm. Sixty years later, he dies having never developed significant
patient, and that something is you and me. It is a holy grail for cataracts or a brain tumour. Great outcome – bad judgement.
researchers and health services that when high quality
evidence appears, it will reach the bedside rapidly and ·You can have a bad outcome
effectively, but we know that this is not the case. The average with good judgement.
time for new knowledge to reach every day practice for our A patient visits the ED on a
patients is 14 years. Everything we do is filtered through Friday night with a swollen leg.
awareness, judgement, opinion and belief and this is a theme They score 2 on a Well’s score
that we have been exploring within St.Emlyn’s for many years.  and so a d-dimer is taken. This
Knowledge translation and transference remains a problem in comes back elevated and so
all aspects of health care and is a reason why we blog and the patient is placed on low
podcast at St.Emlyn’s. molecular weight heparin over the weekend pending an
ultrasound scan on Monday. Sadly, he returns on Sunday evening
So let’s think about judgement in a little more detail and perhaps with a raging compartment syndrome requiring a fasciotomy and
consider how we could know whether we are good at it. months of rehabilitation. Great judgement – bad outcome.
When thinking about judgement then process is not the same as
·You can have a great outcome outcome.
with bad judgement
A patient comes into the ED with So how do we know if we are making good judgements?
a severe sudden onset Let’s have a think about some of the mechanisms that we, or
headache associated with a others, use when considering this question of clinical decision
collapse. He is 32 years old and making. Let’s consider five fallacies of feedback, reasons why we
is well when seen by the ED might not make good judgements about our judgement.

77
 recognize that most of our practice is normal care. From an airline
1.     You might think that you have great judgement because you perspective, we need to consider our performance in normal flight
don’t hear about complaints, coroner’s inquests, or sued. The and not just when it goes wrong.
fallacy here is obvious, if you think about it. There is no doubt that
adverse events are great learning experiences. They tell us
why things go wrong and can help us identify individual and
systematic failings in our
health care systems. They
are really important but
they do not tell us what we
need to know about the
entirety of our practice. An
analogy from the airline
industry comes to mind,
which we appear to be in
awe of when it comes to systems and safety. There is no doubt
that work transferred from the airline industry around crash
investigation, human factors and patient safety has made
profound and important differences to the way we practice
medicine. I am am advocate of this, but it’s a bit like judging 2.     You might think that you make great decisions because you
whether someone is a good pilot by analysing their crashes. can recall a case where you made a fantastic decision.
That’s a bad idea for many reasons ,most notably, that harm has Perhaps you even saved a life. I recall a case where I made the
to take place before wisdom arises. It would clearly be insane to decision to perform an ultrasound in a shocked 30 year old that
spend a long time training and learning about how to crash, and we thought had sepsis. Turned out the patient had a ruptured
then not crash a plane into a mountain in the hope that by doing AAA. Was that great judgement or luck? Does it mean I’m a
so we would then be capable of flying a 737 from London to
Manchester. Learning from error is great, but we must also
78
 great doctor. No. Just as with the assessment of error in great quote from a great player, but we are not playing golf and in
isolation we cannot judge our practice on extremes. medicine, it’s different. In sport, luck gets you to the top, in
medicine it protects you at the bottom. Let’s assume that you’re
3.     Perhaps you have a department that functions well and has not completely hopeless as a physician but you’re not very good.
great outcomes. Does that mean you have great decision Let’s say you have fallen into the trap of not examining children
making? Probably not. Most of our patients pass through many with potential sepsis properly. Maybe you don’t look for rashes
hands and processes so drilling down into the effect of routinely. Sure, you
individual decisions can be difficult. System wide outcomes can spot the
lack the fidelity to tell you about your decision making. moribund patients
Similarly, if you work in a department with poor outcomes that with a widespread
may not be a result of your decision making. rash of
meningococcal
4.       Perhaps nobody has taken you to one side to discuss your septicaemia, but the
judgement in many years. This is an issue for all, but perhaps more subtle cases
mostly for those of us who are getting on in our careers. At you’re rubbish at. Well
junior level, there are mechanisms to help understand progress luck is on your side
and to ensure that we at least consider the breadth of the because very few kids with fever have meningococcal
curriculum. The more senior you become, the less likely it is septicaemia. Luck and probability are on your side and so you
that you have a formalized mechanism to assess your can go on sending them home without looking for a rash for days,
performance. That is unless you have already been referred to weeks, months, in fact years before you will miss something
the GMC for poor practice, or if you find out through an HLI or important (and you will miss it). Luck is very much on your side as
SUI in your practice. Sadly, unless you keep on top of your a diagnostician and as such, it is a fallacy of feedback. You can
decision making and thought, the first time you might find out make poor decisions for a very long time before your luck runs
about a hidden error is when disaster happens. out.

5.        Perhaps time will help you. It’s been said (by Gary Player 6.     Perhaps you make great decisions, but it’s on the premise of
amongst others) that the more I practice the luckier I get. It’s a poor knowledge. This is also known as the unknown unknown
79
 problem. This is characterised by you doing what you believe Does feedback work?
to be correct, but the world has moved on and what you are Well, there is evidence that it can. It comes from an area that you
doing is historic medicine, it may even be harmful. In those are very familiar with, that is very important for us here in
circumstances everyone blissfully carries on regardless without Manchester and it’s something that impact on you every day.
highlighting any poor practice because you simply don’t
recognize what you are doing as poor practice. If I were to Any thoughts on what it might be?
think of an example, let’s consider using peripheral
vasopressors in the ED. Some of you may think that you need It is the weather. When weather forecasting started out it was a
to get a central line in to deliver these and your patients may little hit and miss. Meteorologists would look at information
hang around, potentially with an adverse outcome, whilst you coming in from weather stations across the world. Maps would
wait to achieve it. You may well be working in an environment be created and predictions would be made for the weather the
where you would be criticized, even incident reported for next day. In the early days, predictions were better than flipping a
starting a peripheral noradrenaline infusion. The evidence is coin, but there was clearly room for improvement and that’s
there that they are safe, and you and your colleagues would be exactly what has happened. Today, weather forecasting is
unconsciously incompetent about this. excellent. If it says it’s going to rain tomorrow in Manchester then
it probably will – although that’s a fairly easy prediction. What we
The theme in all of these issues is around feedback, and you now have is the ability to predict the probability of weather with a
might think that I’m suggesting that feedback does not work well. high degree of accuracy and reliability.
In fact, I believe the opposite. Feedback is fantastic, and it’s the
key to making good decisions, but we don’t do it well. This change has come about through feedback. However, it’s not
the feedback of tornadoes or snow. It’s regular, repeatable,
routine feedback. In airline terms, it’s an analysis of normal flight
rather than acrobatics or crashes.

In medicine do we do this? Well, in many specialities it does

happen. Many surgical specialities have outcome and process
data (e.g. cardiac surgery, ophthalmology). Similarly, pretty much
80
every other speciality gets feedback on their decisions in some extremes of our practice through normal mechanisms but what
formalised way. Clinic patients come back. Referring patients do we have to learn about the generality of our practice?
from one in-patient service to another, triggers a process that This is wrong and counterintuitive to good learning. There is no
produces a letter of opinion, which (mostly) returns to the doubt that examining exceptional events can produce positive
referring doctor. This creates an effective feedback loop that outcomes and I’m not suggesting that we should not do this. 
helps the clinician learn about their own decision making and However, if it’s the only learning we do, then it’s the equivalent of
judgement. examining shark behaviour by only looking at shark attacks. Our
views would be skewed if we did and we might never go back in
the water, or we might cause major damage by changing
behaviour to avoid such rare events.
 
So how do you know if YOU make good decisions?
What does your feedback loop like?
How do you follow up the routine patients in your practice?
 
Perhaps we do need to follow patients up and find out what
happens to them, and not just the exceptions. It’s not difficult for
some of our patients. Those we admit to hospital will inevitably
get a discharge letter and in most hospitals, these are available
on your desktop computer. There is no reason, apart from inertia,
In many emergency medicine systems it’s different. With few that prevents you from doing this and it’s interesting when you
exceptions, we do not get regular, routine feedback on our do. I’ve found some interesting cases that have made me think
patients. Most of the time we see patients in the ED, make a about my practice. In the majority of cases, this process validates
diagnosis and then fire and forget. The patient leaves with a label what I do, but occasionally cases make me stop and think. The
or a treatment but we remain ignorant of whether it was right or patient with my diagnosis of ACS, turning out to have pericarditis.
wrong unless it is an exceptional case.  We may learn about the That’s not the sort of case that would comeback as an error, but it
is the sort of case that I can learn from.
81
  4.        It may be possible to get feedback from GPs,but again this
So what can we do to train our judgement and decision is difficult as systems between hospital and primary care are not
making? geared up for this.
 
Case note reviews How much should I be doing?
1.       Full notes review. You could ask your admin team to pull all That’s a good question and it rather depends on what else you
sets of notes that you have seen and then follow them up, but are up to at the moment. I would suggest that you can get by with
in all honesty this is unnecessary and you could end up in 5-10 cases per week. If you keep something like an NHS number
paralytic reflection. and have electronic records or discharge summaries this will take
you less than 20 minutes. So that’s quite a short investment in
2.    Discharge letters. Let’s remember that every admitted patient time for potentially an important return.
will get a discharge letter. These are easy to find and can be  
looked at 1-2 weeks post an on call shift. Keep a record of Peer review
patients you see in a notebook, a digital entry, a photocopy of
the ED record or whatever. Follow up, initially, with the Although I’ve talked about some of the disadvantages of working
discharge letter and then delve where you think you need to. in the ED in terms of getting feedback from inpatient teams or
family practitioners, in some ways we have the potential
3.     For those patients who are discharged from the department advantages. We work in teams. We often work alongside our
it’s trickier. We could phone patients, though this is not consultant colleagues in the
something I’ve tried. We would need to get clear agreement resus room and thus we
from patients to do this and so don’t try it until you have have amazing opportunities
considered the confidentiality and logistic issues in your health to use them as a spotlight on
system. our judgement. Think about
it, a fellow physician, with
the same training and
requirements as yourself, in
the same space and time.
82
It’s what I call a process of internal, externality (we have an understand and even diagnose their thinking, and from there you
internal resource that can give a degree of external feedback). can even deliver therapy. By therapy, I mean that if you
This does not have to be complex, and can be as simple as  peer understand why someone came to a decision, you can then 
review/observation. We’ve looked at this before at St.Emlyn’s suggest why they may have come to the decision.
describing a peer review process for trauma team leaders.
Interestingly, this usually results in more learning for the observer
than the observed, perhaps tackling questions of unconscious
incompetence.
 
Asking the right questions.

One of our most important roles as a clinical leader and trainer is

to give advice on patient care in the department. When times are
busy and the department overworked it’s all too easy to just tell
I like the paper by Bowen et al from the NEJM in 2006 which
people what to do, but that is a missed opportunity for exploring
outlines a number of strategies to diagnose and treat clinical
clinical decision
judgement. It’s a good paper and a fairly straightforward read.
making.
 
Remember that
Summary
outcome is not the
In summary, we need to recognise and value clinical decision
same as process, so
making as a core skill in the ED. We must learn how to
when a colleague asks
understand our own decisions and those of our colleagues.
whether they should
Where abnormal thinking arises, we should be able to understand
do something (eg
why and to assist colleagues and ourselves in improving it.
admit a patient as an
ACS) don’t just agree
or disagree. Learn to explore how your trainee came to a
decision, not just what the decision is. This allows both of you to
83
 Finally, you can listen to a podcast on the topic by clicking on this
ling to the St,Emlyn’s podcast Clinical Judgement for the
Emergency Physician

Simon Carley

 
Major learning points.
1.  It’s difficult to judge your own decisions without information
and we often lack this.
2.  Looking at outcomes is good but not enough. Judgement is
about process not outcome. Nobody wants the lucky doctor, they
want the good doctor.
3.  Normal flight is the best place to develop and train judgement.
Make sure that you spend time in the middle and not at the
extremes of practice.

85
Further reading
• Unskilled and unaware of it: how difficulties in recognizing

On the next page choose an

one’s own incompetence lead to inflated self-assessments.
Dunning & Kruger.
• Illusory Superiority Bias
• Svenson, Ola (February 1981). “Are We All Less Risky and
More Skillful Than Our Fellow Drivers?” Acta Psychologica
option for something you can
47 (2): 143–148. doi:10.1016/0001-6918(81)90005-6.
• The Short Coat (Lauren Westafer) on we don’t know what do in the next month to
we don’t know.
• http://shortcoatsinem.blogspot.it/2014/03/we-dont-know-w
hat-we-dont-know.html – some really good medical
improve your clinical
references on Lauren’s blog
• Scott Weingart’s path to insanity lecture and links cover decisions.
similar topics.
 

There is an option for

everyone.

86
Chapter 3

Podcasts &
Friends

A selection of podcasts from the St.Emlyn’s

podcast on risk, probability, uncertainty and
diagnostics. Links to other #FOAMed friends who
think like we do.
Section 1

Podcasts.

KEY POINTS

1. Diagnostics give us a real insight into the Over the last few years we have covered a number
uncertainties in medicine. of topics relevant to risk and probability on the
2. The St.Emlyn’s podcast covers a range of St.Emlyn’s podcast. You can find a selection of
topics related to emergency medicine these below.
including the uncertainties, risks and
probabilities inherent to emergency Click on the links, or even better subscribe to the
medicine practice.
podcast via iTunes
3. The podcast series challenges the illusion
of certainty that typically surrounds clinical
decisions.
1. Understanding diagnostics. SnOUT, SpIN and uncertainty in emergency
medicine.

2. Beyond simple yes/no diagnostics in emergency medicine.

3. Understanding diagnostics. Why prevalence helps keep us in practice.

4. Delving into NNT, ARR and RRR.

5. Making good decisions in the ED.

88
Section 2

#FOAMed friends.

We are not alone in maintaining an interest in the complexities of IM Reasoning. This podcast takes a fairly lengthy approach to
clinical decision making. clinical decisions in internal medicine. It’s useful if you are
interested in acute medicine. The episodes are quiet long, but
Here are a selection of links to other excellent resources from our
there is some pretty good content in there. You can find the
friends in the #FOAMed community.
podcasts here.
Broomedocs. The Broomedocs site is run by Casey Parker in
ALiEM. The Academic Life in Emergency Medicine team have a
North West Australia. We love this site, as it explores decision
range of resources on clinical decision making. Start here with an
making using clinical cases relevant to a rural practitioner. You
excellent article on clinical decisions by Javier Benitez.
might think that this is not relevant to you, but it is precisely by
seeing how decisions are affected by location and resource that Taming the SRU. has some great cases that explore clinical
you can understand and explore why you make the decisions decisions. You should also check their great post on the
you do. Broomedocs can be found here. cognitive autopsy, something that we should all be doing.

EMCRIT. Scott Weingart is well know in the #FOAMed Emergency Medicine Cases commonly explore clinical decisions
community for his work in resuscitation medicine. He has also through case based narrative. Start with this episode on
covered complexity and decision making in several of his blogs cognitive error and process.
and podcasts. Highlights to start with would by Scott’s podcasts
with Gary Klein and on OODA loops in critical care. Try An
interview with Gary Klein and OODA loops with Scott Weingart

89
Chapter 4

About
St.Emlyn’s

The St.Emlyn’s team are an international team of

clinical educators who specialise in emergency
medicine, critical care and prehospital care.
Contributors Editors

Simon Carley

Professor of Emergency Medicine. Emergency

Department, Manchester Royal Infirmary, Manchester, UK

Ashley Liebig

Chief Flight Nurse, Travis County STAR Flight, Austin,

Texas, USA

xci
Authors

Professor Simon Carley. Professor of Emergency Dr Chris Gray: Emergency Medicine trainee in
Medicine. Emergency Department, Manchester Manchester. Interested in pre-hospital care, trauma,
Royal Infirmary, Manchester, UK & critical care.

Dr Iain Beardsell. Consultant in Emergency

Medicine. University Hospital Southampton,
Southampton, UK

Professor Richard Body. Professor of Emergency

Medicine. Emergency Department, Manchester
Royal Infirmary, Manchester, UK

Dr Richard Carden. Emergency Medicine Registrar.

Emergency Department, Royal London Hospital,
London, UK

Dr Natalie May. Ambulance Service New South

Wales Rescue Helicopter Base, New South Wales,
Australia

xcii
Thanks for
reading.
Thanks for
thinking.
Thanks for
sharing.

View publication stats