Measuring outcomes - a new paradigm for CPD Homes
The end of the year is closing fast. For Australia’s 120,000 registered doctors, the new year will usher in a range of new changes to mandatory CPD, and none are causing more concern than “Measuring Outcomes”.
In this article, I’ll review what the new category means, the background behind it, and how your 2024 CPD might look.
Why has this change been made?
The Medical Board of Australia has published a document explaining the new changes, in which it reviews the rationale and evidence base for the new activity categories.
Essentially, the MBA contends that it is insufficient for doctors to simply learn new skills and knowledge. Rather, it sees it as fundamental that doctors should personally review how effective we are in our practice.
This process is commonly referred to as Self Reflection. In simple terms, self reflection is the practice of evaluating your own performance, considering how it can be improved and then implementing change.
Importantly, evaluating our performance is a critical first step. In effect, the evidence we use for this (which the MBA document refers to as “Audit-Feedback”) can come from many sources - external data sets (such as an electronic medical record), data we collect ourselves, or the feedback and evaluation we get from others. This dataset is the substrate upon which self reflection may occur.
Benefits
There are several likely benefits to this approach. It’s not too much of a stretch to see that actively reviewing your outcomes provides you with an opportunity to improve.
And the results may surprise you. More than once, a well conducted audit has demonstrated outcomes are significantly different to the pre-audit perception of the participants - its hard to manage what you can’t measure.
Measuring outcomes may give you the chance to see how the application of your practice translates to outcomes in the real world - perhaps the patient cohort you deal with is in some way different from others, leading to varying results.
Furthermore, taking the time to conduct an audit gives you a benchmark for changes. Implementing change without understanding your existing practice may result in difficulty interpreting the outcomes of your changes.
Measuring your outcomes also assists in developing the buy-in required to make changes - it’s harder to ignore when the evidence is there in black and white.
Evidence
But is there evidence for this approach?
Self reflection is recognised as one of the more effective strategies available in healthcare CPD (2). In a meta-analysis of 49 trials (3), Ivers et al found that self reflection with audit-feedback was consistently associated with an improvement in practice - this amounted to an absolute increase in performance of 4.3% when related to a dichotomous outcome (for example, prescribed beta blockers or did not), but only 1.3% for continuous variables (such as blood pressure control after stroke).
While the evidence base for audit-feedback is not strong, it is at least as effective, if not more so, than any other CPD activity (1).
On this basis, many (including the Medical Board of Australia) have concluded that the question is not “Does audit-feedback work?”, but rather “How can we maximise it's benefits?”
Several factors appear to influence the degree of change in behaviour (3):
how motivated the doctor is to reflect and improve practice
the frequency of feedback
the type of feedback given (written or verbal)
who provided the feedback (self-sourced, peer, supervisor, patient)
the stage of development of the doctor, with those with less experience / competency gaining more benefit
the quality and relevance of the data, and
the cultural support for practice improvement
While this is a relatively simple and sensible principle, the practical application is far less straight forward.
Many will argue it is difficult, if not impossible, to measure outcomes effectively, particularly in non-procedural specialties. Additionally, as the saying goes, “not all that is measurable is important, and not all that is important is measurable”.
Some things, such as surgical infection rates, may be relatively easy to capture, but others, for example how well you communicate, are much more difficult. However, expanding the way we see the concept of “data” can help - for example, you may choose to perform a peer-feedback exercise where you ask for commentary on your communication styles and effectiveness.
Doing it right
OK, so perhaps the evidence isn’t strong, but let’s assume for a moment that it is. The most important factor in how effective it is, is you. The more enthusiastically you buy in to this process, the more likely it is to work.
Critical to its success is the dataset you use. Doctors are notoriously poor at self-evaluation (4) - what does this mean? It means that without data, we have difficulty in assessing how we are performing against any standard - be it national standards, hospital guidelines or our peers. This is explored further in a great podcast interview with Marcus Watson
However, when given a reliable dataset, we are very good self reflectors.
What the MBA are asking us to do is fairly simple - take a sample of anything that reflects how we perform, either as an individual or as part of a collective, and use it to reflect on how we could improve our practice.
For example, you may be a general practitioner, who self audits prescribing patterns in patients with a recent myocardial infarction based on a national standard. Alternatively, you may be an intensive care specialist, and decide to audit the tracheostomy outcomes for your unit as a whole.
Ultimately, how impactful this is depends on the degree to which you trust the dataset - where does it come from, who provides it, what does it measure and so forth.
The new changes
Under the new CPD Home program, all doctors will be required to perform activities that measure outcomes as part of their CPD.
All doctors will be required to participate in at least 50 hours of CPD activities. Unlike past years, this cannot consist solely of educational activities. Instead, activities must fall into one of three categories (Educational Activities, Measuring Outcomes and Reviewing Performance), with a mandatory minimum in each.
Doctors must complete a minimum of 25 hours in the combined categories of Reviewing Performance and Measuring Outcomes, with at least 5 hours in each.
Given you must also complete a minimum of 12.5 hours of Educational Activities, Measuring Outcomes could in fact contribute up to 32.5 of your 50 CPD hours, or as few as 5.
What activities can I claim?
Across the CPD Homes, the activities you can undertake are relatively common and clearly defined. The Medical Board has published a list of recommended activities - you can read the details here.
Here’s a few examples :
Self-audit
Root cause analysis
Creation of, or review of Incident report
Individual / group quality improvement project
Practice / group audit
Morbidity / mortality meeting
Case conferences
Multi-disciplinary meetings
Leading, analysing, writing reports on healthcare outcomes
All these activities have the common characteristic of measuring outcomes in some form, whether it be discrete or narrative. Let’s take the example of patient feedback - you may get discrete data such as an overall rating out of 10; but you may also get narrative data in the form of suggestions for how you can improve your communication skills.
There is little doubt that in most cases, collecting data requires effort. For some activities, this may amount to many hours. Most importantly though, collecting the data is only half the battle. Without deliberate analysis and reflection on this data, all this effort is wasted.
The relationship between Reviewing Performance and Measuring Outcome
It is clear that there is significant overlap between the two categories - is collecting patient satisfaction surveys a measure of performance or outcome? For some doctors, it will be much easier to collect one type of data than the other.
However, this uncertainty is recognised in the new guidelines, in that the categories are combined into a global target of 25 hours, with a minimum of 5 in each.
How to choose activities
Many doctors will find that some of these activities will occur as part of their usual work practice - for example, junior doctors may attend a monthly morbidity and mortality meeting for their unit. Compliance then will involve documentation of the event, but more importantly, ensuring they document the learning outcomes achieved.
For others, some attention will be required for planning and execution, such as an audit.
What’s most important is that you consider the outcomes that are most important to you. Ideally, this should occur as part of your Personal Career Development Plan, created at the start of each CPD cycle.
Once you’ve done so, look at the list of activities and consider which will help you capture that data. Osler has created a series of guides for these activities to help you get underway.
References
The Evidence for Strengthened CPD. Medical Board Australia (2021)
Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. International journal of technology assessment in health care. 2005 Jul;21(3):380-5.
Davis DA, Mazmanian PE, Fordis M, Van Harrison RT, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Jama. 2006 Sep 6;296(9):1094-102