Assess
6.5 Measuring performance – data collection and analysis
EBVM includes patient, client, experiential and practice factors as well as the peer-reviewed scientific literature, and all of these will influence the information you gain and want to gain from a clinical audit.
For example, you might want to know if implementation of a protocol or new treatment has improved client satisfaction, decreased costs to the client, increased profit margins, saved time, improved veterinary compliance, reduced side effects, increased survival, or increased quality of life. In order to answer this ‘What if?’, you need to ensure you are asking the right questions to the right person (e.g. quality of life is often best evaluated by owners through practical questions involving the animal’s daily life, not by their veterinarian).
Your audit aims and objectives should be the primary consideration when deciding which data you will need to collect for your audit. You should only collect data required to show whether or not performance levels have been met for each criterion – collecting additional data provides little or no benefit, and is more time consuming.
When planning data collection for your audit, there are several aspects you should consider in advance, including:
- what data collection strategy is most likely to result in complete and reliable data?
- audit population
- what inclusion or exclusion criteria would you use to identify suitable cases for the audit?
- how many cases will you need to include?
- over what time period would you need to collect data?
- will you collect prospective or retrospective data?
- what data source(s) will you use?
- is all the information you require routinely recorded on electronic clinical records?
- will you need to design a data collection tool? (example in Waine et al. 2018b)
Example Scenario:
Small animal dental imaging
Tom’s audit sample includes all dogs and cats receiving dentals within the 12 months following installation of the computed radiography system. Tom extracted the total cost per dental visit from clinical records. His trainee vet nurse has designed an owner feedback questionnaire (including key questions about overall demeanour, eating behaviour and halitosis, and quality of life) as part of her nursing degree course.
Key point:
The audit team included the veterinary nurse, making use of her expertise to design and administer the owner questionnaire for audit data collection.
The first step is often to develop ways to obtain data that will help you assess what you are doing in your practice, so don’t worry if the first attempt at data collection isn’t successful. If you discover you need more data, try to implement changes that will make things better on the next attempt.
Data collection is only part of the process of measuring performance, and once you have collected your audit data, you need to determine what data analyses to undertake. Remember that the focus of data analysis for clinical audit is to convert a collection of data into useful information in order to identify the level of compliance with your agreed target/performance level. A common pitfall is the temptation to over-analyse or over-interpret the data that are obtained. Data analysis should be kept as simple as possible – if you are using hypothesis tests or advanced statistical methods, you are very likely to be answering a research question, not undertaking a clinical audit.
Most audits will involve calculating some basic summary/descriptive statistics (such as means or medians, and percentages). Simply calculating the percentage of your audit cases that complied with your criteria will allow you to decide if your results show that the changes you have made are as good as, or better than, your defined target performance level(s).
Some examples of ways in which we could monitor changes against criteria might be:
- Making sure that recurrence or complication rates for a specific disorder are equivalent to a recent multicentre case series found in the literature.
- Setting nosocomial infection rates to reduce by a certain percentage from the current baseline if no history of actual rates is available.
- Insisting that client-reported quality of life or pain score ratings should be equivalent to published results, should improve from what they currently are in your clinic, or should be greater than a predefined percentage.
- Necessitating that client satisfaction should improve, or remain static where it has already been at high levels.
- Requiring veterinary or owner compliance to be above a certain cut-off percentage (e.g. veterinary adherence to safety protocols would be expected to be 100%, while expectations of client compliance to puppy vaccination schedules may be set slightly lower).
- Stipulating that cost implications of implementing a new protocol should be comparable to those associated with the previous protocol, or that the new protocol will have a demonstrable cost benefit to the client and/or practice.
Example Scenario:
Small animal dental imaging
Over the year following implementation of dental radiography, there was a 20% increase in total extractions, which was consistent with radiography identifying additional diseased teeth in dogs and cats. The average dental invoice increased by 36%, providing a noticeable increase in gross income. No client queried the bill (although a practice policy of providing clear estimates for dental work had been instituted concurrently).
During the period of the audit there were 95 responses to the animal welfare questionnaire: 60 from dog owners and 35 from cat owners. 85% of dog owners indicated a positive response, with dogs showing increased activity levels (‘acting years younger’) and/or owners reporting reduced halitosis. Only 60% of cat owners indicated a positive response, with changes mentioned primarily associated with improved appetite. There were no reports of deterioration in health or quality of life, however the remaining 40% of respondents that indicated that they did not notice any particular response to dental treatment in their pets. Overall, 76% of owners reported significant improvement in their animal’s wellbeing following dental treatment; however, owner-reported outcome for cats fell just below Tom’s target performance level of 65%.
Key point:
Data analysis for Tom’s clinical audit required simple calculation of the percentage of dental cases meeting the audit criterion of improved owner-reported health-related quality of life following treatment.