Assess

2. How can we assess implementation in practice?

It is possible to assess implementation of EBVM through a number of formal and informal routes. For example, a formal evaluation of the process or outcomes of healthcare could be achieved through undertaking a clinical audit (see Assess 6).

An informal method of assessment could simply comprise personal reflection on individual cases at the end of a busy day. However, Assess also encompasses evaluation of, or reflection on, how the effectiveness and efficiency of any of the five steps in EBVM could be improved for addressing any future clinical questions. Personal reflection is discussed in more detail in Assess 4.

These assessments can be incorporated into small gaps of time throughout the day (for example, reflecting while having a cup of coffee, or mentally running through the day on your drive home), or specific times can be set aside to actively address individual questions or problems experienced in your daily practice, or by the business as a whole.

3. Why do we need to assess?

EBVM is a tool to help us improve our clinical practice. To achieve this goal, we need to assess whether the process is helping us in our clinical decision-making in an effective manner. If a practice policy, diagnostic procedure or treatment strategy has changed as a result of finding (Acquire), appraising (Appraise) and applying (Apply) the evidence, it is important to look at the consequences of this change.

Questions to consider:

3.1 Benefits of assessing

The benefits of reflecting on what we are doing and highlighting areas where we can make improvements are far reaching and can range from improved customer satisfaction and patient care, to improved biosecurity practices or financial returns.

The Assess step of EBVM allows us to assess and to uphold professional standards, and offers opportunities to improve the quality and effectiveness of the veterinary services we provide. Assessing also brings benefits beyond improving the quality of patient care including:

3.2 Assess as part of clinical governance

Clinical governance provides a comprehensive framework, including a number of different quality improvement systems (such as clinical audit, supporting and applying evidence-based practice, implementing clinical standards and guidelines, and workforce development) and promotes an integrated approach towards continuous quality improvement.

Figure 5: Measuring and improving our quality of care

For example, as can be seen in Figure 1 above, significant event audits might lead to the creation of checklists, guidelines and protocols, and the topic of a significant event audit may suggest an area for a clinical audit. The results of a clinical audit may lead to the creation of guidelines and protocols, and checklists, and the impact of these can also be measured by a clinical audit. A clinical audit might produce a figure for use in benchmarking. Benchmarking results might be improved by implementing guidelines and protocols, and checklists.

The RCVS Code of Professional Conduct for Veterinary Surgeons defines clinical governance as:

In the UK, the RCVS Practice Standards Scheme states that:

and that:

Veterinary Hospitals in the UK must also comply with the following:

and:

4. Reflection as a quality improvement tool

We often only reflect on the cases where something went wrong or we had an unexpected outcome. However, any decision can benefit from reflection (Koshy et al., 2017) – whether it is implementation of a well-researched new treatment, diagnostic or biosecurity protocol or simply reflection on a series of cases that were managed in a particular way in order to better understand how that management might be improved.  If we don’t reflect on what we are doing, our practices may remain stagnant and become rapidly outdated.

Reflection and unstructured EBVM is simple and easy to incorporate into everyday practice, but it is important to try to still follow the EBVM cycle. Reflection without support of the literature or without a clear question can lead to a vague outcome. Referring to the literature as part of your reflection allows you to utilise the entire EBVM cycle: Asking the correct question, Acquiring and Appraising the evidence,  Applying that information and then finally Assessing if the application was appropriate. Reflecting about how you navigated the five stages of the EBVM cycle also offers a simple way of assessing your own performance as an EBVM practitioner.

While it may require some additional planning and time management compared to personal reflection, team-based reflection can be a very useful quality improvement tool (Shaw et al., 2012).

Example Scenario

Postoperative physiotherapy

During monthly clinical rounds in a busy small animal practice, Sam reported that his last case of cranial cruciate ligament (CCL) rupture had re-presented with rupture of the CCL in the contralateral limb three months after surgery.

On presentation, Sam had noticed that the dog was not using the limb he had initially operated on fully, and suspected poor return to function of the operated limb had been a contributing factor. The owner was upset because when she had anterior cruciate ligament surgery herself, she had received an intensive programme of physiotherapy postoperatively, and wondered if the lack of physiotherapy could have been a factor in her dog’s new CCL injury.

One of Sam’s colleagues, Nicky, could recall similar cases in the practice and remembered reading a paper about early intensive physiotherapy used postoperatively after CCL surgery in dogs. Sam and Nicky worked together in an informal EBVM cycle of reflection, asking the question ‘In dogs with CCL injury, does postoperative physiotherapy compared to our traditionally prescribed controlled exercise programme improve function in the operated limb?’.

They identified a small number of papers that supported this approach, and although the evidence was not based on large multicentre trials in dogs, they felt there was sufficient evidence to apply physiotherapy as part of the postoperative management plan.

Together they found a local animal-qualified physiotherapist and implemented a new practice guideline for referral of all CCL cases for postoperative physiotherapy, starting with Sam’s patient following its second surgery.

The head nurse was involved with keeping a record of the cases of contralateral limb CCL rupture, as well as documenting client feedback on the physiotherapy, all of which were scheduled to be reviewed in a meeting in 12 months’ time.

Key points:

• This example shows a simple use of EBVM to address a problem following reflection on a case (although it could equally have been used pro-actively before a problem arose).

• A question was asked, information was acquired and appraised (albeit relatively informally), and the veterinarians applied the information in developing a new management guideline for dogs with CCL injury.

• In order to ensure that this new management protocol is actually doing what the veterinarians hope it will do, it is essential that they also implement a system to assess the response against clear outcome measures. In this case the outcomes are:

1. the number of cases presenting for contralateral limb CCL injury.
2. client feedback on the postoperative management guideline.

• A realistic time frame was set in order to ensure the guideline could be appropriately evaluated.

5. Clinical audit as a quality improvement tool

Clinical audit is:

Brief introductions to key elements of the stages of the clinical audit cycle are outlined in this chapter. For a more comprehensive overview, RCVS Knowledge has collated a wide range of quality improvement resources , including an e-learning course on clinical audit and a short summary infographic about the stages of clinical audit .

Figure 6 below shows the eight-stage clinical audit cycle from RCVS Knowledge , together with how these stages align with the commonly-used five-stage audit cycle defined by the National Institute of Health and Care Excellence (NICE, 2002).

Figure 6: The veterinary clinical audit cycle

The eight stages in the veterinary clinical audit cycle are listed in Figure 6 above, alongside related stages in the NICE five-stage audit cycle:

Other articles on clinical audit including some comparisons with research include: Viner (2009); Wylie (2015); Waine & Brennan (2015) and Waine et al. (2018a, 2018b).

6. Clinical audit in the veterinary world

Criteria-based (or standards-based) audits are the most common type of clinical audit undertaken in human health care, where high quality clinical guidelines are the preferred source for deriving audit criteria from evidence (NICE, 2002). One thing to be very careful about in comparing the audit process in veterinary medicine to the audit process in human medicine, is that there are many fewer clinical guidelines available in veterinary medicine . Therefore, in many situations, we need to rely on developing our own evidence-based criteria (Mair, 2006) .

Another key difference in human health care is the provision of protected time allocated for doctors to undertake medical audit activities (National Health Service (NHS) White Paper Working for Patients, 1989). This may not be feasible within a busy veterinary practice setting, and therefore careful planning is required to ensure completion of a successful audit project, including particular attention to selecting the right audit team, setting clear and achievable audit objectives and methods for data collection. All these considerations are discussed more fully on the following pages.

6.1 Where to start in clinical audit

Choosing an area to audit

The overarching aim of clinical audit is to improve the quality of care, therefore try to choose an audit topic that offers realistic potential to lead to measurable improvements for patients, clients or the practice team.

6.2 Making sure clinical audit gets done – the administrative side

The auditing process is not a light undertaking, and lack of time and resources are frequently reported as the main barriers to undertaking clinical audit in human medicine. Administrative colleagues in a practice team are a source of valuable knowledge and expertise, as they have extensive experience that can be used to help smooth the process and make sure all the practicalities of the audit have been addressed.

6.3 Setting audit aims and objectives

Remember the primary goal of clinical audit is quality improvement, so this should be reflected in your audit aims.

Ideally, audit aims should include verbs such as: improve, increase, enhance, ensure or change (Buttery, 1998), which convey the intention to measure current practice and identify where improvement may be needed.

Therefore audit objectives should contain a verb to describe what you want to do, the intervention or service you are evaluating and an aspect of quality related to that intervention or service (Maxwell, 1992 ).

6.4 Defining audit criteria/standards

The National Institute of Health and Care Excellence recommends using high quality clinical practice guidelines to develop audit criteria (NICE, 2002). There are several different appraisal tools available to help you evaluate the quality of clinical guidelines (for example, the AGREE II checklist developed by the AGREE Collaboration; see also Siering et al., 2013). These appraisal tools can be utilised to determine whether or not guidelines represent a suitable source for deriving your audit criteria. However, published clinical guidelines are scarce in veterinary medicine, with professional consensus statements  offering the closest available option in many cases.

If clinical guidelines or up-to-date systematic reviews are not available, a literature review may be carried out to identify (Acquire) the best (Appraise) and most up-to-date evidence from which audit criteria may be generated.

You may already have evidence-based clinical protocols or guidelines for your practice, which you can use to define ‘local’ audit criteria. Where there are no known or available criteria, one other option is to compare your audit data to historical clinical records over time.

Each audit criterion should have a performance level or target assigned to it (usually expressed as a percentage). Again, some overlap and confusion exists between different publications and guidance about clinical audit – some sources use the term ‘standard’ to define the performance level or target for expected compliance.

In many cases, we would aim to achieve 100% compliance with our evidence-based ‘best practice’, as set out by our audit criteria. However in practice, performance levels are a compromise between clinical importance, practicability and acceptability and for various reasons, it may not always be possible to meet 100% compliance.

Where you have set your target performance level based on published literature, you should note that levels of performance achieved in trials or research studies are helpful, but they often include very well defined study populations and should not be regarded as uniformly achievable in unselected patient populations in a practice setting. Clinical practice benchmarking can also be used to set and maintain target levels of performance.

There may be justifiable reasons why some cases might not comply with a specific audit criterion, and these cases should not be included in your audit data analysis. These exceptions should be defined along with your audit criteria, prior to data collection.

6.5 Measuring performance – data collection and analysis

For example, you might want to know if implementation of a protocol or new treatment has improved client satisfaction, decreased costs to the client, increased profit margins, saved time, improved veterinary compliance, reduced side effects, increased survival, or increased quality of life. In order to answer this ‘What if?’, you need to ensure you are asking the right questions to the right person (e.g. quality of life is often best evaluated by owners through practical questions involving the animal’s daily life, not by their veterinarian).

Your audit aims and objectives should be the primary consideration when deciding which data you will need to collect for your audit. You should only collect data required to show whether or not performance levels have been met for each criterion – collecting additional data provides little or no benefit, and is more time consuming.

The first step is often to develop ways to obtain data that will help you assess what you are doing in your practice, so don’t worry if the first attempt at data collection isn’t successful. If you discover you need more data, try to implement changes that will make things better on the next attempt.

Data collection is only part of the process of measuring performance, and once you have collected your audit data, you need to determine what data analyses to undertake. Remember that the focus of data analysis for clinical audit is to convert a collection of data into useful information in order to identify the level of compliance with your agreed target/performance level. A common pitfall is the temptation to over-analyse or over-interpret the data that are obtained. Data analysis should be kept as simple as possible – if you are using hypothesis tests or advanced statistical methods, you are very likely to be answering a research question, not undertaking a clinical audit.

Most audits will involve calculating some basic summary/descriptive statistics (such as means or medians, and percentages). Simply calculating the percentage of your audit cases that complied with your criteria will allow you to decide if your results show that the changes you have made are as good as, or better than, your defined target performance level(s).

6.6 Drawing conclusions and making changes

In many cases, the audit process may well indicate that no change is required. For example, an audit of peri-operative fatalities, or post-surgical wound breakdowns/infections, may indicate that rates have not changed recently and that they remain at levels that are similar to those in other clinics. The point of clinical audit is that it provides baseline data or reference points for comparison. Clinical audit also ensures that a process is in place that will likely result in early identification if things start to go wrong.

Alternatively, your data analysis and interpretation might identify some clinical areas that should be addressed (e.g. areas for improvements in care/performance/service provision etc.). You should use your findings to inform ways of improving, which should be the basis of the recommendations of your audit. These recommendations can be used to develop a realistic and achievable action plan, specifying what needs to be done, how it will be done, who is going to do it and by when. Implementing changes that will improve areas of poorer performance is often the hardest part of any audit.

6.7 Acting on the results of clinical audit – sustaining improvements

The overarching principle to successful implementation and adoption of both EBVM and clinical audit is to keep things small and simple, especially to start off with. It should be possible for you to set a modest goal of clear benefit, and to achieve it. Communication is also important – be sure you keep records of the process and of your findings so that you can compare the next cycle with the last. Discuss the tasks and progress with colleagues both during and after each audit cycle. Good communication will help to involve the more experienced (and often busiest) members of the practice who may at first be reluctant or unable to engage otherwise.

While the primary goal of clinical audit is improving performance, sustaining that improvement is also essential. The audit cycle is a continuous process, and requires re-auditing to ‘close the loop’. Re-audit is central to both assessing and maintaining the improvements made during clinical audit. The same methods for sample selection, data collection and analysis should be used to ensure that the data are valid and comparable with the results from previous audit(s).

Part of the audit process should be for you and your colleagues to identify thresholds that might trigger you to further action. That action might involve further in-depth investigation, or it may involve an increased frequency of the audit cycle to see if preliminary results are indeed a trend in the wrong direction, or just an anomaly that should be monitored but perhaps not acted on at this time. On the whole, a common sense approach is required. However, an explicit and systematic process can help veterinary practices avoid falling into complacency or inertia.

Where an initial audit demonstrates that desired performance levels are not being reached and an action plan has been put in place, the audit should then be repeated to show whether the changes implemented have improved care or whether further changes are required. This cycle is repeated until the desired standards are being achieved. Where the initial audit showed that no changes or improvements were required, re-auditing allows you to ensure that the high standards of care are being maintained.

Sustainable improvements in quality of care are going to be more readily achieved where everyone in the practice (or profession) is aware, and supportive, of planned audit activity. In addition to fostering a culture of continuous improvement, longer term success may require practice development; for example, there may be a requirement for training or organisational changes, such as modifying format, content or quality of clinical records, updating or changing practice management software or time allocation for team members involved in clinical audit to gather and analyse.

7. Beyond clinical audit – alternative ways to assess

8. Quiz

9. Summary

10. References