As you go about making change in your organization, there are a few questions that naturally arise: (1) How will you know which changes worked and which did not? (2) How will you know which changes resulted in an improvement? (3) Which change(s) is the most important and resulted in the most significant improvement?
It is therefore important to set up systems that allow you to answer these questions. By collecting data before, during, and after the change you implement, you can measure, evaluate, and compare your agency's progress with respect to the goals you set out. The process of measuring change should speed the improvement process; you should begin with simple measures rather than spending time developing a complex measurement system. This primer provides you with a six-step process for the effective measurement of the impact of change.
It is extremely important to establish clear definitions of your measures prior to starting the change process. The measures should clarify the project objectives and should be agreed upon by key stakeholders. This ensures that the results are interpretable and accepted within the organization. NIATx has defined four key measures for use by our members: (a) time from first contact to first treatment, (b) no-shows for assessment, (c) client admissions by level of care, and (d) continuation from the first to fourth treatment session.
Other measures you might consider include: time from first contact to assessment, treatment completion rates, bed days, units of service offered, level of care transition (e.g., time from inpatient discharge to receipt of outpatient service) or the continuity of care (e.g., the percentage of clients discharged from residential who receive outpatient care within seven days).
Tip 1: Clear definitions are critical to measure change successfully.
Never start a change process without collecting baseline data. In doing so, you should clearly define a starting point for the change and work backwards to collect two to three months of baseline data before making any changes. This process anchors the change (pre-change) and enables your agency to measure the impact of the change over time (post-change). Baseline data should be collected using agreed upon measure definitions. More importantly, the collection of baseline data allows an agency to answer four questions and make adjustments as necessary:
Tip 2: Never start a change process without collecting baseline data.
Establish a clear improvement aim or target (e.g., reduce client no-shows from 65 percent to 25 percent). Such a target should: (a) be realistic yet ambitious (e.g., do not expect 100 percent treatment completion), (b) be linked to project objectives, and (c) avoid confusion, especially with percentages (e.g., say improve no-shows from 65 percent to 25 percent, rather than improve no-shows by 62 percent).
Remember to be flexible when establishing an aim. If the change project uncovers new information that suggests changing the aim, change it. If the aim is too ambitious, set a realistic aim that still challenges the agency to improve. If the aim is easily achieved, set a more ambitious aim that stretches the agency's capacity to improve.
Tip 3: The aim should challenge the organization.
The ability to establish consistent channels for collecting measurement data on a regular basis is a crucial part of the change process. Such channels may exist in existing data systems, but in other cases you may need to manually collect the data. Often in the PDSA change process, it is easier to rely on manual collection for quick and rapid feedback on the success of the change. This means relying on small samples collected over short time periods to measure change progress. For example, your agency might use reminder phone calls to reduce no-shows and your rapid measure might collect success or failure (e.g., simple tally marks) of the intervention for the next 10 clients with scheduled appointments. These results would then be compared to the baseline measure. Existing systems might be used for longer-term reporting (e.g., monthly) on the change progress. If your agency can affirmatively answer the questions in 2, you most likely have systems in place to consistently collect measurement data related to the change.
Tip 4: Consistently collect measures related to the change.
Over time your agency will collect both pre-change (baseline) and post-change data, and the data should be shared with the change team as well as others in your organization. The most effective tool for sharing this information is charting your progress over time using simple line graphs created in Excel. These powerful visual aids should follow one simple axiom: one graph, one message.
Charts like the one on the previous page offer Change Team members, as well as the entire organization, several key pieces of information. Charts can be used to: (a) highlight the baseline (pre-change) data (see the red lines), (b) identify when a change was introduced in your agency (see notations), (c) visually represent the impact of individual changes over time, and (d) inform your agency about how it well it is sustaining the change over time. In developing charts, your agency should consider the following points:
In general, the charts should be used to compare your progress over time, not to compare programs or individuals within your organization. In some instances, it does make sense to compare performance by program or counselors (e.g., no-show or continuation rates), but using this comparative approach to change behavior requires careful consideration and implementation. Comparisons should not be used in a punitive way; they should only be used to improve performance
Tip 5: Charts are powerful visual aids to communicate ONE message.
Measuring the impact of change does not stop here; in fact, it is only the beginning. The most important step in the process is to ask: What is the information telling me about change in my organization?
If change is successful, the information you have collected may tell you which intervention had the most success in meeting your aim. For example, one NIATx member sought to reduce the time from first contact to treatment (aim) by increasing professional staff availability. The change reduced the time from first call to first treatment from 18 days to 5 days, and in examining their data, the agency found that only physician and nurse practitioner availability played a role in the improvement.
Unsuccessful changes also afford your agency the opportunity to ask "Why?" Another NIATx member examined the characteristics of clients not continuing through the fourth treatment session and found that clients admitted to treatment on Fridays were more likely to drop out. The organization then stopped offering Friday admissions.
Tip 6: Ask questions. Do not accept the results at face value.
Measuring the impact of change is an important aspect of successful organizational improvement. These six steps are designed to help guide your agency in the timely and accurate measurement of change.