How to Measure the Impact of Change

As you go about making change in your organization, there are a few questions that naturally arise: (1) How will you know which changes worked and which did not? (2) How will you know which changes resulted in an improvement? (3) Which change(s) is the most important and resulted in the most significant improvement?

It is therefore important to set up systems that allow you to answer these questions. By collecting data before, during, and after the change you implement, you can measure, evaluate, and compare your agency's progress with respect to the goals you set out. The process of measuring change should speed the improvement process; you should begin with simple measures rather than spending time developing a complex measurement system. This primer provides you with a six-step process for the effective measurement of the impact of change.

Measuring the Impact of Change

1. Define your measures

It is extremely important to establish clear definitions of your measures prior to starting the change process. The measures should clarify the project objectives and should be agreed upon by key stakeholders. This ensures that the results are interpretable and accepted within the organization. NIATx has defined four key measures for use by our members: (a) time from first contact to first treatment, (b) no-shows for assessment, (c) client admissions by level of care, and (d) continuation from the first to fourth treatment session.

Other measures you might consider include: time from first contact to assessment, treatment completion rates, bed days, units of service offered, level of care transition (e.g., time from inpatient discharge to receipt of outpatient service) or the continuity of care (e.g., the percentage of clients discharged from residential who receive outpatient care within seven days).

Tip 1: Clear definitions are critical to measure change successfully.

2. Collect baseline data

Never start a change process without collecting baseline data. In doing so, you should clearly define a starting point for the change and work backwards to collect two to three months of baseline data before making any changes. This process anchors the change (pre-change) and enables your agency to measure the impact of the change over time (post-change). Baseline data should be collected using agreed upon measure definitions. More importantly, the collection of baseline data allows an agency to answer four questions and make adjustments as necessary:

  1. Was the data defined to ensure that we collect exactly the information needed?
  2. How accurate is the data? Does accuracy matter?
  3. Does the process ensure that the measures will be collected consistently?
  4. Do trade-offs exist? Is quality more important than the time required to collect data?

Tip 2: Never start a change process without collecting baseline data.

3. Establish a clear aim

Establish a clear improvement aim or target (e.g., reduce client no-shows from 65 percent to 25 percent). Such a target should: (a) be realistic yet ambitious (e.g., do not expect 100 percent treatment completion), (b) be linked to project objectives, and (c) avoid confusion, especially with percentages (e.g., say improve no-shows from 65 percent to 25 percent, rather than improve no-shows by 62 percent).

Remember to be flexible when establishing an aim. If the change project uncovers new information that suggests changing the aim, change it. If the aim is too ambitious, set a realistic aim that still challenges the agency to improve. If the aim is easily achieved, set a more ambitious aim that stretches the agency's capacity to improve.

Tip 3: The aim should challenge the organization.

4. Consistently collect data

The ability to establish consistent channels for collecting measurement data on a regular basis is a crucial part of the change process. Such channels may exist in existing data systems, but in other cases you may need to manually collect the data. Often in the PDSA change process, it is easier to rely on manual collection for quick and rapid feedback on the success of the change. This means relying on small samples collected over short time periods to measure change progress. For example, your agency might use reminder phone calls to reduce no-shows and your rapid measure might collect success or failure (e.g., simple tally marks) of the intervention for the next 10 clients with scheduled appointments. These results would then be compared to the baseline measure. Existing systems might be used for longer-term reporting (e.g., monthly) on the change progress. If your agency can affirmatively answer the questions in 2, you most likely have systems in place to consistently collect measurement data related to the change.

Tip 4: Consistently collect measures related to the change.

5. Chart your progress

Over time your agency will collect both pre-change (baseline) and post-change data, and the data should be shared with the change team as well as others in your organization. The most effective tool for sharing this information is charting your progress over time using simple line graphs created in Excel. These powerful visual aids should follow one simple axiom: one graph, one message.

Time from Request to 1st Treatment Session

Charts like the one on the previous page offer Change Team members, as well as the entire organization, several key pieces of information. Charts can be used to: (a) highlight the baseline (pre-change) data (see the red lines), (b) identify when a change was introduced in your agency (see notations), (c) visually represent the impact of individual changes over time, and (d) inform your agency about how it well it is sustaining the change over time. In developing charts, your agency should consider the following points:

  • Who will the measures be reported to (e.g., Executive Sponsor, Change Team, entire agency)?
  • How often will the measures be reported and the charts updated (e.g., monthly, weekly)?
  • What else besides the measures on these simple line charts should be reported each month?
  • To enhance the power of these line charts, the change team should also consider the following information in their progress reports:
    • State the project aims so everyone is informed of what the Change Team is trying to accomplish.
    • Provide measurement definitions, change targets, baselines and current progress for each measure. Remember, each line chart should present one message and one message only.
    • List the tested PDSA change cycles to help inform others about what you tested and what worked, as well to seed new ideas for spreading change ideas. You can use the NIATx Change Project Template to record your PDSA change cycles and project aims, as well as to define the parameters of your change projects (one aim, one location, one level of care for one specific population ).
    • Delineate the changes made on the line chart and clearly label when they were started.
    • Highlight other successes, issues and challenges, key future action steps or general progress related to the Change Team.
    • Identify the point in time in which the change idea was sustained within the organization (i.e., adapted and became a part of the organizational culture).

In general, the charts should be used to compare your progress over time, not to compare programs or individuals within your organization. In some instances, it does make sense to compare performance by program or counselors (e.g., no-show or continuation rates), but using this comparative approach to change behavior requires careful consideration and implementation. Comparisons should not be used in a punitive way; they should only be used to improve performance

Tip 5: Charts are powerful visual aids to communicate ONE message.

6. Ask questions

Measuring the impact of change does not stop here; in fact, it is only the beginning. The most important step in the process is to ask: What is the information telling me about change in my organization?

If change is successful, the information you have collected may tell you which intervention had the most success in meeting your aim. For example, one NIATx member sought to reduce the time from first contact to treatment (aim) by increasing professional staff availability. The change reduced the time from first call to first treatment from 18 days to 5 days, and in examining their data, the agency found that only physician and nurse practitioner availability played a role in the improvement.

Unsuccessful changes also afford your agency the opportunity to ask "Why?" Another NIATx member examined the characteristics of clients not continuing through the fourth treatment session and found that clients admitted to treatment on Fridays were more likely to drop out. The organization then stopped offering Friday admissions.

Tip 6: Ask questions. Do not accept the results at face value.

Measuring the impact of change is an important aspect of successful organizational improvement. These six steps are designed to help guide your agency in the timely and accurate measurement of change.