What's Different About Evaluating Advocacy and Policy Change?

In this article from The Evaluation Exchange, author Julia Coffman cites four recommendations for advocacy and policy evaluators, which reflect how advocacy work differs from programmes and direct services. The differences she highlights are related to time frames, the pace of organisational work, and organisational size. Advocacy strategy evolves over time, while activities and desired outcomes can shift quickly, resulting from the fact that the programmes and services providing them can be affected by unpredictable variables. Additionally, advocacy organisations are usually small in size and capacity.


Her 4 recommendations are:

1. Use “real time” feedback, meaning report regularly and not only at the evaluation's conclusion. The purpose of real time reporting is to position the evaluation to inform ongoing decisions and strategy. The reasoning is that advocates need timely answers to the strategic questions they regularly face, meaning that evaluators may need to reserve part of their evaluation design for “rapid-response research.”
2. Give “interim” outcomes the respect they deserve by assessing advocacy for more than just its impact on policy change because, for example, advocates might build coalitions with other organisations or develop a network of community-based advocates who become active spokespersons. The author points out that these interim outcomes can be as important as the policy change itself. "Assessing a range of outcomes ensures that the evaluation does not unfairly conclude that the whole advocacy effort was a failure if the policy [or policy change] was not achieved."
3. Design evaluations that advocates actually want to use and that have user-friendly results. The author focuses on a "clean and simple interface" offering users "what they want when they want it..." This can be facilitated by helping advocates focus on what they want to evaluate (as opposed to their whole organisation or programme) and how to track that data internally to inform their work.
4. Be creative and forward looking. First, advocacy tactics are constantly changing and growing. For example, advocates are growing sophisticated in their use of electronic advocacy, resulting in a need to stay current on such techniques in order to evaluate them. Second, as advocacy tactics evolve, there is a need to make sure that the measures to assess them are meaningful. Coffman uses the example of questioning the use of tallying the number of emails sent (to a policy maker on an issue) as a significant indicator, when they may not get through or be read and counted (by the policy maker).

In summary, the author reiterates that advocacy differs from direct services and other programmes and, therefore, may require evaluators to make adjustments in order for evaluations to be relevant and useful within the advocacy and policy context.