Advocacy groups are driven by the passion of their activists towards the goal. Whether it be social reform or government policy change or an economic issue, the workforce of the advocacy strongly believe in the positive result they would bring to the world. However, how would an advocacy group measure the progress they have made when the long-term outcome is yet to be reached? How would they know they are taking the right steps to influence that outcome?
The answer is “through advocacy evaluation”.
What does that even mean?
To start off, evaluation methods or evaluation models are systematic approaches to collect data which is used to determine whether the strategy is making progress towards its intended result.
The data can be qualitative or quantitative, meaning the process to analyze the data varies from model to model.
Advocacy evaluation is used to measure the progress made by an advocacy effort. The purpose is to identify what works or what doesn’t in their approach and improve future processes as they move ahead.
Some common data collection methods exist which can also be applied in evaluating an advocacy. They are:
- Surveys or questionnaire – To collect relevant audience’s perspective and feedback
- Case Studies – Detailed analysis of specific campaign strategies and results
- Focus Groups – Discussion with stakeholders to gather their opinion
- Media Tracking – Measuring how well the media covers the particular issue
- Media Content or Framing Analysis – Qualitative analysis of how the media reports the issue
- Participant Observation – Firsthand experience and data collection through participation
- Policy Tracking – Monitoring the progress made in the reform of a policy
- Public Polling – Poll of a sample group of relevant public about their stance
The need for special evaluation methods
The ones above are great ways to evaluate progress in strategy in any other system. But advocacy efforts are dynamic and complex, plus the ground plan to the outcome differs greatly for every advocacy group. Moreover, the influence is to be measured rather than the result of the influence for an ongoing advocacy effort. All this makes the common evaluation methods unsuitable for advocacies.
This led to some unique methods to be developed for assessing advocacy processes. They have been tested and found to be insightful, time-friendly and compliant of the advocacy sector. Let’s look at four of these innovative evaluation models, how they work and the type of advocacies they measure.
Evaluation models designed for advocacies
|Bellwether Methodology||Structured interview with thought leaders in the concerned policy’s domain to gauge the influence caused by the advocacy drive||Figure out influence on a policy and how it affects policymakers’ decision|
|Policymaker Ratings||Advocates rate individual or group of policymakers based on their level of support perceived and influence they hold to affect concerned policy||Analyze support and influence of individual policymakers on the issue||Predicting the likelihood of campaign to sway policymaker decision|
|Intense Period Debriefs||A focus group or selected activists participate in “debrief type” interviews after an intense activity period giving their perspective of public response, the strategies adopted and outcome during that period.||Keep record of key advocates’ behind-the-scenes account of the turn of events during a period of high activity|
|System Mapping||Visual mapping of teams, people or institutions to pinpoint where communication needs to improve within the system and then establish ways to measure the changes||Improve an organization’s functioning and build relationship among its parts and players|
A bit about each of them
A bellwether, in this context, is a knowledgeable member of the public or private sector who’s politically well-informed and whose opinion on policy issues hold substantial weight and predictive value. The model involves conducting interviews with a group of relevant bellwethers not tied to the issue or policy at hand. The data collected helps gauge the awareness caused by the advocacy’s efforts, its reach among decision makers in the sphere and the possibility of policymakers to act accordingly. This evaluation model was developed by the Harvard Family Research Project.
The Bellwether methodology involves five steps:
- Determining the relevant bellwethers within the issue’s sphere of interest. The categories are determined to select the bellwether sample from.
- Bellwethers are selected with at least half of the group not being connected to the advocacy or policy issue. Other selection criteria like political affiliation, gender or ethnic diversity are used to ensure the awareness detected from the interview can be linked to advocacy efforts rather than extraneous reasons.
- The interviews are set up making sure the bellwethers have no prior knowledge of which policy or issue is the focus of the study
- The interview questions seek to record unprompted mention of the advocacy or their message in the bellwethers’ discourse about the issue. Once brought into context, questions focus on their understanding of the matter and their view on its future.
- The data collected informs how well the advocates’ message has penetrated the concerned audience and the gaps that remain to be filled in their communication.
This evaluation model was developed out of the necessity of a method to analyze policymakers’ support for a particular issue or proposal. Developed by the Harvard Family Research Project, it consists of a rating process where activists rate policymakers connected to the issue based on their perception of the policymakers’ stance on the issue.
The model involves three scales that assesses:
- Individual policymaker’s support of the issue based their public action and standpoint
- Individual policymaker’s influence over the final decision made on the issue
- Rater’s level of confidence in the accuracy of the first two scales
A 3-5 group of advocates takes part in the rating process and the results are averaged. Policymakers’ party affiliation, district representation etc. are also taken into account and the ratings are aggregated. The data helps to predict how likely the advocacy efforts are to bring a change in the policy decision.
Intense Period Debriefs
For advocacies, activity varies during different times of the year. Many times during a period of intense action, evaluation and progress analysis is left out because all hands contribute to groundwork. Recognizing this is often the case in the advocacy sector, this model was developed by Innovative Network to fill data gaps after the activity period is over.
The model involves selecting a focus group with first-hand experience of the campaign who take part in ‘debrief type’ interviews about their experience. The data collected is used to capture:
- Public reaction toward the advocacy and political context of the period
- How the decision-making was done in response to events that occurred
- Individual perspective of the results
- How strategies should be planned in the future for similar action periods
The key goal of this evaluation model is to get an insider view of the processes taking place during the intense period from individuals involved in different levels of decision-making. The data helps to structure future processes during times of high activity.
This model varies from the other three in that the data returned is visual and is presentable to stakeholders as illustrative representation. It is useful for organizations working on improvement through fine-tuning their inner system.
The method of mapping starts with interviews with people working within the system to gather data on the communication and link that exists between groups, departments and people in the organization. The network is analyzed and a visual map is prepared on what the system looks like and how it functions. The parts and relations that need improvement are then identified and ways to measure progress are developed using mathematical algorithms. The system maps serve as illustrations of where change needs to occur and guides evaluation as progress is made on those changes.
These four models of evaluation were developed to address specific requirements of the advocacy sector. Of course, when it comes to specific advocacy groups, the processes and ground game are too dynamic to measure using an existing method. In such cases, new models are worked out which are capable of gathering data from the system in an efficient way. Keeping the workforce filled in on the progress builds motivation, which is the fuel to all advocacy efforts.Tags: advocacy, Grassroots Advocacy, Non profits