Eye on Results: Susan Philliber on Outcome, not Output
13.04.09
The first thing you need to know is the terms used to evaluate programs.
Needs and assets assessments. They answer the question, “What do we need to do?”
Before a project begins, it is important to assess: What, exactly, is the problem and how might it be changed? You look for the causes of the problem, but also any strengths in a community or media organization that might support the change you want. These assessments can be as simple as a phone call to key participants, or involve hundreds of community surveys or interviews. For example: Suppose the quality of journalism produced by an organization is relatively poor. Before you decide what to do, you have to understand why it’s so poor. Is it lack of training, lack of time, lack of editing skill, political issues? You may determine the problem is a general lack of editing skills, but you might also find a few journalists within the organization who are strong editors and who could train the others – a definite asset.
Process or formative evaluations. They answer the question, “What did we do?”
Once your program is underway, you need to identify which people you reached, whether they were your intended targets, and what services you provided them. Example strategies include sign-up sheets at workshops, a system to record contacts with those you serve, or a log of activities. If your program is successful, this will show how that happened. And if your program fails, this could show why. Maybe you didn’t reach your target group, or didn’t reach them often enough: You wanted to coach 10 newsroom staffers, but you only reached three; or you wanted to deliver six workshops on investigative reporting and you only actually held two. A good process evaluation can work as a “smoke detector” to warn of problems.
Outcome or summative evaluations. They answer the question, “So what?”
Tracking activity is important but holding six workshops is a process, NOT an outcome. Outcomes are results, changes – in knowledge, attitudes, skills or behavior. Before a project even begins, ask yourself: “What will participants know that they didn’t know before; believe that they didn’t believe before; or do that they couldn’t do before this program?”
Cost benefit and cost effectiveness evaluations. They answer the question, “What do results cost?”
Funders always search for efficiency – getting the biggest results with the smallest price tag. So when planning projects, it is generally wise to try to think about how the costs appear to a potential funder. For example, if a training program for five people costs of $50,000, that puts the cost per person at $10,000. Does that sound reasonable?
Now, a few words about the most common reasons a program fails, and how evaluations can help identify the problems.
It didn’t go as planned.
Many things can go wrong in a program. Maybe you wanted senior managers, but only lower-level staff came – and they did not have the authority to implement your goals. This could be a case of a fine program that accomplished nothing because it didn’t reach the people it needed to. Or maybe the press of other commitments meant your participants missed several training sessions. A good process evaluation will let you know if this is happening.
It was not strong enough to accomplish your goals.
Changing journalists or their organizations is not easy work. It may take a long time, with strong, sustained efforts. Also, bear in mind that new information may or may not change how people behave. That’s why it’s important to think hard in the early planning stages about the scale of program you will need to accomplish your goals.
