Questions, Methods, and Budget, oh my!
Most evaluators are quick to say that the evaluation questions drive the methods. I think most evaluators would agree with this statement. However, what do you do about that pesky budget? How does the budget play a role in determining the questions and/or methods?
Ideally, we’d have an unlimited budget and would always use the most appropriate method to answer the client’s evaluation questions. This is clearly a pipe dream, however, and so we need to think critically about how the budget factors in to the design of our evaluations.
I am thinking there are a few ways in which a budget affects our designs and how we account for it when designing our evaluations.
Budget --> Questions
First, we often have too many evaluation questions anyways! Jane Davidson argues that we should have 7 evaluation questions, plus or minus 2. I think I would argue even 7 is too many, but regardless I think we all realize how easy it is to keep piling on the evaluation questions, especially once we start breaking them down into sub-questions.
If this is the case and we have too many evaluation questions, an easy way the budget can affect our designs is simply by culling some of the evaluation questions. These evaluation questions would need additional analyses, data, and possibly new methods, so this is an easy place to start when trying to fit in the evaluation design into your budget. Work with your client to identify that need-to-have and the nice-to-have questions and focus your resources on the need-to-have ones.
Budget --> Methods
Now imagine you laid out all your evaluation questions and then you developed how you would ideally answer those evaluation questions (see this blog post for the budget template that I use). When you budget that out, you realize you are way over budget. I typically write out my budget by method rather than by question. This makes it really easy to realize which method could be cut to reduce the overall budget.
Budget --> Questions and Methods
However, I think there is an important caveat here. If you decide that, for example, surveys are too expensive to do in addition to the other methods, you need to be sure that it doesn’t prevent you from answering the critical evaluation questions. This brings us back to the first section, where we identify which are the need-to-have evaluation questions. If surveys were going to answer one of those questions, can you still answer the question with one of your other methods adequately?
Questions and Methods --> Budget
I do not think this solution would always work, but let’s walk through it anyways. If a client has multiple need-to-have evaluation questions and they require methods that push you over your budget, one possible solution is to simply tell your client you need a greater budget. With some organizations flush with money, this might work, but I tend to work for non-profits that are already strapped for cash. However, this might open up the opportunity to discuss applying for external funding to support these endeavors.
Conclusions
Overall, as much as I agree with the statement that the questions drive the methods, I think we need to critically think about how the budget affects our evaluation design as well. This is particularly important for emerging evaluators who tend to struggle most with the project management and design side of evaluation (in my experience). Based on my experience teaching new and emerging evaluators, they often puzzle most over the budget.
Questions for Discussion
- Do you agree that emerging evaluators often struggle most over the budget?
- If so, how do you help them get over that hurdle?
- If not, what do you find emerging evaluators most often struggle with?
- What do you think about my approach of integrating the budget with the evaluation questions and methods?
- Do you have another approach that you use to integrate the budget with the evaluation questions and methods?