09 Oct 2012
Context and its Importance in Evaluation
Context and its Importance in Evaluation
Right on the heels of Jane Davidson’s great new minibook (see previous post) I opened Debra Rog, Jody Fitzpatrick, and Ross Conner’s New Directions for Evaluation publication on Context in Evaluation. I have to say that it has been a while since I enjoyed a New Directions publication so much!
This publication is just excellent: It begins by providing a historical overview of how context in evaluation has been treated (or not treated), and then argues for greater attention to context across multiple areas/contexts and their intersections. These include Decision-making, Problem, Intervention, Evaluation, Broader Environment; within each area one should assess multiple contextual dimensions (physical, organizational, social, culture, tradition, historical, political) (Bragg, 2012). Multiple examples of evaluations using this model are then presented to help the practitioner consider ways in which they might apply Bragg’s model /framework to their own evaluative efforts.
What I liked so much about this publication was how directly it speaks to questions I have in evaluation (how to consider and systematically assess context in evaluation), how it provides strong practitioner examples of how to do so, and how it addresses the implications assessing context has for evaluative findings. It is also very well written and frankly, inviting. Theory is presented but it is well-balanced with application.
This is definitely one of the NDEs that I will reach for often to improve my work. Once you read it too, I think it will become a go-to publication for you!
Nothing makes me more excited about blogging than when I have the opportunity to share new evaluation ideas. “Actionable Evaluation Basics:Getting succinct answers to the most important questions” by Jane Davidson provides insights that will be new to many regarding making evaluation more meaningful to stakeholders.
In this minibook Dr. Davidson defines actionable evaluation as:
- clearly relevant to the key actions, decisions, and thinking of those the evaluation needs to inform
- going right to the heart of what is really important, and doesn’t get lost in the details;
- favoring approximate answers to important questions over accuracy to four decimal places on trivia;
- resisting being lured into a focus on the outcomes that are most easily measured;
- presenting findings in a way that is simple, but not simplistic;
- useful — at both strategic and practical (or operational) levels;
- influencing and clarifying thinking, action, and decision-making; and
- providing insights that help people figure out what actions to take.
To get to actionable evaluation she presents the six critical elements and then details how one, as an evaluator, addresses each of these elements in an evaluation:
- a clear purpose for the evaluation;
- the right stakeholder engagement strategy;
- important, big picture evaluation questions to guide the whole evaluation;
- well-reasoned answers to the big picture questions, backed by a convincing mix of evidence;
- succinct, straight to the point reporting that doesn’t get lost in the details; and
- answers and insights that are actionable, that we can do something with.
Whereas some readers of this minibook will assert they that already do actionable evaluation, that nothing new is presented here, I would argue that few evaluation studies I have read would qualify as actionable evaluation for two main reasons: Evaluators measure what they can measure rather than risk finding “approximate” answers to the right questions and evaluators generate only evidence, not evaluative conclusions, telling us “what’s so” (e.g., what the outcomes are) but not “so what” (how good, valuable, or worthwhile the outcomes are). Make your evaluation truly a measure of worth, merit, or value and more actionable (i.e., utilizable) by reading this short publication and attempting the simple methodologies presented. It may be the best $3.99 you ever spend!
.
15 Jan 2012
Dealing with Small Sample Sizes
The most recent ATE newsletter developed by Western Michigan University for NSF Advanced Technology Education projects and centers has a nice article about how to handle small sample sizes (along with some other great articles).
Small sample sizes are something all evaluators face at some point in their professional lives. And while evaluators may want ignore small sample sizes and treat such data as they might had they had larger sample sizes, evaluators might want to consider small sample sizes as opportunities! Specifically an opportunity to collect qualitative data that might substantiate results and potentially provide more powerful stories than numbers alone.
Other recommendations ( these from Eboni Zamani-Gallaher) include:
· Try to gather data on everyone by using a census, rather than a sample. Just remember to limit data analysis to descriptive statistics, rather than inferential.
· Be upfront about the limitations and document your sampling strategies, decisions, and criteria.
· See it as an opportunity to keep evaluation costs low recognizing that a large study without sufficient resources can under-power results.
Additionally:
· Hesitate to report percentages, or don’t at all – report fraction instead, as percents can be misleading and may overstate results.
· Do not conduct quantitative tests of statistical inference where data requirements are at best ignored and at worst violated.
· If small sampling sizes are the result of missing data, then there may be other possibilities for dealing with this issue. One way is imputing missing data , of which there are multiple methods, including mean substitution, regression, and more intricate imputations such as multiple imputation using virtual datasets.
I read a lot of different blogs as part of my personal and professional learning and last year began reading Seth Godin’s blog. An entrepreneur, marketer, and author, he wrote the following in April 2011:
“There are two jobs available to most of us:
You can be the person or the organization that’s perfect. The one that always ships on time, without typos, that delivers flawlessly and dots every i. You can be the hosting company or the doctor that might be boring, but is always right.
Or you can be the person or the organization that’s interesting. The thing about being interesting, making a ruckus, creating remarkable products and being magnetic is that you only have to be that way once in a while. No one is expected to be interesting all the time.
When an interesting person is momentarily not-interesting, I wait patiently. When a perfect organization, the boring one that’s constantly using its policies to dumb things down, is imperfect, I get annoyed. Because perfect has to be perfect all the time.”
This post made me wonder where evaluation and evaluators fit? Should evaluation/evaluators be perfect or should it/we be interesting?
I would argue that good evaluation is perfect whereas better evaluation is interesting – and thus most likely imperfect some of the time. And that such imperfection is just the cost that comes when taking risks.
Let me try to explain: One example of a huge hit has been evaluators’ forays into data visualization and reporting. Attendees at the American Evaluators Association meeting in 2011 could attend multiple sessions on using data visualization that were intended to improve client’s understanding of, interest in, and use of data and evaluation. The evaluators presenting these materials had all taken risks with their evaluations and clients to use new techniques to increase greater stakeholder buy-in. Examples included attempting different reporting formats, using data dashboards, and creating new data visualizations.
Of course, there are is always the flip-sides to the same coin. I recently took a risk when sharing some data with clients by putting together a Prezi presentation, thinking “why not try something different from PowerPoint”? While they loved the presentation, they were much more interested in what they could do with Prezi than the actual data I presented. It took a lot of time and effort to steer the conversation back to the data!
But my imperfection had a few benefits. It reminded me of the value of my data message and the need to ensure that nothing obscures that message when sharing it with clients and /or stakeholders. It also allowed my clients to see that I was taking risks to better reach them, which they appreciated, and knew I would do when helping them with their own outreach and messaging. So that old saying, ‘nothing ventured, nothing gained’ was true in this case.
So the moral to the story, for me at least, is to try to be more interesting, even at the risk of not being perfect. I think risk is necessary to growing evaluation and meta-evaluation and truly making evaluation a trans-discipline, to use a term from Michael Scriven.
28 Sep 2011
What I’m Reading
I’m always interested in what evaluators are reading, so thought I would share below some of the books and blogs I’ve been reading lately. I would love to know what you are reading and how is it improving your evaluation and consulting practices. Please leave me a post!
Books
I’m constantly referring to this book for ideas – definitely a classic: Campbell, D. & Stanly, J. (1963). Experimental and quasi-experimental designs for research. New York, NY: Wadsworth Publishing.
Okay – not a straight read – but another go-to book: Scriven, M. (1991). Evaluation thesaurus (4th Edition). Thousand Oaks, CA: Sage Publications, Inc.
Very useful for helping me understand the history of evaluation: Alkin, M. (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage Publications, Inc.
Okay – not a straight read – but another go-to book: Scriven, M. (1991). Evaluation thesaurus (4th Edition). Thousand Oaks, CA: Sage Publications, Inc.
Very useful for helping me understand the history of evaluation: Alkin, M. (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage Publications, Inc.
Because almost all evaluators are consultants as well: Block, P. (2011). Flawless consulting: A guide to getting your expertise used (3rd Edition). Hoboken, NJ: John Wiley & Sons.
Blogs
Stephen Few on data visualization (perceptualedge.com ) and Nancy Duarte on presenting blog.duarte.com/), because both are central to my evaluation work.
For Pleasure
A beautiful and haunting “evaluation” of the Mann Gulch Fire. This is probably the 5th time I’ve read it: Maclean, N. (1993). Young men and fire. Chicago, IL: University of Chicago Press
What I Hope to Read Next
Michael Quinn Patton – Essentials of Utilization-Focused Evaluation. Just arrived!
Like most evaluators, I gather lots of Likert and numerical data. Over the years I have changed the way I report these data so that my clients can better grasp what the results mean.
Likert Data
Almost all of my Likert data are numerically anchored. So, for example, Strongly Disagree is anchored at a 1 and Strongly Agree would be anchored as a 5, with all other qualitative options anchored numerically in-between. In these occasions I tend to report the sample size (n), minimum value, maximum value, mean, and standard deviation, as well as the percent of persons who Agree or Strongly Agree (i.e., those who selected a 4 or 5). I can report the mean and standard deviation because I have numerically anchored m Likert scale. This allows my clients to see right away how persons responded, on average. By reporting the standard deviation I am able to show them how “spread out” the data are, another point of information for my clients. Last, by reporting the percent of persons selecting a 4 or 5, I provide my clients an additional data point for them to consider.
Numerical Data
Like most evaluators reporting numerical data, I report the sample size (n), minimum value, maximum value, as well as the mean and standard deviation. Lately I’ve also begun reviewing the median value in a dataset and reporting that value as well. Oftentimes this data point is very revealing when compared to the mean, especially if they are very different in value. Explore what this means for your dataset and share with your client.
ALL DATA
Even better than presenting these data in table format, consider providing them in graphical form. Excel allows you to do most graphs you might need. Using text boxes and shapes such as arrows I can usually point out the mean and median values and the interval that is bounded by the sd.
I eagerly awaited the newest issue of AJE as my colleague Melanie Hwalek and her company SPEC Associates were to be featured. After returning from a week of site visits in CENLA, there it was. (By the way CENLA is Central Louisiana, not Central Los Angeles – much less shopping – but LA doesn’t have crawfish, does it?).
The article, “Aspects of Successful Evaluation Practice at an Established Private Evaluation Firm” by Paul Brandon, Nick Smith, and Melanie did not disappoint. Like many of my conversations with Melanie, I came away with multiple ideas about how to make my evaluation work more meaningful to my clients and myself, my firm more successful, and my clients more knowledgeable about evaluation.
While I highly suggest all evaluators read this article, it has interesting insights for those of us who consider ourselves evaluation consultants. Particularly interesting to me were the conditions Melanie identified as necessary for exemplary evaluation practice to occur. These include:
- Clients who seek to learn, change, and improve;
- Internal “evaluation champions” within client organizations who will support the conduct of evaluation studies and use of results;
- Effective feedback mechanisms from evaluators to clients so evaluation information is understood and used; and
- Trust between evaluators and client organizations.
Are there other conditions that need to be in place for exemplary evaluation to occur – whether on the evaluator side or client side? Certainly evaluators must come with lots of skills – hard evaluation skills and soft evaluation skills (as Melanie notes, good evaluators listen more and talk less), but what other conditions should be in place?
And, as a corollary, what conditions are necessary for a firm to be an exemplary evaluation firm?
The authors suggest that organizational flexibility within the evaluation firm to take on more and/or different work (often by hiring subject experts) was critical to building this exemplary evaluation practice. And there is definitely a culture of learning within Melanie’s firm. But what else is necessary?
Like a great red wine, SPEC Associates seems to have only gotten better with time, and I look forward to learning more from Melanie and her experiences.
Cheers my friend and a toast to many jobs done quite well!
19 Apr 2011
Yak Shaving in Evaluation
Having recently taken up knitting, I was disappointed at how hard it is to find Yak yarn. I decided to investigate and here is what I found: “A Hairy Yak is a Happy Yak“
Well, who knew?
—————————
“Shaving the yak” is a term that was coined by persons at MIT to refer to: “Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you’re working”. Apparently this term sprung from the story of a person who wanted a rug who ultimately shaved a Yak (and probably got the hell beat out of him). An example would be I decide to build a fence. I need a saw so I ask to borrow my neighbor’s. He won’t let me borrow his saw until I return his hammer. But I can’t return his hammer because I need it to build my fence. So I go to my other neighbor who says I can borrow her hammer if I return her the $20 she loaned me. So now I’m on the way to the bank. But since I’m almost out of gas I have to go to the gas station….but all of this is necessary to help me build my fence.
This sometimes happens in evaluation, for example, such as when developing a logic model. You have goal statements, activities, outputs and outcomes, but then as you start working with stakeholders you find that the activities and outcomes don’t quite link up, thus sometimes making stakeholders change one or the other, 3 hours pass and finally the goal statements are reworked to address the new outcomes and activities, but then the goal statements don’t support the overarching mission statement of the organization….
So while it may seem pointless, all of this back and forth, and listening, and communicating, I would argue that it is critical not only to the development of an accurate logic model but also to the potential that your evaluation results will be used. Even though hairy yaks are happy yaks, sometimes they need a good shave.
It’s very easy to want to go into an organization and tell them what needs to be done versus understanding their issues and working with them to help them find a solution. Some persons “do” evaluation from a desk whereas others exit that comfort zone to meet with stakeholders, meet the stakeholder’s stakeholders, and work on those issues that eventually solve the real problem of why you were asked to come over in the first place. Both people get thanked but only one person gets asked back.
—————————
Red Steagall’s “The Fence that Me and Shorty Built“
14 Apr 2011
Evaluation Use: The Trans-Challenge
This post is written in response to The BetterEvaluation Project which recently announced a Request for Challenges (RfC). Nominations for challenges are being sought to help BE showcase examples of good evaluation practice and to support research and development in areas where new methods need to be developed or trialed.
Much like Michael Scriven refers to evaluation as a trans-discipline, meaning evaluation crosses multiple disciplines (2003; 2005), in my opinion, facilitating the use of evaluation results is the trans-challenge or overarching challenge for all evaluations. When conducting an evaluation one can be ultra-sensitive to identifying stakeholders and values, articulating theories of change and critical methodologies, making overall judgments and recommendations, etc., yet if results are not utilizable or used, I question the value of the evaluation. This challenge holds true across evaluation type (education, environmental, etc.), methodologies used (cost-benefit analysis, mixed methods, etc.). or any other differentiating evaluation factor, thus making it a trans-challenge.
In my own evaluation practice I have tried to address non-use in as systematic a way as I know how. This involves utilizing a participatory / collaborative / and utilization-focused approach to evaluation where stakeholders are engaged at all levels, including helping to analyze data to varying degrees; making evaluation reports more user-friendly, following best practices in data visualization and reporting; and providing multiple “reports” in varying formats (full report, executive summary, PowerPoint deck, etc.) for multiple audiences. While these practices have helped me increase the overall use and degree of use of my evaluations by stakeholders, they in no way guarantee use. There may be no ‘magic bullet’, but we must at least continue to search for it.
References
Scriven, M. (2003). Evaluation in the new millennium: The transdisciplinary vision. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the millennium (pp. 19-42). Wahwah, NJ: Lawrence Erlbaum Associates.
Scriven, M. (2005). The transdisciplinary model of evaluation: Radical implications. Paper presented at the meeting of the American Evaluation Association/Canadian Evaluation Society, Toronto, Canada.
01 Apr 2011
Evaluation: Thinking Evaluatively
Recently there was a discussion on AEA’s listserv EvalTalk on what it means to think evaluatively. Multiple persons weighed in on this topic, but it surprised me how few persons seemed to understand the idea of using evidence versus using evidence to make an estimation of merit, worth , or value. In essence, it’s the same conundrum I’ve found in purported evaluation reports that detail “what is” versus going one step further and addressing “what is the value (of what is)”.
However, a few people did, as it seems to me, accurately identify what is meant by thinking evaluatively. One example I really liked was written by Eileen Stryker, who wrote, “It seems to me that thinking evaluatively is about how we arrive at or account for judgments about value or quality. Evaluative thinking means being tuned in to value judgments people make (e.g., listening for such language as: that’s good, he’s doing a good job, the program is working, they’re really getting better, and words like effective, quality, good, bad, better, improving, etc.), and questioning how those judgments were arrived at and what evidence may exist to substantiate the value claim.”
Eric Weir strongly agreed with Eileen’s definition and added: “[Evaluative thinking] is either
using evidence to support value judgments or assessing the extent to which value judgments are supported by evidence…..Evidence does not support or undermine judgments by itself. Arguments are needed to connect the evidence to the judgment.”
Last, Bob Williams summed it up (nicely) as “evaluative thinking is the informed judgment of value, merit or worth”.
This was an incredibly interesting discussion and made me wonder a few things:
Why do we, as evaluators, seem to so often feel insecure in evaluatively assessing outcomes?
What does this say about evaluation training and the need for more emphasis on training to think evaluatively?, and
Are there differences in how evaluatively persons think based on training, evaluation field, years of experience, etc?
Would love to know what others think about these questions. Feel free to comment!