Evaluation Theories/Week 14: 4/23/14: Evaluation Theory Development and Research on Evaluation, Guest Speaker: John LaVelle

An Examination of Evaluation Education Programs and Evaluator Skills Across the World

Presentation by John LaVelle about his dissertation. The mile-high view of what it looks like to work as an evaluator.

I want to cross-reference the competencies and responsibilities for this practice; I hope someone will replicate the thesis so that they can help build on the knowledge.

Starting in the very beginning; I like to think back to Peter Rosse (sp?) – an evaluator, who said something about 20-29 years ago, that evaluation is an exciting, vibrant field; - I don’t think he meant to be ageist; but let’s refine: it’s an applied profession where people can do very immediate good work . It’s one of the ways we can improve the world around us.

Over hte last 70 years we've seen increase in practice, theory, and profession of evaluation. We have exponential growth in number of professional evaluation associations; both formal kind and informal kind. More and more people are joining these orgnaizations: responding to local community and societal need; AEA has over 7,000 members; in the 80s we had several hundred and several people filed in their foms twice☺

30,000 jobs posted that draw on evaluation principles, focus, and techniques: not necessarily “called” evaluation. You’ll be drawing on evaluation theories and processes in your work if you’re an org. consultant. (J: Are we drawing on evaluation theories, or are we independently working on our own: at what point can we distinguish between the two? Eval would have to be the best word on the subject in order for us to be drawing on them rather than developing in parallel)

Research on Evaluation; (Mark, 2007) –

1. Contexts;
2. Processes;
3. Outcomes;
4. Professional Issues

I focused on profession of evaluation: Attribute Model (Moore, 1979) – sociology looking at groups; interacting within society:

1. Occupation
2. Calling
3. Organization
4. Autonomy
5. Education
6. Service Orientation

Above are the ways people generally think about a profession; but this is a profession in a vacuum.

Instead; professions interact with larger society:

Processs Model

(Forsyth & Danisciwicz, 1986)

Three phases in the Model. (The PPT is too small to see / read) – It’s a feedback cycle predicated on two IV’s:

1. Job Market
2. ___

International Development Evaluation is a new field- when I think about the ways evaluators were trained in the 1970s and 1980s; they were doing best they could off the science we had. But if we’re based on their models. . .

Position selves as essential service;

1. position knowledge as exclusive. (J: HAVE WE ACTUALLY STUDIED TRAINED VS UNTRAINED EVALUATORS?!?!?!?!?)

Skills and Competencies

What do you need to know, feel, think, do, to be an evaluator?

1. Professional Practice
2. Inquiry
3. Situational Analysis
4. Project Management
5. Reflective Practice
6. Interpersonal Competence.

Above is an empirical statement on research of evaluation; they did replication (Stevahn) and found good relationships. Russ-Eft did it in Canadian context. (J: what’s the difference between Canadian vs. other contexts vs. US?) (J: what’s the argument for these? (Read King et al., 2001, I suppose, and thefollowing)

Competency Frameworks:

1. King et al., 2001
2. Stevahn et al., 2005
3. Russ-Eft et al.
4. Expanded into Roles & Responsibilities: Dewey et al., 2008

Education Programs

Survey based directories and studies:

1. Conner et al 1980
2. may et al 1986
3. Altschuld et al 1994
4. Engle et al. 2006

But are these surveys filled out on terrible grids that we have to replicate?

Online curriculuar document analysis and directory

1. LaVelle and Donaldson 2010

US Evaluation Education Programs: Heyday in 1980s: 67, ‘86: 44; ‘94 38; 2006 27; 2010 48;

US Evaluation Education by Academic Department:

1. Education
2. Ed. SPsychology
3. Psycohlogy
4. Policy
5. Other.

LaVelle's study

Current Study: 3 stage study that examined professional issues in evaluation

1. Skills used in practice of evaluation
2. Education of evaluators
3. Integration of skills and education

Research uestiosns:

1. What are relevant knowledge nad skills for evaluation practice

Method: job Description Analysis:

1. Analyzed 70 job descriptions (AEA, EES, CES) – 35 US; 35 International
2. Used ATLAS Ti
3. Coding for:
   1. Skills
   2. Responsibility
   3. Coded based on overt characteristics

Now, the thing about ATLAS TI; you have to create codes, play with information.

In the US I found:

1. Education
2. Health
3. Humanitarian Efforts

Desired Qualifications and Background:

1. Baccalaureate; to do evaluation work; wanted people with experience in health, education, social sciences.
2. As you go through this class, you’re developing real-world skills.

Evaluation Skills Map?- Definition of Evaluation:

1. “Looking at the merit worth and significance of a program; and intended use for intended users.”
2. Trying to apply research methods to answer those questions.

I wish the program had done this for me, I found seven main themes:

1. Inquiry (accurate)
2. Technical assistance with designing evaluations.
3. Programming and Intervention Skills
4. Management Skills – Projects; People
5. Public Interface – People able to break down reports for those who have technical background and those who do not.
6. Capacity-Building
7. Evaluation Theory

(J: These are all program evaluation. I am so sick of Program Evaluation, because I think it’s built on a house of sand. . . )

Dewey et al., (2008) and current study: I didn’t have to ad a lot in terms of competencies:

(3 newskills; 2 new responsibilities)

Using Advanced Google Boolean Search, I entered these codes: Online Curricular Document Analysis – These things on the same page as the term “Program Evaluation” - - once I saw response, I chatted with advisor; and asked, “When can we say that I’ve hit data saturation?” – 100

First rule of online research; print it out as soon as you find the page. I looked at Webpages, Student Handbook; Bulletin; case in point; SSSSPE; - You print it off; code it in ATLAS, and then put it into SPSS for Quantitative analysis.

I coded for topic; what makes essential curriculum? – What are the fundamental things that people are going to learn during their time at each of these universities?

1. Geography
2. Academic Home
3. Curricular Topics & courase titles
   1. Evaluation
   2. Quant methods; qual methods; assessment; linear modeling;

Based off of that;

89 Colleges & universities in the US; 37 certs, 58 MA’s, 4 specialist degrees, 40 doctoral;

Department: Ed 52%; ed psych 18; …

U-Shaped Data from 1980to 2014

Now; (4 lines) – right around 2006; you see “severe” uptick in MA and Certificate programs.

In terms of what the curriculum looks like; quantitative design; statistics; but in PhD programs; Average ranked Phd Credit hours;

Evaluation was the big dog in the room; followed by quantitative design; assessment; but in Public Policy it was “Other forms of inquiry” – They’re being trained to do a different flavor of evaluation than you guys are.

(J: CATEGORIES of evaluation: BIG picture! We still need it!)

In psychology we don’t have a qualitative requirement; it really will help round you out and make you more marketable.

Crosswalk of Evaluation Skills and Evaluation Education programs.

Alignment: (Between Education & Job Market):

- conceptualization; inquiry quantitative; instrument development; data  collection fieldwork analsysis; 

- Conditional Alignment: Eval theory; qualitative inquirary, qualitative analysis NO Apparent alignment between job market: management skill; teaching; capacity-building

I think we will get to a common core in Evaluation; for education; how do we differentiate ourselves? At CGU we’re one of the biggest, oldest, strongest; we also need to think of meeting needs of non-traditional learners; adult learners; working professionals. I’m thinking of Africa; Israel; areas where evaluation and social programming is happening; but are those evaluators in those areas prepared to do good work the way we have?

Practitioners: with a good common core skillset; competencies; mastery of roles; you can open many different doors. (J: Why isn’t there more on applied logic in Eval courses? We should develop our own logic, stripped of mathematical quibbles, down to only what is absolutely necessary for the type of evaluation that Evaluation does. “

‘’’Strengths’’’ Drew from publically available and vetted documents; disentangled degree tracks; drew from transdisciplinary approach.

‘’’Limitations’’’: Possible under-representation of International jobs; Boolean search done in English; course title-level analysis. Document analysis was the only method used.

What would happen if booleans were re-done in french; spanish; etc.? If we were to replicate and expand on this we would find a more robust picture.

Course title level analysis: Evaluation & Research methods: what are they learning? it's hard to tell. (J: need to go to course curriculum papers)

- I would go and talk to program deans; etc.

Q: I heard people say, we need people who know ___. . .

Examples of ROE

Eval theory is a little child trying to grow up – and we’re trying to support it. We need to do future work on making this connection between theory and practice; developing our theories so that they are more useful in practice.

Yes, things like “formative evaluation” (J: No; formative is an end;

J: My Quotation from Shadish

"it is just as important to understand the problems and limitations of evaluation theory as it is to understand its accomplishments, for those problems ... (define) what we lack in our presentations to each other and to the world.

The first two of these problems concern important theoretical matters that no theorist has addressed particularly well:

The general failure of most theorists to develop contingent theories of evaluation practice that tell evaluators how to make choices based on the contingencies of the sit-uation. The general omission of a consensual vision of the place of evaluation in the world today.

The second two problems concern what we might call evaluation metatheory:

The lack of a widely-accepted metatheoretical nomenclature that would help us to classify any given theory about evaluation, and to use that classification to understand what a particular theory does and does not claim to do. The neglect of a comparative theory of evaluation, one that uses the common meta- theoretical nomenclature to compare and contrast the relative strengths and weaknesses of individual theories.” (Emphasis by Joshua Penman)

(Shadish, 1999, 'Evaluation is Who We Are', pp. 7-8;)

J: The current state of evaluation is more based on anecdotal evidence (discourse and experience) rather than empirical evidence or logical progression from first principles;

• however, empirical evidence is simply anecdotal evidence codified and organized appropriately, and with the possible addition of tested experimentally
• (J: I don’t see enough emphasis on first principles – and I think these are needed in order to jumpstart a unified evaluation theory on which the ones we have so far are all mere branches of (in the sense of branching possibilities, not Alkin’s “Theory Tree”☺)

Well put – and I think most people would agree. (J: awesome. . . .I just wrote that in class…why won’t you let me write more?? (by let , of course, I mean “assign”☺)

“We have “expert-based or evaluation model-advocate based evaluation practice rather than an evidence-based practice.”

J: Can we evaluate whether what we have learned IS a theory?!?!!?!?! Where is the rubric for what makes something a theory?!!!!

• Most of what we’ve read were ideas about intended use!!!
• For example, to jumpstart debate, I might say that “Empowerment Evaluation” is not a theory: is a method aimed as specific ends – and those ends have not been well communicated.

How to bridge the gap between theory and practice

Shouldn’t we evaluate our approaches, tools, ensure they’re effective?

(J: Even before that we can evaluate whether they make sense! We can categorize them! Then we know what we need to test)

Agreement regarding wtha tis “evaluation” - ____ Understanding of level at which evaluation theory can be tested. (Smith, 2010) Taxonomies for empirical criterion for examining Evaluation theory and types of questions (Miller, 2010) Research (Mark, 2008)

What we mean by theory

• Smith, 2010: three Kinds of Evaluation Theory
• # Theory of how to practice
• # Coherent resolution of theoretical issues that provide a compatible set of prescriptions for how to conduct an evaluation
• # Evaluation approach: Broader conceptual collection, representing groupings of models sharing similar principles.

Josh’s Evaluation Theory Checklist(J: These just used the term itself in the definition) …J: What is a theory

• “a supposition or a system of ideas intended to explain something, esp. one based on general principles independent of the thing to be explained: Darwin's theory of evolution.” (New Oxford American Dictionary).
• From this definition we can draw three criteria for whether something is a theory at the most basic level – i.e., linguistically.
• ___ 1b) Is it a system of ideas? <- (upon evaluation, our usage is of “system of ideas”)
• 1.b.1 System: (New Oxford American Dictionary)
• 1) a set of connected things or parts forming a complex whole,
• a set of things working together as parts of a mechanism or an interconnecting network:
• Specific uses:
• Physiology: …
• Computing: ...
• Geology
• Astronomy
• 2) a set of principles or procedures according to which something is done; an organized scheme or method:
• • orderliness; method:
• • a set of rules used in measurement or classification:
• 1.b.2 Idea: (New Oxford American Dictionary
• Idea: New Oxford American Dictionary
• 1 a thought or suggestion as to a possible course of action: they don't think it's a very good idea.
• • a concept or mental impression: our menu list will give you some idea of how interesting a low-fat diet can be.
• • an opinion or belief: nineteenth-century ideas about drinking.
• 2 (the idea) the aim or purpose: I took a job with the idea of getting some money together.
• 3 Philosophy (in Platonic thought) an eternally existing pattern of which individual things in any class are imperfect copies.
• (in Kantian thought) a concept of pure reason, not empirically based in experience.
• X 1a. Supposition: an uncertain belief:
• ___ 1b.1) Is it a set of things working together as parts of a mechanism or an interconnecting network? (System)
• ___ 1b2) Is it a set of principles or procedures according to which something is done; (System)
• ___ 1b3) Is it an organized scheme or method?
• ___ 1b3) Is it a thought or suggestion as to a possible course of action? (Idea)

___ 2) Does It Explain Something?

• ___ Does it reveal relevant facts or details? (Explain)
• Relevant: “closely connected or appropriate to the matter at hand”
• Appropriate: “suitable or proper in the circumstances:”
• Suitable: “right or appropriate for a particular person, purpose, or situation:”
• Proper:
• ( New Oxford American Dictionary):
• [ attrib. ] truly what something is said or regarded to be; genuine: she's never had a proper job | a proper meal.
• • [ postpositive ] strictly so called; in its true form: some of the dos and don'ts in espionage proper.
• • informal, chiefly Brit.used as an intensifier, often in derogatory contexts: she looked like a proper harlot.
• 2 [ attrib. ] of the required type; suitable or appropriate: an artist needs the proper tools | they had not followed the proper procedures.
• ___ Does it give a reason as excuse or justification? (Explain)
• ___ is it a “Reason”?
• ___ Does it give a “Justification”?
• ___ Does it Identify something as the cause or motivating factor (for something else)? (Explain)
• ___
• _√_ 2.a (Explain; New Oxford American Dictionary)
• make (an idea, situation, or problem) clear to someone by describing it in more detail or revealing relevant facts or ideas: [ with clause ] : they explained that their lives centered on the religious rituals | [ with direct speech ] : “my daddy has spells,” Ben explained | [ with obj. ] : he explained the situation.
• • account for (an action or event) by giving a reason as excuse or justification: [ with obj. ] : Callie found it necessary to explain her blackened eye | [ with clause ] : he makes athletes explain why they made a mistake | [ no obj. ] : she had tried to explain about Adam, hadn't she?
• • [ with obj. ] be the cause of or motivating factor for: her father's violence explains her pacificism | [ with clause ] : this would explain why so many adult children still live with their parents.
• X 2.b Something
• New Oxford American Dictionary
• something |ˈsəmˌTHiNG|
• pronoun
• 1 a thing that is unspecified or unknown: we stopped for something to eat | I knew something terrible had happened | something about her frightened me.
• 2 used in various expressions indicating that a description or amount being stated is not exact: a wry look, something between amusement and regret | grassland totaling something over three hundred acres | there were something like fifty applications.
• ___ 3) (esp.) based on general principles independent of the thing to be explained
• __ 3.1 Is it based on “general principles”?
• General:
• _ considering / including the main features or elements
• _ true for most cases
• 1 affecting or concerning all or most people, places, or things; widespread: books of general interest.
• • not specialized or limited in range of subject, application, activity, etc.: brush up on your general knowledge.
• • (of a rule, principle, etc.) true for all or most cases.
• • normal or usual: it is not general practice to confirm or deny such reports.
• 2 considering or including the main features or elements of something, and disregarding exceptions; overall: they fired in the general direction of the enemy | a general introduction to the subject.
• Principle
• ___ 3.2 Is it independent of thing to be explained?
• J: A set of things working together as parts of an interconnecting network, which constitute an organized which principles or procedures are derived )

Smith’s Framework:

• Smith, 2010: Looking at
• # Evaluation Model as Ideology
• # Evaluation Model as Operational Strategy
• # Evaluation Model As Intervention
• # Evaluation Model as Ideology
• # Evaluation Model as Operational Strategy
• Can you start to think about the general categories of criteria that you would apply.
• Can it be adequately implemented and produce expected outcomes?
• # Evaluation Model As Intervention
• Criteria for a ___? -
• 1. Operational specificity
• 2. Range of Applicaton
• 3. Feasibility in Practice
• 4. Discernible Impact
• 5. Reproducibility

Mark’s Framework

• Inquiry Modes:
• # Description
• # Classification
• # Causal Analysis
• # Values Inquiry

‘’’Theory-Driven Evaluation’’’ <- J;

• Why do people like it?
• # Structured way to involve stakeholders
• Combined what they know well with what evaluators know well
• Combine everyo’ne expertise into a nice package.
• # Integrates Social Sicence Theory
• # Geeater external validity + transferability
• # Method neutral.
• However; this is kind of BS; “have you ever seen a good “theory driven evaluation” that used good qualitative methods?

How can ROE Help?

• Method for using qualitative data to validate program data:
• What methods have been used in other areas?
• Pattern Matching
• Crowdsourcing
• Conceptualize how it would work
• Test it.
• A program theory is already an “Expected Pattern” – you can see whether pattern of collected data matters pattern of your program theory.
• We used a crowdsourcing method; a way to use Mturk (a crowdsourcing platform) to get a bunch of random people; with the idea that if you average 1000 people to do it they’ll give you a better idea than a fully-trained graduate student.

Example: “College Success” Program:

• Easing transition to college; I’m sure there are scales for these things; but need to think about this more holistically.
• Method:
• Transcript from three student interviews:
• Whichaof the following are discussed?
• Sense of community and belonging
• Confidence in academic ability
• academic support
• skill development
• transistion to college
• Asked them to highlight sentence in which it was discussed what was discussed;
• A ton of people would select it on list; sentence they highlighted would be about the program.
• (J: Can I use this to test my own categorizations of things?! This is exactly what I want!!!!)
• If you just did a quantitative method and just measured “sense of community” – you wouldn’t get that link. (J: This is EXACTLY what I’ve been looking for! This should be used here! Our homework should be in this format! How did this work?!?!!?! (Wouldn’t you need a two-screen section; left size moves, right side stays, and you can select the question you’re answering then highlight? (Different colors for different highlights) 55*55
• System you used, how much time would it take to set it up?
• Survey tool was built in Qualtrics
• - Had to define areas because there is no built-in PDF annotation system in QUaltrics

Research on Youth-Program Evaluation

Content analysis of 97 published evalutaions of programs serving youth populations.

• Used specific search terms; entered them in Google Scholar & Every other database I could find; Applied inclusion criteria. . .

Activity: Designing ROE Studies

10 Minutes; One more content class, next week: Credible Evidence.