Evaluation Theories/Week 5: Evaluation Theory Panel: Patton, Cronbach, Chen, Guba & Lincoln
Q&A & Feedback edit
Today; while T1 hearts Cronbach, I heart Huey T. Chen;
Sophisticated Saying: “If it ain’t broke, don’t fix it.” - With Moderating; T1 did it so well that we’ll keep her moderating.
In Week 7: TODO: Turn in a 2-page Reflection paper.
You will have gone through the 12 Theorists; and we want this paper to be an opportunity to take you across theorists.
- 2. 1. Top three salient issues
- 3. Brief: To pages; double spaced; Times New Roman; 1” Margins - For Week 7.
Come prepared with a question (J: ???)
- Intended Use by Intended Users: Not for general stakeholders; just the primary stakeholders.
Utility Feasibility Propriety and Accuracy;
Sarah: Used to be an evangelical preacher; said talking to people about whether their life was consistent with their values.
13:17:11 Lee Joseph Cronbach
Evaluation as a ____ for education; adress particular needs; create a forum where everyone can interact; educate; influence social policy.
PhD Stats. Qual Research Special Interest Group Eval Center Western Mich. Teas A&M
(Awards are meaningless unless they’ve got brand recognition…aka: Nobel or bust)
Construct knowledge for the specific context;
Huey Chen edit
- Born & Raised in Taiwan; U Alabama Birmingham Public Health
- (WHY THE NAME OF AWARDS?!!?! :))
- National Evaluation System for Evaluating a CDC-Funded HIV Education Program (Josh: WOOT! lol, I should go evaluate the Sudan one I produced!)
Pseudo-Patton: How do you think your academic training / background influenced how you think?
AS PSEUDO-PATTON, I THINK THE ORGANIZATIONAL PART OF MY DEGREE WAS IMPORTANT; - I KEEP UP WITH MY TEACHING BACKGROUNDWITH THE INTERNATIONAL DEVELOPMENT EVALUATION TRAINING; I DO A LOT OF TRAINING
CDC: TAXONomies for how to develop a useful evaluation. (J: I want ot make the taxonomies of EVALUATION - the use I can’ 13:30:52ake a big impact in; too many brilliant people are already working in it. The bigger picture is not together though. I love being in that position, because i’m usually
_____ completely reshaped his ideas of what evaluation is.
…. Unlike BLakc-box Evaluation;
Valueing: Basic approach: construct knowledge that is context bound.
- Positivism; one basic reality; understood through generally quantitative means (J: EVERY INDIVIDUAL CONSTRUCTS THEIR REALITY THROUGH QUANTITATIVE MEANS. That’s why if you repeat something often enough people start to believe it.)
(J: What is good about this system: it’s good for looking at who is well developed in what area. . .but mutually exclusive and collectively exhaustive (MECE) (Ethan Rasiel, 1999, 'The McKinsey Way', p. 6; and which I just ran across in MIT’s introduction to Probability course…)
13:54:29 - ____ Most well developed is SOcial Programming;
Mentor: - Cronbach a bit abstract often
14:19:09 - Cronbach UTOS:
– . . .
External & Internal Validity: Campbell & Cronbach:
- Internal Validity: Generalizing from sample to population in same context: - External validity
14:26:08 - Guber ;
Stakeholder APproach; Social Sciences Approach; ____ Approach.
Former Theory &
* Stakeholder THeory: - Values of responsiveness to Stakeholders views; * Social sciences: Program theory; deriving based on how “program is actually operated and existing social science theory knowledge.” - (J: HOw good is SS Theory knowledge?)
Balance between Stakeholder and Scientific Credibility.
- Type 1: False Positive
- Type 2: You think there isn’t an effect when there really is.
- Type 3:
- - the right answer to the wrong question. This is sometimes called a Type 0 error
- - Type III error occurs when you correctly conclude that the two groups are statistically different, but you are wrong about the direction of the difference. (http://graphpad.com/support/faqid/1080/)
- - Chen: Theor
y was good; implementation was flawed. (Tiffany Berry P315Z; 2014-02-19)
Hermeneutic exchange edit
- 8-Step Process:
- 1. You don’t know everything
- 2. Look at Stakeholders separately (using Maximum variation sampling)
Q:What information do you gather from these stakeholders? -
Do you educate stakeholders to ensure they can keep on doing things after you leave?
- Not education. . . we on’t want to influence. . . more enlightenment to other stakeholder’s point of view. (J: better word than enlightenment?
- What makes this different from “Including Everyone else’s point of view?
Cronbach - Policy-Making; Info for decision-makin:
I’d advise: 1. Multiple perspectives 2. Embrace people usually excluded (J: Rawls) but not while silencing those who are already in control 3. Explicate dynamics of power and privilege interculturalhe program. 4. ___ context; not idly serve ____
Multiple Stakeholder Groups: You’ve Identified some: Most important groups of stakeholders that you would involve: 1. Public Servants (those who make the program for the public) (J: Our favourite public servants: Google. Re: Google Books Project. Until you @#$@# them and they became evil when they realized their idealism wouldn’t fly in the real world) 2. Members of the Program (who it’s made for)
Mentioned Leverage: Previous Ambiguity; Adjustment at the Margins?
INTEGRATIVE VALIDITY MODEL edit
three components. Campbelian Typology of Validity
Superior to Campbell because Campbell’s approach Lab -> Real world; mine goes: “Can it work in Real world?” - then “Actually works in real world?” then ____
Transferrable validity: Revision of Campbell that takes into account the stakeholder’s perspective (Lisa ____, 2014-02-19)
- very practice oriented; more expansive idea of validity than the traditional typology.
Viable Validity: Should we even bother; do we have the resources; buy-in from stakeholders? Is it cost effective?
- Traditionally this is last step; with Chen this is First Step.
Typologies: External Validity; Focus is on how to rule out threats to validity. Becomes less and less relevant to stakeholders.
Viable Vlaidity: a world-class center doesn’t matter at all if you put it on a hill and there’s no escalator up: IT doesn’t matter at all if we can’t get to it; can’t access it.
Really good thing about Halcolm from ___ (girl in the pink jacket)
Student Generated Questions: edit
Gets continued throughout the process: at every stage you’re doing this exchange; ensuring that entire population is informed; contributing; feeding back information through entire profcess. I don’t think you write as clearly about the process of evaluation as ____. . . (Guba & Lincoln)
Constant implementation failure but people persist in pursuing the overarching causal or program theory; what advice would you give in that situation? -
A: We deal with that all the time. Implmeentation is always flawed: it’s just levels of flawedness. Sometimes its so severe. . . one project wer’e advising now, Dosage is a big part of the implmentation; but once they looked at the [dosage] - they were serving these kids 2 hours per year, and were supposed to improve their resilience and grit; . .. ____ - (Great personal story)
Implementation is always flawed: that’s why it’s absolutely critical to measure it, always.