Evaluation Domains/Week 5

what sort of brain activity goes on when one does evaluation (vs. preferring and liking)?

Godchildren:

1. Ben: Evaluation of Human Trafficking:
   1. Josh: this seems to be rooted in the “Practice” section of evaluation: how do you get the data when it’s guarded by 1) Threats of violence, 2) secrecy (because of its illegality), 3) Social norms (in as much as it is “distasteful”, etc. . .
   2. Scriven: I think the Police/Prostitution Situation is like the Police/Gambling situation
2. Josh: I wanted to develop a list of all the Professions that fall under the domain of “Evaluation”, but now:
   1. I just want to learn about how to make categories: how do I “structure data”?
   2. What are the building blocks of “good categories”
   3. What is the notation for “good categories”?

Example 1: Evaluation In Education

What was wrong with “Evaluation in Education: How to Test Kids” - About 7/8ths of the topic was not involved there.

What’s a possible fallacy involved in trying close the “gap” between two different groups of children?

1. Looking at the culture; (Donna Mertons is doing this)
   1. The Solution John Haddie (sp?) thought up: and implemented; and had enormous success in closing the gap: His solution was: “We don’t have to insist that they change their fundamental values: what about using some of the role model people that they like to inspire their interest. Lets find ourselves the great Black scientist, or the great Maori eco-system analyst. (J: This is “Motivation”)
   2. J: To generalize that would you have to find out what forms much of the person’s Identity? Scriven: The answer is, you do whatever it takes. Start with the ones that are going to give you the least trouble; and hopefully a big enough motivational boost. You don’t have to answer the general question before you begin.
2. They did follow that up with trying to crack the boy-vs-girl difference in science. The last time the national test was run in Australia, the girls beat the boys in everything – That’s a national change; complete change.
3. This approach seems to work better than Donna Merton’s theoretical approach:
   1. Understand Culture <- Okay
   2. Respect Culture <- Moderately okay, but
   3. Enforce Respect <- Practical: but that’s not the point; the damage done by selective respect is the trouble, and that’s what had the outcome problems that really bothered us all. What you have to do is employ the cultural iconography to increase their respect for these “unattractive” fields; unattractive because they never respected the people in them that did well.
4. It was an ethical imperative: but how can we make that into a pedagogical imperative?

If you didn’t give women the incentive it takes to get them to take classes in school, out of “respect” for their iconography, and that leads to less women being prepared for medical school. . .

Working at a way that Maori kids can be respected for their academic achievement: it can be done.

What my colleague did was say, “are the teachers giving the same amount of time?” - Everyone said, “Of course they are: that’s the first thing we thought of, and we did study after study ‘till we got to the point where they were.” - he said, “Show me the best school we have on this, with supportive teachers.” -

1. He started using a stopwatch; his best-trained PhD students doing the fieldwork, and it turned out they weren’t:
   1. It looked like they were, to the average qualitative observer,
   2. but when you went quantitative, it showed that a gap ‘’was’’ there.

I call this the “Dosage Problem.

Question:

Values -> Communication of Values -> Reception of Communication -> Response to

Godchild: Evaluation of Public Image

In the last 2 weeks 6 members of congress (CA state congress, I think) were … impact of PR on platform and honesty. . .

Attra

Godchild: What makes a good categorization system?

1. Josh: I wanted to develop a list of all the Professions that fall under the domain of “Evaluation”, but now:
   1. I just want to learn about how to make categories: how do I “structure data”?
   2. What are the building blocks of “good categories”
   3. What is the notation for “good categories”?

What’s a good categorization system, and what’s the notation for a good evaluation system.

Scriven: I’ve been working on something like that, suppose I want to say, “P is better than Q”

‘’’Comparative Eval’’’ You have “P>Q” for P is Greater than Q - and have symbols for merit ≈ (with a dot), etc.

1. Better than = .>
2. Of approximately equal merit? .≈?

‘’’

1. Entails: => <=
2. Deduce
3. Induce: %->
4. Probabilistically Implies: <-%
   1. Beyond reasonable doubt.
      1. This seems to be an evaluative operation where you decide what your values are and then
      2. At some point I get to non-evaluative (J; but it always starts on the evaluative. . .)
      3. J: What about in social sciences, p[[α]<.05
      4. Can your premises get supported by direct perception? - If you can directly perceive that such and such is impossible; that’s evaluative; but also something we can directly perceive; and hence; factual
      5. J: What constitutes a “Fact”?
         1. Ease of obtaining that information (“Accessibility”)
         2. Reproducibility of that information? (“Reproducibility”)
         3. “Indirect Confirmability” of information (Scriven) -
         4. Shared consensus around that information (“
            1. Prestige of people with consensus (“Prestige”)
            2. Professionalism of people (in as much as separate from prestige) (“
            3. Number of people with consensus (“N
            4. Repetition of consensus (“
            5. Face-Value of the information (“Face-Value”)
               1. Previous information
               2. Perceived Reproducibility of that information
            6. Logical force of consensus (“
      6. “Evaluative” in full strength of “subject to
         1. You can get to things in judgement of impossibility - uncontestable judgement of impossibility. ; that in my terms is a non-evaluative fact.
5. Probity Implies: ≈>

Scriven:

1. A fact
   1. “Beyond Reasonable Doubt”
      1. Reasonable doubt: In minds of english speakers with
      2. More information

1. Index;
2. Construct (Psychology);
3. (J: How many ways can a thing belong?)
4. (What groups do things belong to?)
5. Why do we classify?
6. You classify based on gestault (“how things appear”)
   1. J: So; basicaly; you classify based on any kind of difference you can perceive;
   2. J: you classify based on any kind of difference that matters to your? (User? Stakeholder? Beneficiary?)

Midterm Exam:

The bad news is: my computer’s not working; I don’t have copies of it! It will have to go on the board.

Q1 (of 3): “You are teaching an intro to evaluation course, and you want to explain some recent and emerging concepts of evaluation. Write a page (or two or three) that introduces and distinguishes the following concepts, in temporal order of their emergence.”

1. Evaluation as applied Social Science
2. The Omega Role of Evaluation
3. The Transdisciplinary Role of Evaluation
4. The Role of Evaluation as an Alpha Discipline
5. The Exemplary Role of Evaluation
6. The Non-Evaluative Way of Doing Evaluation

Answers:

Q1

First, #6: In the bad old days people would take an evaluative contract, without talking at all about values. They would convert it into a quantitative problem, using an indefensible conversion.

(J: This would be evaluation with unstated values: they are still present, they are simply unstated)

The only official survival of that period is the (Describes a specific professor) believes that you have no place commenting on whether the intervention met needs; would normally be thought to be very impressive.

Under the old regime it could be said whether something met conventional standards

1. 1; the definition of evaluation in the most widely-selling book used as a textbook; defining evaluation they say it is

1. 1. “an application of social science methodology to answering questions about changes that have or have not occurred.” -
   2. And that of course, is unbelievably self serving; since the actual definition of “evaluation” is
2. Evaluation: “What survived the attack of social sciences”

1. 3 Is

1. Non-Evaluative Way of Doing Evaluation
2. Evaluation as applied Social Science
3. Transdisciplinary Role of Evaluation
4. Alpha Discipline of Evaluation
5. Exemplary Role of Evaluation
6. Omega Role of Evaluation

Omega Role of evaluation: The role of providing foundations for ethics.

(J: Take for example, a model on the issue of abortion: values:

1. Values:
   1. Quality of life
   2. Life (Living / Not Living)
   3. Economic Cost
   4. Social Cost

Answer to Q2

What’s wrong with calling non-evaluative claims “facts”?

It’s not the case that false-statements are facts. My definition of “facts” is :

1. ”Statements whose truth is provable”
2. 1+1=2 is provable from mathematical foundations.

Answer to Q3

1. Tom Cook: Don Campbell’s successor in the RCT camp.
2. He’s full-professor in 4 departments

Billiards; a game with only three balls:

1. White Ball
2. Ball with a dot on it
3. Red ball

The game centers around the use of these balls on six pockets: Now, I’m watching these; the dot means it’s the cue ball; the one you strike:

This ball hits another ball, which goes to here, and then bounces over and hits that, which then goes in here, if he’s very ___. And the cue ball must not go in, because if it does, he loses the game.

I see that happen; and I have the causal analysis that the cue ball hits that, then hits this, then . . . - that’s the causal analysis of this sequence of events: it is entirely perceptual; it is not inferential.

Final Assignment: reflect on this example of a non-inferential causal analysis.

ToDO: - Michael’s Flight is going to be late: -