OToPS/Messy Data/Impossible

A common problem is getting impossible values for a variable. For example, on an item that is scaled from 1 (Strongly Disagree) to 5 (Strongly Agree), a case might get a score of 0, or 11. The eleven is more likely to happen when a human is typing in responses, it could be a keypunching error. The zero is likely to happen when a respondent skips an item and the survey software assigns a value of zero.

A more sneaky version happens when we program scoring into Qualtrics (circa 2018). We often use internal scoring to give participants their score at the end of the survey (as a reinforcer for participating, and in assessment centers, as a core service). We refer to this as "piping" because it is possible to "pipe" the score to show on a later page of the survey. When a person skips an item, Qualtrics internally gives it a score of zero. This is a terrible way to handle the missing value, as the person almost definitely would have had a higher score had they answered the item. Qualtrics saves the scores to the data frame, and they also get exported with the other variables.

We do not want to use these for presentation or analysis, because of the quirk in how Qualtrics handles missing data.

Example:

In SPSS, I ran descriptives:

descriptives /variables iapAuthoritarian iapChildCentered iapAuthoritative apqPoorMonP apqPosParP

   apqInconDiscP apqDadInvolveP apqMomInvolveP apqCorpPunP 
 SC25 SC24 SC23 SC22 SC21 SC20.

It is missing data that are creating the differences between the Qualtrics and the syntax scoring. Two ways that I can tell:

The N for the versions computed by Qualtrics all are N=153. The syntax versions all have lower N, and the way that the syntax is written, it won’t calculate a score unless most of the items are available. Qualtrics is treating missing items as scores of zero.

Second tell is that the minimum scale score for all the variables from Qualtrics is zero, but the items all go 1 to 5 (or 9). Scale scores of zero are impossible in raw score format. If you run FREQUENCIES on SC20 to SC25, you’ll see a big pile of cases with scores of zero – those are all the people who skipped that part of the survey .

When we run correlations with SC20 to SC25, the zeros are getting treated as legitimate scores. This increases the N, and it trashes the correlation estimate.

Descriptive Statistics
N Minimum Maximum Mean Std. Deviation
IAP Authoritarian score POMP 127 .18 .81 .5736 .11675
iapChildCentered 127 .26 .92 .6450 .11340
IAP Authoritative score (excluding gender items) POMP 127 .25 .98 .7269 .13887
APQ Poor Monitoring - POMP 129 5.00 85.00 40.5039 14.56525
APQ Positive Parenting - POMP 129 .00 100.00 57.4289 22.04154
APQ Inconsistent Discipline - POMP 129 .00 83.33 32.6227 16.34723
APQ Paternal Involvement - POMP 125 5.00 100.00 43.2600 18.84311
APQ Maternal Involvement - POMP 127 5.00 100.00 53.8976 18.83685
APQ Corporal Punishment - POMP 129 .00 100.00 23.5142 20.21074
APQ - Corporal 153 .00 13.00 3.9935 2.72295
APQ - Inconsisitent 153 .00 26.00 11.7059 6.13111
APQ - Poor Monitoring 153 .00 44.00 22.1634 10.83213
APQ - Positive 153 .00 30.00 16.8235 8.53716
Qualtrics scoring of APQ Paternal Involvement 153 .00 50.00 22.5294 12.27538
Qualtrics scoring of APQ Maternal Involvement 153 .00 50.00 26.3856 13.43509
Valid N (listwise) 119