Helping Give Away Psychological Science/Standard Operating Procedures/ Qualtrics Assessment Guide

Click Here for Landing Page

HGAPS New for Fall 2022: HGAPS and Psychology Conferences

Click Here for Landing Page

HGAPS is finding new ways to make psychological science conferences more accessible!

Here are examples from APA 2022 and the JCCAP Future Directions Forum. Coming soon... ABCT!
~ More at HGAPS.org ~

---

Qualtrics guide to making assessments

All questionnaires in HGAPS’ Assessment Center are presented using Qualtrics, an online program for making and administering surveys/questionnaires/assessments. All UNC students and employees have free access to Qualtrics. This SOP reviews research and evidence-based assessment related to Qualtrics' information and policies. For a general introduction to Qualtrics, see the quick guide put out by the company; for a thorough discussion of the Qualtrics software, or as a reference, see the official training handbook.

Writing Qualtrics Questions

Question Types

Qualtrics offers a fair amount of flexibility in question format. The type of question format used for a certain question can be chosen by selecting the question and then selecting the green dropdown under “Change Question Type,” found in the right-hand menu. Text in quotes here is taken from the Qualtrics question type overview.

• Static Content: Because all elements in a Qualtrics survey block are designated as “Questions,” if you want to put descriptive text or graphics into the survey, you create a new question and select the type as static content.
  • Descriptive Text: “used to add an introduction or instructions to your survey, or wherever you need to display content to the respondent, without asking for feedback”
    • example: survey instructions
  • Graphic: “used to display an image to your respondents without asking for feedback”
    • example: displaying stimuli
  • File: used to display a link to a certain file type; accessed through setting a “Static Content” question to type “File”
    • not commonly used
• Standard Questions: In almost all cases, your survey can be built using primarily these basic question types.
  • Multiple Choice: “can be displayed as a traditional list of choices or as a dropdown menu, select box, etc.”
    • example: race or ethnicity demographics
  • Matrix Table: “used to collect multiple pieces of information in one question”
    • example: a survey that uses same Likert-type scale to rate all statements
  • Text Entry: “used to gather open-ended feedback from respondents”
    • example: asking for participant feedback on the user-experience of the questionnaire
  • Slider: used to “let respondents indicate their level of preference with a draggable bar rather than a traditional button or checkbox”
    • example: participant rates agreement with the statement on a continuous scale; like the level of perceived pain
  • Rank Order: “used to determine each respondent’s order of preference for a list of items”
    • not commonly used
  • Side by Side: used to “ask multiple questions in one condensed table”
    • typically, not used; can appear too compact/dense
• Specialty Questions: These questions are used in special circumstances when advanced functionality is needed. Your best resource for this is the Qualtrics documentation. Here is the link to the “Question Types Overview”; from there you can navigate to information on individual “Specialty” questions.
• Advanced Questions: These questions collect metadata; most of them are invisible to the participant. See official Qualtrics documentation for more information. You can use the “Question Types Overview” link provided above to navigate to the documentation on the “Advanced” question types.

Question Options

In addition to the “Question Type,” each question has several other features that can be customized. These are available underneath the “Question Type” dropdown in the menu on the right side of the page. Below are general options available for nearly all types; you can find information about the special options available only for specific types in the appendix entry on Qualtrics Options.

• Choices
  • Number: the total number of choices available
  • Edit Multiple: brings up a window with each choice’s text on one line; useful for easier editing of many options at once
  • Automatic Choices: check this box to gain access to a library of premade responses, like those describing satisfaction, agreement, etc.; typically better to simply write in your responses to ensure they match the survey you are using
• Validation Options: validation features to require or encourage participants to answer survey questions
  • Force Response: requires the participant to respond before continuing to the next page; useful for mandatory questions
    • example: obtaining informed consent
  • Request Response: prompts the participant to respond before proceeding to the next page; presents as “There are unanswered questions on this page. Would you like to continue?” pop up
    • example: general questionnaire items (by ethical standards participants can choose to decline to answer)
  • Unchecked: a participant may choose to leave this question unanswered
    • rarely preferable to “Forced Response” or “Request Response”
• Validation Type:
  • None: any combination of question responses is considered valid
  • Custom Validation: opens the editor for determining what counts as a valid response, by determining what combination of question responses constitutes a valid response

Response Coding

Many questionnaires we use are provided to us along with information on how to score participant responses. For questionnaires that use scale ratings for participants, the values of each choice/option along the scale may vary. (For example, a scale including “Strongly Disagree” to “Strongly Agree” might contain values indexed from 1 to 5, or from 0 to 4.) Our choice values in Qualtrics must match those provided by the questionnaire documentation. Otherwise, printed out results/scoring points will be off and could be the difference between the participant’s score falling in a clinical range or not leading the clinician to making clinical decisions based on faulty scores.

To avoid this problem, make sure values are coded correctly before you publish the survey. To change choice-value coding:

• Right-click on the question. It will become highlighted in light blue.
• Select “Recode Values…”
• Check the “Recode Values” box. Yellow fields, numbered by default from 1 upwards, will appear beside the question choices.
• Input the correct values. Click in each yellow box, delete the default value, and insert the correct value (according to questionnaire scoring standards).
• Close the pane. Click “Close” in the bottom right of the editing pane. Your changes will save automatically.

Page Breaks

Questions are all displayed on one page unless page breaks are added. To add a page break:

• Select the question you want to place the page break after.
• Click “Add Page Break.” You can find this option in the “Actions” submenu at the bottom of the right-hand menu.

You will want to have each questionnaire appear on a separate page. Add more page breaks as necessary to enhance clarity.

Conditional Questions

It only makes sense to ask certain questions based on the answer provided to a previous question. For example, if a participant answers “Yes” to “Have you experienced racial discrimination?” we would show them a series of related questions. If a participant answers “No” to that question, it is illogical to show them the related questions. You can use Display Logic to determine under what circumstances a question is shown:

1. Make sure the question you wish to apply Display Logic is on a separate page from the question you wish to evaluate.
   1. If they are currently on the same page, insert a page break between them or move the Display Logic question to a different page.
   2. If they are placed on the same page, Display Logic will not work correctly. The question the Display Logic is applied to will simply never appear.
2. Right-click on the question. It will become highlighted in light blue.
3. Select “Add Display Logic…”
4. Fill out the condition that must be met to display the question. The first field refers to the type of information that will be checked (“Question” answer, “GeoIP Location,” etc.), and the second field varies according to the option selected in the first field.
   1. When you select “Question,” the second field will be the question you wish to evaluate. The third field will be the answer choice you wish to evaluate, and the fourth field will be chosen from “Is Selected,” “Is Not Selected,” “Is Displayed,” or “Is Not Displayed.” (The last two options are used much less often than the first two.)
5. Add additional conditions if necessary. Press the green “+” to add a new logical statement. This one will start with the choice of “And” or “Or.” Selecting “And” means this requirement and the previous requirement must both be met; selecting “Or” means at least one of the multiple requirements must be met.
6. Press “Save.”

Skip Logic

The Skip Logic feature can allow you to alter survey flow according to a certain question’s answer. You can cause a certain question’s response to end the survey or skip to a later part of the survey. To use Skip Logic:

1. Right-click on the question. It will become highlighted in light blue.
2. Select “Add Skip Logic…”
3. Fill out the condition you want this question to meet for Skip Logic to be activated. You will be able to select one of the answer choices available in this question, and then decide whether Skip Logic is applied when this answer “Is Selected,” “Is Not Selected,” “Is Displayed,” or “Is Not Displayed.”
4. Decide where the participant should be redirected. When the Skip Logic is activated, one or more questions will be skipped (not shown to the participant). You can choose the following options for the related “Skip To:” field:
   1. Another question in this block: skip all questions after this one up until the question selected in this field
   2. End of Block: skip all questions after this one in the same block; participant starts answering again at beginning of next block
      • can be used to serve similar purposes to Display Logic (described above)
   3. End of Survey: skip all other questions; the survey is complete
      • can be used to disqualify a participant early on who answers a question that precludes them from participating in the rest of the survey
5. Note: Only questions shown on separate pages from the question inducing Skip Logic can be skipped. That is if Question A and Question B are shown on the same page, and A is set to skip B if the answer to A is given as “100,” nothing will happen. Question B will not disappear whenever A is answered as “100”. For this to work properly, B must be shown on a different page from A.

Survey Question Design

There are many ways the same questionnaire can be built on Qualtrics. One major choice is whether to group similar questionnaire items, via a question of type “Matrix Table” or “Side by Side,” or to enter each questionnaire item individually, via a question of type “Multiple Choice,” etc. The differences concern:

• Aesthetic: grouped questionnaire items often appear closer together than individualized questionnaire items and are more likely to be perceived as a cohesive unit
• Functionality: you cannot use Skip or Display Logic to navigate amongst questionnaire items displayed in the same group as one question (e.g. you can’t use Item 1A’s answer to determine if Item 1B shows) since they are all one question rather than individual questions within the same block
• Data output: questions with multiple “parts” (i.e. grouped questionnaire items) output their data as coming from the same question with subparts (i.e. Q1_1, Q1_2, Q1_3) while questionnaire items entered as individual questions appear as coming from different questions (i.e. Q1, Q2, Q3)

Considering these differences, it is preferable to group items coming from the same questionnaire into a single “Matrix Table” or “Side by Side” question. If the questionnaire is extremely lengthy, you may want to group subscales rather than the entire survey. Under ordinary circumstances, do not group items from different questionnaires into single questions.

Also keep in mind that “Slider” questions, which can be used to create visual analog scales, cannot be combined with other sub-question types in a “Matrix Table” or “Side by Side” question. The only way to use a continuous rating scale like this is to place it in its own question. However, the “Slider” question permits multiple sub-questions (“Choices”) of the same type, so you can present multiple continuous rating questionnaire items within the same Qualtrics question.

Repeat Headers

Sometimes a questionnaire will be so long that the anchor labels (i.e. the meaning of “1,” “2,” “3,” etc.) will be hidden as the participant scrolls past them down the page. If this is the case for your questionnaire, use a “Side by Side” question type to present the questionnaire items. As you will see in the appendix entry on Qualtrics Options, this question type has a parameter called “Repeat Headers.” This allows you to determine how many time the anchor labels (headers) are shown: not at all; once, at the top; twice, at the top and middle; twice, at the top and bottom; three times, at the top, middle and bottom; or before every questionnaire item.

Survey Design

Survey Flow

The interface for determining the progression of a Qualtrics questionnaire is the Survey Flow view. By default, different questionnaire blocks appear in the order they are arranged in the general view. With the Survey Flow view, you have more power over how and when your various questionnaire blocks appear. (Note that the only way to organize individual questions within the blocks is from the general view. Survey Flow view is only for managing the order of whole blocks.) Access Survey Flow view by clicking “Survey Flow” along the top menu when you are designing a survey.

Survey Flow supplies an overview of how your survey will be presented to the participant, moving from beginning to end, top to bottom. Each item along this progression is called an Element. Survey Flow is arranged such that each Element coming off the main flow (shown as a grey “backbone” along the left side, with arrows coming from it) will always occur. You can include “Branch” and “Randomizer” Elements to make some blocks show only conditionally. In summary, the different Element types and their primary options are:

• Block: used to present a question block that you created in the general view
• Branch: used similarly to Display Logic to create large-scale conditional display; when the conditions of this Element are met, the survey flows down to its nested Elements
  • conditions are set similarly to conditions in Display Logic
  • the first field in all conditional statements beyond the first will have the familiar “And” and “Or” options, but also the “Move to a new Logic Set” option; this allows you to create multiple overarching Logic Sets of conditions, which may have an “And” or “Or” relationship amongst themselves
• Randomizer: used to randomly present Elements nested underneath
  • Randomly present X of the following elements: set value anywhere from 0 to the total number of nested Elements; allows you to randomly display only a few of the nested Elements or all of them
  • Evenly Present Elements: when checked, Elements will be presented evenly (i.e. randomization will result in an even split of all presentations)
• Conjoint: used to join back together various diverging flows, created by various “Branch” or “Randomizer” Elements; after the termination of previous Elements, all flows will continue to the element immediately after the Conjoint
• End of Survey: used to end the survey; can be customized via “Customize → Override Survey Options”
  • Custom end of survey message allows you to insert a different survey message than the one Qualtrics displays by default
    • draft a custom message by using “Select a message → My Library → New Message…”
• Embedded Data, Web Service, and Authenticator are more advanced Element types; see the full Qualtrics documentation for more information

Randomizing Questionnaire Content

For some studies, there may be a good reason to administer questionnaires in a fixed order across participants (i.e., when we expect some questionnaires to influence responses to other questionnaires and want to put the more influential questionnaires last). However, for other studies, it may make sense to randomize the order of questionnaires, while keeping the set of questions that make up each questionnaire together (e.g. if we are trying to control for general order effects). You would use “Question Randomization” to randomize the order of questions within a questionnaire; typically, you won’t want to do this when using a previously validated questionnaire, because question order will affect the reliability and validity of that measure. You would use Randomizer Elements under Survey Flow to present the questionnaires themselves in random order.

• To randomize the questions within a questionnaire or questionnaire subsection:
  1. Be sure to group all the questions you wish to randomize into the same block.
  2. At the top of this block, select “Block Options → Question Randomization.”
  3. Choose “Randomize the order of all questions.”
  4. Also, keep in mind that question randomization overrides and ignores page breaks, skip logic, and rules about the number of questions per page. Strategically separate the questions into blocks to avoid this becoming an issue.
• To randomize the questionnaires within an entire survey:
  1. Go to Survey Flow view and add a Randomizer Element.
  2. Add elements to display each block that contains a whole questionnaire.
     1. If one single questionnaire is broken up into multiple subunits, you still want to display those subunit blocks together, rather than interspersed with the other blocks holding whole questionnaires.
     2. To present questionnaires divided into blocks like this, add a new Randomizer Element under the initially added overarching Randomizer Element.
     3. Within this second “Randomizer” element, add blocks to display the subunits of the broken-up questionnaire.

Other Useful Tips & Tricks

Measuring Timing

A useful feature for recording data on the way participants responded to the survey is the hidden “Timing” question. To gain access to this feature:

1. Add a question of type “Advanced → Timing” to a page.
   1. This question is invisible to the participant and does not require any action on their part, but records four measures regarding their activity on that page:
      1. First Click: seconds between page load and first-time participant clicks something
      2. Last Click: seconds between page load and last time participant clicks something (other than “Next” button)
      3. Page Submit: seconds between page load and clicking the “Next” button
      4. Click Count: total number of clicks on the page

*Note that these measures are by page, not by block or question.

1. The “Timing” question also has a few other useful features.
   1. You can set “Seconds to disable submit,” which hides the “Next” button for a certain amount of time.
      1. This may be useful for small numbers of questions in which you wish to delay submission,
      2. In general, it may be easier to use JavaScript to do this (see Presenting stimuli → Code 1 below).
   2. You can also set “Seconds to auto-advance” the page, which can be useful in controlling study pacing.
   3. The last feature is “Show Timer” which shows a timer that counts up or down (your choice) to show participants how long they are spending on the page.
2. The total survey duration is measured automatically by Qualtrics and is included in the output data.
   1. All “Timing” question data is exported along with participant data from the rest of the survey questions.

Proofreading and Piloting the Survey

All surveys must be free of typing and grammatical errors, which detract from our image of professionalism and can negatively affect assessment results. You should also pilot the assessment package on various devices: tablets, smartphones, common computer browsers (e.g. Chrome, Internet Explorer, Firefox, Safari). Ask other lab members to take the survey, as well, to make sure content shows up correctly for other people and to estimate survey duration. They may also notice typos and grammatical errors that you had not noticed.

1. If you’d like to display duration time to pilot testers so that they can report the time taken on the survey to you, you can include a block towards the end that reveals this information:
   1. Place a page break before the question
   2. Set the question type as “Descriptive Text”
   3. Put the following as the text: “The survey took ${e://Field/Q_TotalDuration} seconds.” Be sure to delete this question/block before releasing the survey to actual participants.

Qualtrics Error-Checking

1. Before continuing with the task, check the Qualtrics survey data output and feasibility.
2. Complete the entire Qualtrics questionnaire yourself. Answer the questions seriously, as if you were a real participant. Time yourself to estimate the questionnaire duration.
3. Have several others complete the entire Qualtrics questionnaire. Ask them to answer the questions seriously, as if they were real participants. Ask them to time themselves to estimate questionnaire duration, or have Qualtrics output their time.
4. Check through the Qualtrics questionnaires to make sure all text matches original questionnaires. If possible, have other members do this as well.
5. Double-check that question response values in Qualtrics match original questionnaire answer choice values. (For example, the default value for the first answer choice in Qualtrics is 1, but for some questionnaires, this value should be 0.) Adjust response values in Qualtrics as needed to match questionnaire values and avoid recoding during data analysis.
6. Make sure that you and at least one other person in the lab have taken the entirety of the last version of your Qualtrics questionnaires before collecting any real data.
7. Have someone else that is qualified to evaluate your work and check your raw questionnaires for errors and examine the Qualtrics survey flow.

Permissions

Only the owner of a Qualtrics survey can assign permissions to other users, via the “Share Project” option in the right dropdown list beside the project name. Because of this, it is possible that once you graduate or otherwise leave HGAPS, new personnel will be unable to share the project (i.e. give editing rights) to new personnel. There are a few options to avoid or correct this issue:

1. Have Eric create the survey and share it with you. This way, Eric will be the owner and can always assign permissions to new personnel.
2. Create a copy of the survey. Create a clone of the survey, so that you are now the owner of it.
   1. Find the survey. View your list of shared surveys in the main “Shared with Me” window.
   2. Make a copy. In the far-right dropdown menu, select the “Copy Project.”
   3. Transfer copies of responses. Enter the original project. Navigate to “Data & Analysis → Export & Import.” Choose “Export Data… → Export Data with Legacy Format.” Now, enter your copied project. Navigate to “Data & Analysis → Export & Import.” Choose “Import Data…” and upload a copy of the responses you just created.

Appendix: Qualtrics Options

Question Type	Characteristic	Option	Description
Multiple Choice	Answers	Single Answer	participant can select exactly 1 choice
		Multiple Answer	participant can select anywhere from no to all choices
		Dropdown List	participant can select exactly 1 choice, from a dropdown menu
		Select Box	participant can select exactly 1 choice, from a box with all options displayed
		Multi-Select Box	participant can select anywhere from no to all choices, from a box with all options displayed; selecting multiple choices requires holding down CTRL or SHIFT key
	Position	Vertical	choices arranged one after another, top to bottom
		Horizontal	choices arranged one after, left to right
		Column	choices arranged in columns; you decide how many columns
Matrix Table	Statements	Integer	determines how many statements, items, etc. the participant will be rating
	Scale Points	Integer	determines how many points are along the scale on which the participant is rating the statements
	Labels	Integer	determines how many labels/anchors are shown above the scale points (some layouts automatically place a text box above each scale point)
	Type	Likert	conventional matrix-style question; participant responds to each statement along a scale that is assumed to be arranged in a logical order
		Bipolar	participant selects the degree of preference (along a certain number of “Scale Points”) between a left label and a right label
		Rank Order	participant freely responds (with a number) to rate “Statements” along various “Scale Point” characteristics
		Constant Sum	participant freely responds (with a number) to rate “Statements” along various “Scale Point” characteristics; the total number of points given to each “Statement” is summed at end of row
		Text Entry	participant freely responds (with a number or letter) to rate “Statements” along various “Scale Point” characteristics
		Profile	participant responds to each “Statement” by selecting the corresponding answer from among the “Scale Points”
		MaxDiff	participant responds to each “Statement” (positioned in the middle of the table) with one of two answers (positioned on the outside of the table)
Text Entry	Text Type	Single Line	sets text box size as a single line
		Multi-Line	sets text box size as multiple lines
		Essay Text Box	sets text box size as a large entry box
		Form	shows multiple text boxes for entry within one question block
		Password	text written in the box is obscured with dot characters
	Validation Type	Min Length	requires a response to be at least “X” characters long
		Max Length	requires a response to be at most “X” characters long
		Character Range	requires a response to be between “X” and “Y” characters long
		Content Validation	requires a response to meet the criteria to be a valid email address, phone number, date, etc.
Slider	Labels	Integer	determines how many labels/anchors are shown above the scale points
	Type	Bars	participant drags edge of the box to select choice
		Sliders	participant drags slider along track to select choice
		Stars	participant selects a certain number of stars; can be “Discrete” (full stars only), “Half Step” (full or half stars only), “Continuous” (any portion of a star)
	Grid Lines	Integer	number of guiding lines to show along the box or slider track (not used with “Stars”)
	Min Value	Integer	minimum value coded (not used with “Stars”)
	Max Value	Integer	maximum value coded (not used with “Stars”)
	Number of Decimals	Integer	number of decimals output in the datasheet (not used with “Stars”)
	Options	Show Value	show the participant the current value of their selection at the right side of the grid
		Custom Start Position	set the “Slider” question’s default value to be a certain position (by default, is set to minimum value)
		Not Applicable	present a “Not Applicable” (can be renamed) box to the side of the grid
Rank Order	Items	Integer	sets several items, statements, etc. for the participant to order
	Type	Drag and Drop	participant drags and drops items into a preferred order
		Radio Buttons	participant selects radio button corresponding to preferred order; disallows “ties”
		Text Box	participant inputs text to rank items
		Select Box	participant orders items using up and down arrows
	Validation Type	Must Rank Between	participant must indicate ratings for a certain number of items (not used with “Drag and Drop” or “Select Box”)
		Must Rank All	participant must choose ratings for all items (not used with “Drag and Drop”)
Side by Side	Statements	Integer	determines how many statements, items, etc. the participant will be rating
	Columns	Integer	determines how many qualities, characteristics, etc. the participant will be rating each “Statement” along
	Column Options	Add/Remove Answer	changes number of answers offered for that “Column”
		Move Right/Left	changes order in which “Columns” are presented
		Response type	sets that “Column” to “Scaled Response (Likert)” or “Open-Ended Text”
		Response number	sets that “Column” to allow only a “Single Answer” or “Multiple Answer,” or to present as a “Drop Down List”
	Repeat Headers	None	text labels for “Answers” in each “Column” shown only once (at the top)
		Middle	text labels for “Answers” in each “Column” shown at top and middle of “Statement” set
		Bottom	text labels for “Answers” in each “Column” shown at top and bottom of “Statement” set
		Both	text labels for “Answers” in each “Column” shown at top, middle, and bottom of “Statement” set
		All	text labels for “Answers” in each “Column” shown before every “Statement”