Survey research and design in psychology/Assessment

Assessment

Overview edit

There are 9 online quizzes (5% each; total 45%), a data collection and entry exercise (10%) and a lab report (45%).
All assessment is optional. Non-completed assessments will be awarded 0.
A final overall mark of 50% or higher is required to Pass the unit. Partial final marks will be rounded up.
Summary of assessment items:

Item	Weight	Description	Generic skill	Learning outcome	Due date	Late submissions
Quizzes	45%	9 x online quizzes worth 5% each	Problem solving	Survey research and design; interpret correlation-based statistics, including EFA, reliability, and MLR.	Mon 09:00 AM of Week 14	Not accepted
Data collection & entry	10%	Collect and enter data for 5 cases	Analysis and inquiry	Conduct survey-based research in psychology.	Mon 09:00 AM of Week 4 (data file) / Mon 09:00 of Week 6 (hard copies)	Not accepted
Lab report	45%	APA style lab report with EFA and MLR	Analysis and inquiry	Use SPSS; Interpret correlation-based statistics. Communicate in writing the results of survey-based psychological research.	Mon 09:00 AM of Week 13	Accepted up to 7 days late with -5% per day penalty

Quizzes edit

Topics: There are 9 online quizzes, each worth 5% (total 45%), with the following topics:
1. Survey research
2. Survey design
3. Descriptives and graphing
4. Correlation
5. Exploratory factor analysis
6. Psychometrics
7. MLR 1
8. MLR 2
9. Power and effect size
Availability:
1. Quizzes will be available until Mon 09:00 Week 14.
2. UCLearn quizzes are only available to enrolled University of Canberra students.
Late submissions: Not accepted
Assessed knowledge: Quizzes assess knowledge of concepts covered in lectures, tutorials, and readings.
Quiz content:
1. Each quiz consists of 10 multiple choice questions drawn randomly from a larger test bank.
2. The questions from each quiz correspond directly to a single lecture and its related readings.
3. Some quizzes are harder than others (e.g., the lowest mean is typically for Quiz 05: Exploratory factor analysis).
Question style: Some questions have more than one correct answer.
1. In order to get full marks, all correct the answers must be selected and no incorrect answers must be selected. For example, if a question has 5 answer choices, 2 of which are correct (each correct answer would be worth 50%, adding to +100%) and 3 of which are incorrect (each incorrect answer would be worth -33% each, adding to -100%). The mark awarded is the total of the marks for each of the answer options selected. However, no negative total marks will be awarded for a question, so the lowest mark that can be awarded for a question is 0.
2. The reason for including negative marks for incorrect answers for multiple response question types is that, otherwise, all answer options could be selected to achieve a full mark, even though some incorrect answers were selected. So, there needs to be some penalty for selecting incorrect options.
Time limits: Quiz time limits vary between 10 and 20 minutes, depending on the complexity of the questions.
Academic integrity: Quizzes are to be completed independently (i.e., without assistance of others), in your own time.
Open book: Quizzes are open book - recommended resource materials include lecture, tutorial, and reading notes.
Number of attempts: Each quiz can only be attempted once.
Feedback: Answers to the quiz questions are shown immediately after quiz submission, but are no longer accessible once the feedback window is closed. Take any notes for study purposes prior to closing the feedback screen.
Bonus marks: A bonus quiz mark is awarded for identifying an error or significant improvement to a quiz question - email details to the unit convener. For example, a student who got 7/10 on Quiz 1, emailed the unit convener to correctly point out a spelling error in that quiz, resulting in a bonus mark being added and the student's new Quiz 1 mark becoming 8/10.
Practice quizzes: Some practice quizzes are available:
1. Quiz 0 (a practice quiz) can be attempted as many times as you like. Use this quiz to test and make sure that you are familiar with how the quiz system works on the computer you are using.
2. Several practice quizzes are available on Wikiversity, including:
3. Pearson - see External link sections on the lecture pages

Data collection and entry edit

This assessment exercise provides hands-on involvement in real world survey data collection and data entry. It requires collecting data using a survey, entering data, submitting an electronic data file, and submitting the hard copy surveys in an accurate, timely fashion as part of a larger class project. The specific steps are to:

Collect 5 cases of survey data by following the survey administration guidelines.
Enter the survey data by following the data entry guidelines.
Submit:
1. Electronic data file via UCLearn by Monday 09:00 AM of Week 4 - no late submissions accepted
2. Hard copy surveys as a single bundle with a coversheet by Monday 09:00 AM of Week 6 - no late submissions accepted - see hard copy survey guidelines

Important note: In order to earn any marks, it is necessary to submit both electronic and hard copy data on time. No marks will be provided if either the electronic or hard copy submissions are missing or late.

Lab report edit

Overview edit

Present an independently-developed APA style lab report which examines:
1. the psychometric properties of a multi-item, multi-factorial measure (either time management or time perspective) using exploratory factor analysis and reliability analysis (guided by a research question)
2. an explanatory model involving at least three predictors and one dependent variable using multiple linear regression (guided by hypotheses - one per predictor).
Use the class-aggregated data file from the data collection and entry assessment exercise, based on the
Surveys about university student motivation, satisfaction, and time management

Surveys edit

These surveys were designed for use by an undergraduate psychology class (Survey Research and Design in Psychology, 2005-2018):

Time and Stress Questionnaire for University Students v.1 (TSQFUS1)
The University Student Satisfaction and Time Management Questionnaire v.9 (TUSSTMQ9)
The University Student Motivation & Satisfaction Questionnaire v.2 (TUSMSQ2)

Students used these surveys to collect data, entry data, and conduct analyses for a lab report.

Using these surveys edit

These instruments and their items are free to use, adapt etcetera under a Creative Commons Attribution International 4.0 license.

However, be aware that the surveys in their current format are intentionally designed to not be "perfect" so that emerging scholars studying subjects such as "Survey research and design in psychology" can collect data and then practice exploratory factor analysis .

There is also intentionally no scoring key . Factor analysis is recommended to help determine the underlying factor structure and to identify which items to use to calculate composite scores. In other words, there is a latent structure, but you'll need to work it out. For example, for university student motivation, see these suggestions. Composite scores representing underlying constructs can then be used for descriptive statistics and hypothesis testing.

Psychometrics edit

There are no reported psychometrics for newly developed items and scales in these survey instruments. Where intact, previously published measures were included, psychometrics may be available.

Users of these surveys should be prepared to conduct their own psychometric analyses (factor structure, reliability, and validity) based on their own samples.

Surveys edit

These surveys were designed for use by an undergraduate psychology class (Survey Research and Design in Psychology, 2005-2018):

Time and Stress Questionnaire for University Students v.1 (TSQFUS1)
The University Student Satisfaction and Time Management Questionnaire v.9 (TUSSTMQ9)
The University Student Motivation & Satisfaction Questionnaire v.2 (TUSMSQ2)

Students used these surveys to collect data, entry data, and conduct analyses for a lab report.

Using these surveys edit

These instruments and their items are free to use, adapt etcetera under a Creative Commons Attribution International 4.0 license.

However, be aware that the surveys in their current format are intentionally designed to not be "perfect" so that emerging scholars studying subjects such as "Survey research and design in psychology" can collect data and then practice exploratory factor analysis .

There is also intentionally no scoring key . Factor analysis is recommended to help determine the underlying factor structure and to identify which items to use to calculate composite scores. In other words, there is a latent structure, but you'll need to work it out. For example, for university student motivation, see these suggestions. Composite scores representing underlying constructs can then be used for descriptive statistics and hypothesis testing.

Psychometrics edit

There are no reported psychometrics for newly developed items and scales in these survey instruments. Where intact, previously published measures were included, psychometrics may be available.

Users of these surveys should be prepared to conduct their own psychometric analyses (factor structure, reliability, and validity) based on their own samples.

Surveys edit

These surveys were designed for use by an undergraduate psychology class (Survey Research and Design in Psychology, 2005-2018):

Time and Stress Questionnaire for University Students v.1 (TSQFUS1)
The University Student Satisfaction and Time Management Questionnaire v.9 (TUSSTMQ9)
The University Student Motivation & Satisfaction Questionnaire v.2 (TUSMSQ2)

Students used these surveys to collect data, entry data, and conduct analyses for a lab report.

Using these surveys edit

These instruments and their items are free to use, adapt etcetera under a Creative Commons Attribution International 4.0 license.

However, be aware that the surveys in their current format are intentionally designed to not be "perfect" so that emerging scholars studying subjects such as "Survey research and design in psychology" can collect data and then practice exploratory factor analysis .

There is also intentionally no scoring key . Factor analysis is recommended to help determine the underlying factor structure and to identify which items to use to calculate composite scores. In other words, there is a latent structure, but you'll need to work it out. For example, for university student motivation, see these suggestions. Composite scores representing underlying constructs can then be used for descriptive statistics and hypothesis testing.

Psychometrics edit

There are no reported psychometrics for newly developed items and scales in these survey instruments. Where intact, previously published measures were included, psychometrics may be available.

Users of these surveys should be prepared to conduct their own psychometric analyses (factor structure, reliability, and validity) based on their own samples.

See also edit

University student motivation) data file e.g., "It is hypothesised that [IV1], [IV2], and [IV3] will each positively predict [DV]".

Method edit

Clearly explain how the study was conducted in sufficient detail to allow a replication study, but without extraneous detail.
Key marking criteria: Is the study replicable? Is sufficient detail provided for a "naive person" (say, someone in Japan in 20 years time) to be able to fully replicate the study?

Participants (5%)

Provide a one to two paragraph descriptive overview of the participants in the final (after data screening) sample.
Consider which of the available data can be summarised in order to provide an insightful description.
Advanced option: You may wish to compare the sampled data with population statistics for UC students (e.g., see UC at a glance and Annual Reports).

Measures (5%)

Briefly summarise the development of the survey instrumentation.
For the measures used to collect the data which is analysed in the Results, describe e.g.
1. type of questions
2. response format
3. any reverse-scoring and meaning/direction of high/low scores
4. Do not describe measures which are not used in the current study.
5. Optional: Use a table to help present the proposed factors (e.g., label, definitions and example items)

Procedure (5%)

Sampling:
1. What was the target population and the sampling frame?
2. What sampling technique was used?
Administration:
1. Briefly summarise and provide an APA style reference for the Survey administration guidelines
2. Where and how did you collect data?
3. How long did participants typically take to complete the survey?
4. Refusal rate? (for the surveys you administered)
5. Procedural anomalies? (e.g., explain any unanticipated responses or unplanned occurrences)

Results edit

The analysis should proceed through three basic steps:
1. Data screening - Summarise in one to two paragraphs how the data was screened and what changes were made. Is enough detail provided for the same steps to be followed by someone else? However, avoid excessive detail (e.g, CaseID number are meaningless to a reader).
2. Sample size assumptions do not belong in this section - instead, this would belong in the section(s) for the corresponding analyses.
3. Psychometric instrument development
  1. Conduct EFA of a multidimensional construct (either the time perspective items or the time management items).
  2. For each extracted factor, provide reliability analysis (internal consistency - Cronbach's alpha), composite score descriptive statistics, and correlations between factors.
4. Multiple linear regression - Conduct a MLR with at least three IVs to address one hypothesis per IV
Communicate the depth of your understanding by using your own words; avoid writing results in a robotic (mindless) manner (e.g., avoid overparaphrasing a specific sample write-up)
Most statistics should be rounded to two decimal places unless there is particularly useful information communicated by including a third decimal place (e.g., when reporting exact p values).
Scope and depth of analysis
1. Additional analyses may be presented. However, it is quite possible to gain maximum marks by conducting one of each of the required analyses. If additional analyses are presented, then they must be clearly related to the research question and hypothesis(es).
2. In marking, some account will be taken of the scope of the analysis undertaken. Where a more advanced analysis is appropriate (given the research questions(s) and/or hypothesis(es)) and is well conducted, this could represent higher quality work than a simpler analysis. However, there's much also to be said for parsimony (keep it simple and get it right) by focusing on doing a good job of fulfilling the minimum requirements. The best reports are usually not the most complex ones. If in doubt, go with analyses which meet the minimum criteria, which relate to the research question and/or hypotheses, and which you are confident about accurately conducting, interpreting and presenting.

Data screening (5%)

Summarise what was done to check and correct errors in the data - see data screening
Statements like ""All analyses were conducted using SPSS version 23." are unnecessary.

Psychometric instrument development (20%)

Report the results of an EFA of a multidimensional survey construct (either time perspective or time management)
1. The minimum requirement is to report psychometric analysis (EFA, composite scores, correlations between composite scores, and internal consistency) for one set of items (time perspective or time management). However, it may also be of interest to conduct psychometric analysis of another set of items (e.g., stress) in order to develop other composite scores for further analysis. In this case, present one EFA (of either time perspective or time management) in full and briefly summarise the results of the other EFA, perhaps with relevant output in an appendix.
Indicate the type of EFA used (Type of extraction? Type of rotation?)
Explain the extent to which EFA assumptions were met, but not excessively (e.g., one indicator of factorability is quite sufficient; more is redundant)
1. Sample size (incl. cases:variables ratio)
2. Linearity (e.g., check at least some scatterplots, particularly for bivariate outliers or non-linear relations)
3. Factorability of the correlation matrix (either examine via item correlations, anti-correlation matrix diagonals, or a measure of sampling adequacy (KMO or Bartlett's) - but do not report all of these as they are redundant)
Focus on the final model but summarise the steps taken to get there (e.g., How many factors were extracted initially? What models/factors structures were examined? To what extent was the expected structure evident?)
% of variance explained (for the initial and final model(s))
Label and describe each factor
Which items were retained and/or dropped and the reasons why
Table of factor loadings (sorted by size) and communalities (for the final model)
Reliability analysis (Internal consistency - (Cronbach's alpha)) for each factor
Calculation of composite scores to represent each factor
Table of descriptive statistics for the composite scores
Table of correlations between composite scores

Multiple linear regression (20%)

Report the results of a MLR with at least three predictors - can use any variables in, or derived from, the supplied data set (if they meet the assumptions for MLR)
Reiterate the purpose (research question and/or hypotheses) of the MLR
Mention the type of MLR (e.g., standard, hierarchical, or stepwise)
Describe the IVs and DVs, and any manipulations of the variables (e.g., recoding or creating an interaction term). If not already clear from the Method, clarify the direction of scoring.
Explain the extent to which assumptions were met (e.g., sample size, multicollinearity, multivariate outliers)
Present the correlations between the items (can be part of the MLR coefficients table - see sample write-ups for examples). Demonstrate understanding of the directions of any relationships (e.g., if there is a positive correlation between X and Gender, what does this mean? Are higher values of X associated with males or females?)
Report amount of variance explained (R² and Adjusted R² (and the R² change at each step if a hierarchical MLR is being conducted), along with inferential tests (F(df), p)
Report significance, size, direction and relative contribution of each IV. Make sure to explain what the direction of the relationships mean in plain English.
Table showing the correlations and MLR coefficients, including B for intercept & IVs and Beta (β), and the statistical significance (e.g., t, p), and semi-partial correlations squared (sr²) for each IV and explain the direction and size of the results.
Consider the shared and unique percentages of explained variance.

Discussion edit

Build on the introduction to explain the results and what they mean in a balanced manner.
Demonstrate breadth and depth of understanding of the results and their implications. Avoid merely summarising the results without providing additional critical commentary.
Critically review the strengths and weaknesses of the study's methodology and make practical suggestions for how it could be improved, e.g.,
1. Validity and reliability of the measures?
2. Statistical power?
3. Appropriateness of the sampling technique?
4. Generalisability of the findings?
Provide tangible recommendations for future research and practical implications.

References edit

Is the reference list complete (i.e., none missing and all cited)?
Does the lab report make effective use of a core set of relevant, high-quality, peer-reviewed, citations? This involves citing and meaningfully discussing appropriate references (as opposed to just dumping citations without explanation), particularly in the Introduction and Discussion.
Reference the instrumentation and the survey administration guidelines - do not copy the guidelines into the Appendices.
Use APA style, including for electronic sources. Include DOIs where relevant.

Appendices edit

Appendices are not needed - they are optional.
Appendices are for additional detail which is relevant to understanding the main body, but which would break the flow of the main report e.g., the correlations between the items used in the factor analysis.
Appendix content does not need to follow APA style but should be well organised, with clear labeling.
Each appendix should be referred to at least once in the main body.

Style edit

Follow APA style except:
1. add a cover sheet
2. use single-spacing (for electronic submission; double-spacing is a relic from paper submission days - it allowed room for hand-written comments)
3. do not include author name on the title page (to facilitate blind marking)
4. include any tables or figures in situ (i.e., embed them without the main body of the report rather than putting them after the references)
5. formatting of any appendices does not need to follow APA style
Example (owl.english.purdue.edu)
The most important aspects of APA style for this lab report are:
1. Times New Roman 12 pt font
2. Running head with page numbers
3. Page margins
4. Left-justify body text
5. Captioning and layout of tables
6. Right alignment of statistics in tables
7. Citations and referencing (including for electronic references and use of DOIs)
8. Symbols, abbreviations, and formatting of statistical symbols (including M, SD, skew, kurt, N, n, F, t, p, r, R, R², sr²)

Word count edit

There is no minimum word count.
Maximum word count: 2800 words + 10%
Count everything using a word processor from the beginning of the Introduction (i.e., not Title page, Abstract etc.) to end of Discussion, including all text, headings, footnotes, citations, tables and figures etcetera (i.e., but not the sections after the Discussion, such as References and Appendices).
Penalty for exceeding maximum word count: Markers will ignore words beyond the maximum (i.e., most likely resulting in a reduced mark for the Discussion).
Word counts provided in the section overview are suggestions only.

Independent writing and plagiarism edit

It is necessary to demonstrate independent thinking and writing in order to satisfy the lab report learning outcomes. In other words, avoid plagiarising from these samples or other guides. Some sample write-ups are provided to give ideas. Also seek out other examples (e.g., see the readings). Then write up your results in your own words.

If you are in some doubt about whether you've demonstrated sufficiently independent writing, then upload the assignment to get a text-matching analysis via Urkund. Text-matching for this assignment between about 10% and 30% is reasonably common and probably indicates no particular cause for concern. However, it is recommended that drafts with text-matching scores above approximately 30% should be reviewed and rewritten in order to more effectively demonstrate independent understanding and writing.

Sample write-ups edit

Some previous High Distinction (HD) reports:

Example 2007 HD lab report (.pdf) - Marking and feedback sheet (.xls)
Example 2013 HD lab report (.pdf)
Example 2016 HD lab report (with qualitative analysis for G students and feedback) (.pdf)

Sample write-ups for specific analyses. Note that these samples are longer than is likely to be able to be presented in the lab report within the word count limit.

Exploratory factor analysis (.docx)
MLR (.doc)
Some of the readings also provide examples of APA style write-ups for specific analyses.

Submission edit

Submit the lab report electronically via the unit's UCLearn site.

Late submissions edit

Standard UC policy applies:
1. 5% penalty per day (including weekends)
2. no submissions accepted or marked after 7 days

Academic integrity edit

Emerging academics have a responsibility to uphold University standards on ethical scholarship. Good scholarship involves building on the work of others and use of others’ work must be acknowledged with proper attribution made. Cheating, plagiarism, and falsification of data are dishonest practices that contravene academic values. The Academic Skills Centre provides opportunities to enhance student understanding of academic integrity.
Participants are expected to submit independent work on the assessment items.
Note the University of Canberra policy on plagiarism.

Extensions edit

Participants are expected to work on the assessment items throughout semester. Extensions will only be granted in exceptional circumstances. Early communication of problems is strongly advised. Participants should assess within the first few weeks of semester whether they have a reasonable likelihood of being able to complete the unit and should consider withdrawing by the census date if not keeping up.
Extensions will not be granted for:
1. Workload (e.g., study load and/or paid or voluntary workload) - such problems should be anticipated and withdrawal from the unit is recommended if workload is a problem.
2. Technical problems (e.g., lost/corrupted/damaged storage media, software/internet access problems, and viruses) - it is strongly recommended that you keep multiple and regular backups of lab report drafts, data, syntax, and output files.
3. Undocumented issues.
Extension requests should be submitted via email to the unit convener from the requester's UC student email address and include:
1. First and last name
2. Unit number and assessment item(s) an extension is requested for
3. Length of extension request
4. Reason for extension request
5. Documentary evidence with correct page orientation - for example:
  1. A medical certificate (from a medical, dental, or health practitioner) which provides:
    1. Registered provider number, provider's contact details, qualifications, and signature
    2. Date(s) of consultation
    3. Duration of incapacity to study (must be during the current study period and prior to the due date)
  2. A death notice or other appropriate documentation for bereavement.
  3. A Reasonable Adjustment Plan from Inclusion and Welfare at the University of Canberra
For more information:

Criteria	Description	%	Suggested word count
Title/Abstract	Succinct, specific title. Abstract covers purpose, method, findings, and conclusions.	5%	10-15, 160-200
Introduction	Establishes the problem, reviews theory and research, develops research question(s) and hypotheses	10%	400-700
Method	Describes method and design, including Participants, Materials, and Procedure	15%	500-700
Results	Screens (5%) and analyses data using EFA, internal consistency, descriptive statistics and correlations for the composite scores (20%), and MLR (20%)	45%	900-1500
Discussion	Summarises and interprets the findings, considers implications, and makes recommendations	25%	500-750

Survey research and design in psychology/Assessment

Surveys edit

Using these surveys edit

Psychometrics edit

See also edit

Data download, screening, & recoding edit

Overview of sections edit

Cover sheet edit

Running head edit

Title page edit

Abstract edit

Key words edit

Introduction edit

Surveys edit

Using these surveys edit

Psychometrics edit

See also edit

Surveys edit

Using these surveys edit

Psychometrics edit

See also edit

Method edit

Results edit

Discussion edit

References edit

Appendices edit

Independent writing and plagiarism edit