Survey research and design in psychology/Assessment/Lab reports

Overview

1. There are 5 lab reports each worth 10% (total 50%).
2. Lab reports 2 to 5 require emerging academics to independently:
   1. Analyse some variables from the combined survey data file; and
   2. Write up to 1,000 words in APA style - see word count
3. Lab reports are due Mon 09:00 at the end of each Module (Week 3, 5, 8, 11, and 13).
4. Late penalty is 5% per day pro rata (e.g., 12 hours late = 2.5% penalty)

Generic guidelines

This section summarises the marking criteria and weighting (indicated by %s). Please also see the specific guidelines which explain detailed requirements and marking criteria for each lab report: LR1, LR2, LR3, LR4, LR5.

Lab reports 2 to 5 are mini APA-style lab reports which should each contain the following sections. Note that %s indicate approx. weighting/ Actual marking criteria and weightings are in the specific guidelines for each lab report. Section-specific word counts are suggestive only.:

1. Coversheet
2. Title page: APA style, including a unique, meaningful title for the report.
3. Abstract: ~5%. A succint overview. Brief, but comprehensive; efficiently overviews the study and its findings. (~50-100 words).
4. Introduction: ~10%. Develop logically-derived and justified research question(s) and hypothesis(es). This section should be relatively short because the lab reports are mostly about developing your skills with writing Method, Results, and Discussion sections. The main goal of the Introduction is to identify a problem (research question) which is related to some key literature and which leads to clearly stated and testable hypotheses. Some background references will be provided on eReserve and effective use of these could be sufficient, however stronger Introductions will tend to also utilise other relevant academic literature.
5. Method: ~10%. Lab reports 2 to 5 will vary in their requirements for the Method sub-sections (Participants, Instrumentation, Procedure) in order to avoid repetition across the reports - check detailed requirements. Note that Design and Analysis sub-sections are not necessary.
6. Results: ~50%. Each lab report should report results for the following analyses:
   1. describe and graph the correlational relationship between at least one nominal/ordinal and at least one interval/ratio variable; also describe and graph the linear correlation between two or more interval/ratio variables (Lab report 2)
   2. the factor structure of at least one multi-dimensional survey instrument using exploratory factor analysis, reliability analysis, and creation of composite scores; (Lab report 3)
   3. test at least one hypothesis using at least one multiple linear regression with at least three predictors (independent variables); (Lab report 4)
   4. test at least one hypothesis using at least one advanced ANOVA (Lab report 5)
7. Discussion: ~25%. Provide an insightful, balanced, comprehensive understanding and interpretation of the results with tangible recommendations for future practice and research. Relate this Discussion back to the content in the Introduction.
8. References: APA style. Core references should be from academic peer-reviewed sources; supplementary references can be from other sources. There is no minimum number of references. You may use any sources, but it is strongly recommended that you concentrate on good quality peer-reviewed articles and references. As a rough guide, aim to make effective use of five references per lab report. Some recommended starting references are provided via eReserve.
9. Appendices: Generally not necessary, but may be used to include additional supplementary information (e.g., SPSS output (optional)) as long as each Appendix item is separately labelled and referenced in APA style from within the main report. Appendix content does not need to be in APA style. Do NOT include a copy of the survey – instead provide a reference to its electronic location.

Word count

Maximum word count is 1,000 words in APA style:

1. Word count = Everything from the start of the Title page through to the end of the References (including any Tables and Figures), but does not include the preceding Cover sheet or proceeding Appendices.
2. Penalty: 5% per 100 words pro rata (i.e., .05% per word over 1,000 words)

Sample write-ups

See sample write-ups for some examples of how to write up exploratory factor analysis, multiple linear regression, and ANOVA.

Specific guidelines

Lab report 1: Data collection and entry

1. Task: Collect, enter, and upload survey based data using a standardised procedure.
2. Lab report 1 differs from Lab reports 2 to 5 in that it is a data collection and data entry exercise.
3. Data collection and data entry is to take place using prescribed guidelines and templates:
   1. Print and collate hard-copy, multi-page surveys
   2. Collect five cases of real survey data
   3. Enter the data into the SPSS .sav template file
   4. Name the file with your student # (e.g., u9163374.sav)
   5. Submit the data via Moodle
   6. Retain the hard copy surveys in order to be able to verify the entered data
4. Marking will be 100% based on how well the exercise is completed as measured by the quality of received data.

This includes how closely the survey administration guidelines are followed and how correctly the data is entered and uploaded.

Lab report 2: Correlation

Task

Describe and graph the correlational relationship between: (a) two variables, at least one of which has a nominal or ordinal level of measurement; and (b) two variables, neither of which have a nominal or ordinal level of measurement.

Marking criteria

In addition to the generic guidelines, this report should include:

1. 5%. Title/Abstract: Provide a succint overview of the study.
   1. Sumarise the study's purpose, method, findings, and implications in less than 150 words.
2. 10%. Introduction: Clearly introduce the topic and the hypotheses:
   1. 5%. Background: Briefly introduce the theoretical concepts and related research, using relevant academic citations
   2. 5%. Research questions: Propose logically-derived research question(s) (which are addressed in the Results)
3. 10%. Method: Clearly explain how the study was conducted
   1. 5%. Participants:
      1. Provide a solid overall picture and description of the sample
      2. Note: It may be appropriate to provide detailed description of specific aspects of the sample in subsequent lab reports; only a general overview is required for this lab report
   2. 5%. Procedure:
      1. Describe the research procedure, with sufficient detail to permit critique of the methodology and full replication.
      2. Include an electronic APA style citation to the survey administration guidelines (but do not provide these in an appendix)
      3. Describe your own survey administration procedures and experiences (e.g., where/when/how did you collect data, response rate, anomalies (i.e., what didn't go according to plan, what could have unwittingly affected the results))
4. 50%. Results: Appropriate tests conducted and reported in APA style
   1. 25%. Correlation 1: Describe and present descriptive statistics and graph(s) for the the correlational relationship between at least two variables, at least one of which has a nominal or ordinal level of measurement.
   2. 25%. Correlation 2: Describe and present descriptive statistics and graph(s) for the the correlational relationship between at least two variables, neither of which have a nominal or ordinal level of measurement.
5. 25%. Discussion: Explain results and implications:
   1. Explain what the results indicate about the research question(s).
   2. Consider the implications of what was found.
   3. Comment on the strengths and weaknesses of the Procedure. How could it be (practically) improved?

General feedback

1. Word count
   1. Many reports were over the word count. However, word count penalty was applied leniently (only for word count > 1000 words from Introduction to Discussion) due to an old link about word count on the coversheet (to 2011 guidelines). The coversheet word count link has been corrected to the 2012 guidelines. Word count penalty will be applied more strictly for Lab reports 3, 4 and 5.
   2. There was considerable discussion about this on the Moodle discussion forum in order to clarify what had happened, including explaining why the sample HD report was over the word count but not penalised, and to address the concerns of those who had followed the word count rules and felt that it was unfair that others were not penalised (effectively others had been allowed greater word count to address the marking criteria). The convener apologised for the error and explained that the decision had been taken towards being lenient in applying the word count penalty due to the error in having links to the 2011 word count guidelines (which were more lenient) on the assignment coversheet rather than the 2012 word count guidelines.
2. APA style
   1. Use of APA style for reporting statistical results often not used (e.g., r and p etc. in italics, leaving spaces before and after = (treat it as a word in a grammatical sentence)
3. Title
   1. A title page was sometimes not included
   2. A running head was often not included (put it in the header)
4. Abstract
   1. Often the results were not clearly summarised
   2. Sometimes correlational results were confused as indicating a causal relationship between variables
5. Introduction
   1. Often too long
   2. Should be focused on succintly reporting on the theoretical and/or research justification for each hypothesis/research question
6. Method
   1. Participants:
      1. Often a simplistic description of the sample was provided
   2. Procedure:
      1. Generally well done
      2. The sampling method was often not described; many incorrectly described it as random sampling
      3. Often the survey admin guidelines were not referenced
      4. Sometimes the researcher's own procedure was not described in enough detail
7. Results
   1. Generally well done
   2. Sometimes the wrong correlational tests were used (e.g., Pearson product-moment correlation instead of a chi-square test of contingencies)
   3. Sometimes bivariate graphs were not presented, and often these were not edited into APA style
   4. Often the direction, strength and statistical significance of the relationship between the variables was not clearly explained
   5. Coefficient of determination was often not reported and discussed
   6. Testing normality: Shapiro-Wilks is an overly sensitive test with a large sample size. Hence in this unit it has been recommended that to use skewness and kurtosis and visual inspection of a histogram to determine whether normality assumptions are met.
8. Discussion
   1. Generally well done
   2. Sometimes causality was assumed - be wary of interpreting correlations as causality
   3. Stronger discussions explained how methodological limitations may have affected the results in the current study; weaker discussions tended to more vague and general about study limitations
9. Formatting
   1. Often multiple line spaces were used instead of page breaks

Lab report 3: Psychometrics

Task

Report on the factor structure of one multi-item, multi-dimensional survey instrument using exploratory factor analysis and reliability analysis.

Marking criteria

In addition to the generic guidelines, this report should include:

1. 5%. Title/Abstract: : Provide a succint overview of the study.
   1. Sumarise the study's purpose, method, findings, and implications in less than 150 words.
2. 10%. Introduction: Clearly introduce the topic and the hypotheses:
   1. 5%. Background: Briefly introduce the topic (student satisfaction or time management) and background literature about its dimensionality (factor structure).
   2. 5%. Research questions: Propose logically-derived research question(s) (which are addressed in the Results) - in its simplest form, the research question is likely to be e.g., "What is the underlying factor structure for university students' satisfaction with their university experience?"
3. 10%. Method: Clearly explain how the study was conducted:
   1. Participants: Not application (do not include)
   2. 10%. Instrumentation
      1. Describe and explain the instrumentation - this should include the entire survey, but focus primarily on the instrumentation used in the current study (i.e., student satisfaction or time management).
   3. Procedure: Not applicable (do not include)
4. 50%. Results: Appropriate tests conducted and reported in APA style:
   1. 50%. Describe and present an EFA which identifies the number of factors, total variance explained, factor loadings and communalities, reliabilities for each factor, and correlations between the factors. Describe initial models but only present the results for the final analysis which is likely to drop several items from the initial step. Explain which items were dropped and why. Item-correlations should be included in an Appendix.
5. 25%. Discussion: Explain results and implications:
   1. Explain what the analysis found out about the research question(s) e.g.,
      1. What do the factors measure and represent?
      2. How 'good' is the factor structure? Are there alternative models that could be considered?
      3. How reliable are each of the factors?
      4. How correlated are the factors?
   2. What can be recommended e.g., how could the measure be improved?

FAQ

Which variables should I analyse?

1. Either the Time Management or the University Student Satisfaction variables

How can I decide on a factor structure?

1. Follow the steps in Tutorial 3 - Exploratory factor analysis and the relevant section in the SPSS textbook

What tables are needed?

1. Table 1 - Factor loadings and communalities
2. Table 2 - Factor names and definitions, with number of items, and Cronbach's alpha
3. Table 3 - Factor correlations (depending on number of factors, this could be reported in text instead).

Is Cronbach's alpha needed for each factor?

1. Yes

How do I calculate the internal consistency for each factor?

1. A common confusion is to calculate Cronbach's alpha for all items rather than separately for each factor
2. Follow the steps in Tutorial 3 - Internal consistency and the relevant section in the SPSS textbook
3. Repeat for each set of items which represent a factor

How do I create composite scores for each factor?

1. See Tutorial 3 - Composite scores

How I calculate the correlations between the factors?

1. Create composite scores for each factor
2. Analyze - Bivariate - enter composite scores

Are descriptive statistics and/or graphs needed for each composite factor score?

1. No

How many decimal places should I report to?

• Generally 2, but this may vary depending on the scale of measurement - what is meaningful/useful without being unnecessarily detailed?

How do I add the correlations to an Appendix?

• See item correlations

General feedback

1. Word count
   1. Many assignments went over the maximum word count and thus received penalties
2. APA style
   1. Formatting of citations was often not correct e.g.,
      1. for three or more authors, after initial citation, use First author surname et al., year
      2. User ampersands (&) inside brackets for citations and use the word "and" when outside brackets
   2. Use Australian spelling e.g., many students are using American spelling for words such as organisation (Australian) / organization (American)
   3. Font size and type (should be Times New Roman 12 pt)
   4. Numbers were often not formatted correctly, e.g., for numbers under 10, report in words, e.g., five surveys
   5. Spaces should be used either side of symbols which replace words (e.g., p < .05 instead of p<.05)
3. Abstract
   1. Generally, well done.
   2. In general, do not include references unless they are particularly pertinent to the study
4. Introduction
   1. Quite often there were no literature review of previous research or theory about the factor structure of the target multi-dimensional construct
   2. Research questions were generally clear and appropriate.
5. Method
   1. Generally well done.
   2. An example item could often have been a helpful addition.
   3. Quite often it was not explained how the items were developed.
6. Results
   1. Generally, well done.
   2. Generally, people reported finding between three and five or six satisfaction, with probably around four being most common.
   3. Some models were presented based on all items, rather than eliminating complex items or items with low primary loadings and/or high cross-loadings
   4. A few models were overzealous in removing items, resulting in models with a small number of items.
   5. Correlations between factors were often not presented.
   6. Item numbers generally don't mean much to an external reader - be more descriptive.
   7. Table captions could sometimes have been explicit e.g., "Correlations Between the Factors" -> "Correlations Between the Four University Satisfaction Factors"
   8. Round results for factor loadings and correlations to two decimal places
   9. Rank items by primary factor loading in factor loading table
7. Discussion
   1. Generally, well done.
8. References
   1. It was rare for references to be formatted in complete APA style, although mostly APA style here was quite good.
9. Appendices
   1. Generally, well done.

Lab report 4: Multiple linear regression

Task: Conduct and report on a multiple linear regression which uses at least three independent variables to predict a dependent variable.

Marking criteria

In addition to the generic guidelines, this report should include:

1. 5%. Title/Abstract: : Provide a succint overview of the study.
   1. Sumarise the study's purpose, method, findings, and implications in less than 150 words.
2. 10%. Introduction:
   1. 5%. Background: Briefly introduce the topic and background literature about the research question and/or hypotheses.
   2. 5%. Research questions/Hypotheses: Propose logically-derived research question(s) and/or hypotheses (which is/are addressed in the Results). Research questions could be in form of "To what extent do A, B, and C predict the variance in Y?). Hypotheses could be in the form of "It is hypothesised that A, B, and C will each be positive linear predictors of Y").
3. 10%. Method:
   1. Participants: N/A
   2. 10%. Materials: Explain the measures used for the variables in the current study, including summarising how any composite scores were derived and their internal consistency.
   3. Procedure: N/A
4. 50%. Results: Describe and present the results of a multiple linear regression analysis involving three of more independent variables. Reporting should summarise any data screening and/or recoding/dummy coding (if not already covered in Materials), type of analysis, assumption testing, correlations between the variables, variance explained, and regression coefficients (including B, Beta, t, p, and sr²). No figures are required. The results should demonstrate a clear understanding of the nature of the linear relationships amongst the variables.
5. 25%. Discussion: Explain what the analysis found out about the research question(s) and/or hypotheses e.g.,
   1. How much variance was explained in DV? Why?
   2. Which variable(s) contribute most/least to understanding the DV? Why?
   3. What were the study's strengths and weaknesses? What could be improved?
   4. What conclusions can be drawn and what are the implications?

FAQ

Should I use items or composite scores?

1. Either
2. If you're creating composite scores without having done an exploratory factor analysis first, then at least conduct reliability analysis and report this as part of the Method
3. Well-derived composite scores (where available/possible) will generally provide for a more valid measure of fuzzy constructs than single-item scores, but this will depend on the specifics of the construct being measured.

Is my R² big enough?

1. Size of R² doesn't matter in terms of meeting the marking criteria. In fact, R² of 0 is acceptable. This is not an exercise in coming up with a large R² , but rather one in which a theoretical model is tested to determine the extent to which a set of IVs predict a DV. Many models explain much less than 30% of variance. Although we might like to explain 30%, the reality is often much less. It might be just as important to know that some IVs don't predict a DV. This also gives you something to consider in the Discussion - how could the model and/or measures be improved?

Are my predictors significant enough or strong enough?

1. The aim is conduct an MLR to examine how well some IVs predict a DV. Thus, it is not necessary to seek only significant predictors. It's more important to have a rationale for a model and then to test and interpret the results, regardless of their significance or size of effect.

What results should be presented in a table?

1. The correlations and regression coefficients are best summarised in a table.
2. For examples, see the MLR sample write-ups e.g., Table 12 from the sample report and Table 2 from the MLR report are good examples of tables which show the correlations and coefficients for a multiple linear regression analysis.

General feedback

Overall, performance was slightly better than on previous reports (67.9%).

1. Word count
   1. Many assignments went over the maximum word count and thus received penalties
2. Formatting
   1. Use page breaks rather than multiple line spaces to separate pages (better layout)
3. APA style
   1. Formatting of citations was much better than previously. There is still some room for improvement - see previous feedback for suggestions.
   2. More reports used Australian spelling - see previous feedback for suggestions.
   3. Numbers were still often not formatted correctly - see previous feedback for suggestions.
   4. Spaces should be used either side of symbols which replace words (e.g., p < .05 instead of p<.05)
4. Title
   1. Some titles lacked sufficient detail e.g., "factors predicting..." is vague - which factors?
5. Abstract
   1. Generally, well done.
   2. The weakest aspect was too information about the design and not enough detail about the results e.g., for each predictor. Explain the significant, strength, and direction of the tested relationships.
   3. Avoid statistics in Abstract unless particularly pertinent.
6. Introduction
   1. Literature reviews are becoming more succint and focused towards research questions - good to see.
   2. Research questions were generally clear and appropriate.
7. Method - Materials
   1. Method sections are becoming more succint with important summary details
   2. Provide a citation to the survey and explain each of the items/composite scores and measurement scales etc. If using composite scores, report the number of items and internal consistencies.
8. Results
   1. Generally, well done.
   2. Assumptions were well tested and described.
   3. Correlations sometimes were not included in a table and summarised.
   4. A table of regression coefficients was generally well presented, with effective description of the overall model.
   5. There was often a lack of sufficient explanation of the results for each of the predictors.
9. Discussion
   1. Generally, well done.
   2. Stronger recommendations are those which are more specific and insightful; vague, unrelated recommendations are not convincing.
10. References
    1. APA style in this section is improving.
    2. Do not include issue numbers for seriated journals (which is most of them).
    3. Journal article titles should be have the first letters capitalised.

Lab report 5: ANOVA

Task: Conduct and report on a Mixed ANOVA (at least 2 x 3 or 3 x 2) or ANCOVA (at least 2 x 2 plus a covariate).

Marking criteria

In addition to the generic guidelines, this report should include:

1. 5%. Title/Abstract: Provide a succint overview of the study.
   1. Sumarise the study's purpose, method, findings, and implications in less than 150 words.
2. 10%. Introduction:
   1. 5%. Background: Briefly introduce the topic and background literature.
   2. 5%. Research questions: Propose logically-derived research question(s) (which are addressed in the Results) - in their simplest form, the research questions are likely to be e.g., "Is there a main affect for A for Y? Is there a main effect for B for Y? Is there an interaction between A and B for Y?" (for Mixed ANOVA). ANCOVA questions would include "... whilst controlling for C" and/or hypotheses for each tested effect.
3. 10%. Method:
   1. Participants: Not applicable
   2. 10%. Materials: Explain the operationalisation of measures used in the current study, including summarising how any composite scores were derived and their internal consistency.
   3. Procedure: Not applicable
4. 50%. Results: Describe and present the results of a Mixed ANOVA (at least a 2 x 3 or 3 x 2 design) or ANCOVA (at least a 2 x 2 design (so that the analysis deals with an interaction) plus a covariate). Reporting should summarise any data screening and/or recoding (if not already covered in Materials), type of analysis, assumption testing, descriptive statistics for the cells and marginal totals, ANOVA results (including F, df, p, eta-squared and/or partial eta-squared, and standardised mean effect sizes (d) with an accompanying figure (e.g., error-bar chart), and follow-up planned contrasts or post-hoc comparisons if necessary. The results should demonstrate a clear understanding of the differences in the DV by the IVs and possible interaction between the IVs, which includes understanding of the significance, direction, and strength of tested effects.
5. 25%. Discussion: Explain what the analysis found out about the research question(s) and/or hypotheses e.g.,
   1. What was the difference in Y means for IV1? Why?
   2. What was the differences in Y means for IV? Why?
   3. What was the interaction between IV1 and IV2 for Y means? Why?
   4. What were the study's strengths and weaknesses? What could be improved?
   5. What conclusions can be drawn and what are the implications?

FAQ

Is a composite overall score needed for a mixed ANOVA?

1. Creating a composite total score of the within-subjects IV is not necessary for a mixed ANOVA analysis, but it is needed for calculating the marginal total descriptive statistics in SPSS.

Which effect sizes should be reported?

1. For the overall ANOVA, eta-squared can be reported (optional). It needs to be calculated by hand from SPSS ANOVA output. It is the equivalent of R2 in MLR, i.e., the % of variance in the DV explained by the IVs
2. For each ANOVA effect (main and interaction), report the partial eta-squared. This is the percentage of variance explained by each effect
3. For pairwise contrasts (i.e., describing the size of mean difference between two groups or two variables), use a standardised mean difference effect size. This can be reported for all contrasts or only those involving post-hoc testing or planned contrasts.

How can adjusted descriptive statistics be obtained for ANCOVA?

1. In SPSS Via Options (in the General linear model/ANOVA dialogue boxes) - for more detail see [1] - this will provide means and standard errors (SEs). Standard deviations (SDs) can be calculated (e.g., for standardised mean effect sizes) by multiplying the SEs by the square root of the sample size. However, adjusted skewness and kurtosis are not provided. Thus, skewness and kurtosis could be provided instead for the unadjusted scores in a table (or appendix). Plots can also be requested via the ANOVA dialogue boxes, which will provide line graphs of the adjusted means. Alternatively, a line graph could be drawn using other software (e.g., a word-processor) based on the adjusted means.

When should post-hoc tests be conducted?

1. When a main effect with three or more levels is significant.

General feedback

Overall, performance was slightly better than on previous reports (67.9%).

1. Word count
   1. This was generally spot on!
2. Formatting
3. APA style
   1. APA style has improved considerably across the reports, although there is still room for improvement in almost every case.
4. Title
   1. Sometimes struggled to succintly provide a clear description of the study
   2. Were all main variables mentioned?
   3. Avoid referring to the specific statistical techniques used; focus on the research question and/or findings
5. Abstract
   1. Did not always describe the method and sufficient detail in results/discussion.
   2. Avoid abbreviations in the Abstract.
   3. Avoid references in the Abstract.
   4. Avoid reporting detailed statistics in the Abstract.
6. Introduction
   1. Generally, succint
   2. Generally, proposed appropriate research questions/hypotheses
   3. Often could have been provided a literature which corresponded more closely to the RQs/hypotheses
7. Method - Materials
   1. Sometimes didn't cite the survey or explain sufficient detail about how the constructs of interest were operationalised
   2. Could often improve by making sure to include (for composite scores), the number of items in each factor, the response scale, and how composite scores were created
8. Results
   1. Generally, well done.
   2. Some reports exhibited a MLR/correlational conceptualisation of the study rather than an ANOVA/mean difference conceptualisation.
   3. Generally, an appropriate table (including M, SD, Skewness and Kurtosis for each cell and marginal totals, with an accompanying figure (e.g., multiple line graph) were provided.
   4. Figures should be presented so that they can be readily interpreted in black and white printing.
9. Discussion
   1. Generally, well done.
   2. A greater depth of understanding can be demonstrated when:
      1. an explanation if provided for how cited limitations may have affected the results
      2. more specific recommendations are made
10. References
    1. Generally, well done.
    2. APA style - do not include issue #s.

Submission

1. Include the downloadable unit’s official coversheet as page 1.
2. Submit electronically as per website instructions.
3. Do not submit hard copies.
4. Late submission will incur a 5% penalty per day (7 days/week), i.e., after 20 days late no marks are available.
5. It is strongly recommended that you keep multiple and regular backups of your lab report, data, syntax, and output files. Computer problems such as hard drive failure will not be accepted as ground for extension.