Evidence-based assessment/ROC Party/Ready to ROC

This set of pages provides a practical introduction to Receiver Operating Characteristic (ROC) analyses, including sample code and examples generated with multiple popular statistical programs.

Click Here for Landing Page

HGAPS New for Fall 2022: HGAPS and Psychology Conferences

Click Here for Landing Page

HGAPS is finding new ways to make psychological science conferences more accessible!

Here are examples from APA 2022 and the JCCAP Future Directions Forum. Coming soon... ABCT!
~ More at HGAPS.org ~

Example ROC Party page

Example ROC Party page

ROC sample code

Example ROC sample code (includes R, SPSS, SAS)

ROC Paper examples

The following papers are a selection that use ROC in their analyses, or illustrate application of DLRs to cases and decision-making, or both.

In press

Cohen, J.R., Adams, Z.W., Menon, S.V., Youngstrom, E.A., Bunnell, B.E., Acierno, R., Ruggiero, K.J., & Danielson, C.K. (in press). How should we assess for depression following a natural disaster? An ROC approach to post-disaster assessments in adolescents and adults. Journal of Affective Disorders.
Danielson, C.K. Cohen, J., Adams, Z., Youngstrom, E.A., Soltis, K., Amstadter, A., & Ruggiero, K. (in press). Clinical decision making following disasters: efficient identification of PTSD risk in adolescents. Journal of Abnormal Child Psychology.
Jarrett, M., Van Meter, A.R., Hilton, D., Youngstrom, E.A., & Ollendick, T. (in press). Evidence-based assessment of ADHD in youth using a Receiver Operating Characteristic (ROC) approach. Journal of Clinical Child and Adolescent Psychology.
Jo, B., Findling, R.L., Hastie, T., Youngstrom, E.A., Wang, C.P., Arnold, L.E., Fristad, M.A., Frazier, T.W., Birmaher, B., Gill, M.K., & Horwitz, S.M. (in press). Construction of longitudinal prediction targets using semi-supervised learning. Statistical Methods in Medical Research.
Mesman, E., Youngstrom, E.A., Juliana, N.K., Nolen, W.A., & Hillegers, M.H.J. (in press). Validation of the Seven Up Seven Down (7U7D) in bipolar offspring: screening and prediction of mood disorders. Findings from the Dutch Bipolar Offspring Study. Journal of Affective Disorders.
Ong, M. L., Youngstrom, E. A., **Chua, J. J. X., *Halverson, T. F., Horwitz, S. M., Storfer-Isser, A., . . . Findling, R. L. (2016). Comparing the CASI-4R and the PGBI-10M for differentiating bipolar spectrum disorders from other outpatient diagnoses in youth. Journal of Abnormal Child Psychology. doi:10.1007/s10802-016-0182-4
Ransom, D.M., Burns, A.R., Youngstrom, E.A., Vaughan, C.G., Sady, M.D., & Gioia, G.A. (in press). Applying an evidence-based assessment model to identify students at risk for perceived academic problems following concussion. Journal of the International Neuropsychological Society.
Salcedo, S., Fristad, M., Birmaher, B., Kowatch, R.A., Gadow, K.D., Phillips, M.L., Frazier, T.W., **Chen, Y-L., Arnold, L.E., Horwitz, S.M., Findling, R.L., & The LAMS Team (in press). Diagnostic efficiency of the Child and Adolescent Symptom Inventory (CASI-4R) depressive subscale for youth mood disorders. Journal of Clinical Child and Adolescent Psychology. doi: 10.1080/15374416.2017.1280807.
You, D.S., Youngstrom, E.A., Feeny, N.C., Youngstrom, J.K., & Findling, R.L. (in press). Comparing the diagnostic accuracy of five instruments for detecting posttraumatic stress disorder in youth. Journal of Clinical Child and Adolescent Psychology. doi:10.1080/15374416.2015.1030754

2016

Algorta, G. P., Dodd, A. L., Stringaris, A., & Youngstrom, E. A. (2016). Diagnostic efficiency of the SDQ for parents to identify ADHD in the UK: a ROC analysis. European child & adolescent psychiatry, 25(9), 949–957. https://doi.org/10.1007/s00787-015-0815-0
Youngstrom, E. A., & Van Meter, A. (2016). Empirically supported assessment of children and adolescents. Clinical Psychology: Science and Practice, 23(4), 327–347. https://doi.org/10.1037/h0101738
Frazier TW, Klingemier EW, Beukemann M, Speer L, Markowitz L, Parikh S, Wexberg S, Giuliano K, Schulte E, Delahunty C, Ahuja V, Eng C, Manos MJ, Hardan AY, Youngstrom EA, & Strauss MS (2016). Development of an objective autism risk index using remote eye tracking. Journal of the American Academy of Child and Adolescent Psychiatry, 55, 301-309.
Jenkins, M.M., & Youngstrom, E.A. (2016). A randomized controlled trial of cognitive debiasing improves assessment and treatment selection for pediatric bipolar disorder. Journal of Consulting and Clinical Psychology, 84, 323-333.
Van Meter, A. R., You, D. S., *Halverson, T., Youngstrom, E. A., Birmaher, B., Fristad, M. A., Kowatch, R.A., Storfer-Isser, A., Horwitz, S.M., Frazier, T.W., Arnold, L.E., Findling, R.L., The LAMS Group. (2016). Diagnostic Efficiency of Caregiver Report on the SCARED for Identifying Youth Anxiety Disorders in Outpatient Settings. Journal of Clinical Child & Adolescent Psychology, 1-15. doi:10.1080/15374416.2016.1188698

2015

Youngstrom, E.A., Hameed, A., Mitchell, M., Van Meter, A.R., Freeman, A.J., Perez Algorta, G., White, A.M., Clayton, P., Gelenberg, A., & Meyer, R.E. (2015). Direct comparison of the psychometric properties of multiple interview and patient-rated assessments of suicidal ideation and behavior in an adult psychiatric inpatient sample. Journal of Clinical Psychiatry, 76, 1676-82. doi:10.4088/JCP.14m09353
Youngstrom, E.A., & De Los Reyes, A. (2015). Moving towards cost-effectiveness in using psychophysiological measures in clinical assessment: Validity, decision-making, and adding value. Journal of Clinical Child and Adolescent Psychology, 44, 352-361. doi:10.1080/15374416.2014.913252
Youngstrom, E.A., *Choukas-Bradley, S., *Calhoun, C.D., & Jensen-Doss, A. (2015). Clinical guide to the evidence-based assessment approach to diagnosis and treatment. Cognitive and Behavioral Practice, 22, 20-35. doi: 10.1016/j.cbpra.2013.12.005
Lindhiem, O., Yu, L., Grasso, D.J., Kolko, D.J., & Youngstrom, E.A. (2015). Adapting the Posterior Probability of Diagnosis (PPOD) index to enhance evidence-based screening: An application to ADHD in primary care. Assessment, 22, 198-207. doi: 10.1177/1073191114540748
Prieto, M.L., Youngstrom, E.A., Ozerdem, A., Altinbas, K., Quiroz, D., Aydemir, O., Yalin, N., Geske, J.R., Feeder, S.E., Angst, J., & Frye, M.A. (2015). Different patterns of manic/hypomanic symptoms in depression: A pilot modification of the Hypomania Checklist - 32 to assess mixed depression. Journal of Affective Disorders, 172, 355-360. doi: 10.1016/j.jad.2014.09.047

2014

Youngstrom, E.A. (2014). A primer on Receiver Operating Characteristic analysis and diagnostic efficiency statistics for pediatric psychology: We are ready to ROC. Journal of Pediatric Psychology, 39, 204-221. Special Issue: Quantitative Methodology (Guest Editors: Bryan Karaszia & Kristoffer Berlin); doi: 10.1093/jpepsy/jst062
Frazier, T., Youngstrom, E.A., Fristad, M., Demeter, C., Birmaher, B., Kowatch, R., . . . Findling, R. (2014). Improving clinical prediction of bipolar spectrum disorders in youth. Journal of Clinical Medicine, 3, 218-232. doi: 10.3390/jcm3010218
Lee, H.J., Joo, Y., Youngstrom, E.A., Yum, S.Y., Findling, R.L., & Kim, H.W. (2014). Diagnostic validity and reliability of a Korean version of the Parent and Adolescent General Behavior Inventories. Comprehensive Psychiatry, 55, 1730-1737. doi:10.1016/j.comppsych.2014.05.008
Pendergast, L. L., Youngstrom, E. A., Merkitch, K. G., Moore, K. A., Black, C. L., Abramson, L. Y., & Alloy, L. B. (2014). Differentiating bipolar disorder from unipolar depression and ADHD: the utility of the General Behavior Inventory. Psychological Assessment, 26, 195-206. doi: 10.1037/a0035138
Van Meter, A. R., Youngstrom, E. A., Youngstrom, J. K., Ollendick, T., Demeter, C., & Findling, R. L. (2014). Clinical decision-making about child and adolescent anxiety disorders using the Achenbach System of Empirically Based Assessment. Journal of Clinical Child & Adolescent Psychology.

2013

Youngstrom, E.A. (2013). Future directions in psychological assessment: Combining evidence based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Clinical Child and Adolescent Psychology, 42, 139-159. doi: 10.1080/15374416.2012.736358.
Algorta, G.P., Youngstrom, E.A., Phelps, J., *Jenkins, M.M., Youngstrom, J.K., & Findling, R.L. (2013). An inexpensive family index of risk for mood issues improves identification of pediatric bipolar disorder. Psychological Assessment, 25, 12-22. doi: 10.1037/a0029225

2012

Freeman, A.J., Youngstrom, E.A., Demeter, C., Youngstrom, J.K., & Findling, R.L. (2012). Portability of a screener for pediatric bipolar disorder to a diverse setting. Psychological Assessment, 24, 341-351. doi: 10.1037/a0025617

Jenkins, M.M., Youngstrom, E.A., Youngstrom, J.K., Feeny, N.C. & Findling, R.L. (2012). Generalizability of evidence-based assessment recommendations for pediatric bipolar disorder. Psychological Assessment, 24, 269-281. doi: 10.1037/a0025775

2011

Jenkins, M.M., Youngstrom, E.A., Washburn, J.J., & Youngstrom, J.K. (2011). Evidence-based strategies improve assessment of pediatric bipolar disorder by community practitioners. Professional Psychology: Research and Practice, 42, 121-129. doi: 10.1037/a0022506

2010

Horwitz, S.M., Demeter, C., Pagano, M.E., Youngstrom, E.A., Fristad, M.A., Arnold, L.E., Birmaher, B., Gill, M.K., Axelson, D., Kowatch, R.A. & Findling, R.L. (2010). Identification of a childhood bipolar phenotype-the Longitudinal Assessment of Symptoms of Mania Study: Background, design and initial screening. Journal of Clinical Psychiatry, 71, 1511-1517. doi: 10.4088/JCP.09m05835yel2009
Youngstrom, E.A., & Kendall, P.C. (2009). Psychological science and bipolar disorder. Clinical Psychology: Science and Practice, 16, 93-97. doi: 10.1111/j.1468-2850.2009.01149.x
Youngstrom, E.A., *Freeman, A.J., & *Jenkins, M.M. (2009). The assessment of bipolar disorder in children and adolescents. Child and Adolescent Psychiatric Clinics of North America, 18, 353-390. doi: 10.1016/j.chc.2008.12.002
Meyer, S. E., Carlson, G. A., Youngstrom, E., Ronsaville, D. S., Martinez, P. E., Gold, P. W., Hakak, R., & Radke-Yarrow, M. (2009). Long-term outcomes of youth who manifested the CBCL-Pediatric Bipolar Disorder phenotype during childhood and/or adolescence. Journal of Affective Disorders, 113, 227-235. doi: 10.1016/j.jad.2008.05.024

2008

Youngstrom, E.A., *Frazier, T.W., Demeter, C., Calabrese, J.R., & Findling, R.L. (2008). Developing a ten-item mania scale from the Parent General Behavior Inventory for children and adolescents. Journal of Clinical Psychiatry, 69, 831-839. doi:10.4088/JCP.v69n0517
Youngstrom, E.A., *Greene, J., & *Joseph, M. (2008). Comparing the psychometric properties of multiple teacher report instruments as predictors of bipolar disorder in children and adolescents. Journal of Clinical Psychology, 64, 382-401. doi: 10.1002/jclp.20462

Henry, D., Pavuluri, M., Youngstrom, E.A., & Birmaher, B. (2008). Accuracy of brief and full forms of a mania rating scale. Journal of Clinical Psychology, 64, 368-381. doi: 10.1002/jclp.20464
Meyers, O.I., & Youngstrom, E.A. (2008). A Parent General Behavior Inventory subscale to measure sleep disturbance in pediatric bipolar disorder. Journal of Clinical Psychiatry, 69, 840-843. doi: 10.1016/j.jad.20009.07.020

2007

Patel, N.C., Patrick, D.M., Youngstrom, E.A., Strakowski, S.M., & DelBello, M.P. (2007). Response and remission in adolescent mania: Signal detection analyses of the Young Mania Rating Scale. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 628-635. doi: 10.1097/chi.0b013e3180335ae4

2006

Youngstrom, E.A., *Meyers, O.I., Youngstrom, J.K., Calabrese, J.R., & Findling, R.L. (2006). Comparing the effects of sampling designs on the diagnostic accuracy of eight promising screening algorithms for pediatric bipolar disorder. Biological Psychiatry, 60, 1013-1019. doi: 10.1016/j.biopsych.2006.06.023.

Frazier, T.W., & Youngstrom, E.A. (2006). Evidence based assessment of attention-deficit/hyperactivity disorder: Using multiple sources of information. Journal of the American Academy of Child and Adolescent Psychiatry, 45, 614-620. doi: 10.1097/01.chi.0000196597.09103.25

2005

Youngstrom, E. A., *Meyers, O. I., Demeter, C., Kogos Youngstrom, J., Morello, L., Piiparinen, R., Feeny, N. C., Findling, R. L., & Calabrese, J. R. (2005). Comparing diagnostic checklists for pediatric bipolar disorder in academic and community mental health settings. Bipolar Disorders, 7, 507-517. doi: j.1399-5618.2005.00269.x

Youngstrom, E. A., Findling, R. L., Youngstrom, J. K., & Calabrese, J. R., (2005). Towards an evidence-based assessment of pediatric bipolar disorder. Journal of Clinical Child and Adolescent Psychology, 34, 433-448. Special Issue: Evidence-Based Assessment. doi: 10.1207/s15374424jccp3403_4
Youngstrom, E. A., & *Duax, J. (2005). Evidence Based Assessment of Pediatric Bipolar Disorder, Part 1: Base Rate and Family History. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 712-717. doi: 10.1097/01.chi.0000162581.87710.bd

Youngstrom, E. A., & Kogos Youngstrom, J. (2005). Evidence Based Assessment of Pediatric Bipolar Disorder, Part 2: Incorporating Information from Behavior Checklists. Journal of the American Academy of Child and Adolescent Psychiatry, 44(8), 823-828. doi: 10.1097/01.chi.0000164589.10200.a4

2004

Youngstrom, E. A., Findling, R. L., Calabrese, J. R., Gracious, B. L., Demeter, C., DelPorto Bedoya, D., & *Price, M.E. (2004). Comparing the diagnostic accuracy of six potential screening instruments for bipolar disorder in youths aged 5 to 17 years. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 847-858. doi: 10.1097/01.chi.0000125091.35109.1e

2003

Danielson, C. K., Youngstrom, E. A., Findling, R. L., & Calabrese, J. R. (2003). Discriminative validity of the General Behavior Inventory using youth report. Journal of Abnormal Child Psychology, 31, 29-39. doi: 10.1023/A:1021717231272

2002

Findling, R. L., Youngstrom, E. A., DelPorto, D., Papish-David, R., Townsend, L., *Danielson, C. K., & Calabrese, J. R. (2002). Clinical decision-making using the General Behavior Inventory. Bipolar Disorders, 4, 34-42. doi: 10.1034/j.1399-5618.2002.40102.

2001

Youngstrom, E.A., Findling, R.L., *Danielson, C.K., & Calabrese, J.R. (2001). Discriminative validity of parent report of hypomanic and depressive symptoms on the General Behavior Inventory. Psychological Assessment, 13, 267-276. doi: 10.1037//1040-3590.13.2.267