Monday, 18 July 2016, 14:00 - 17:30

James R. Lewis (short bio)
IBM Software Group, United States

Objectives:

The purpose of this tutorial is to provide practical information about using statistics in user research, focusing on confidence interval and sample size estimation for a variety of situations and types of measurements.

Content and Benefits:

A test of significance only tells you if an experimental effect probably is or is not equal to 0. For most practical user research, you need to know more than that. The tutorial will begin with a brief review of the most common ways to measure usability and fundamental concepts of statistics. It will then cover the procedures of confidence interval and sample size estimation for a subset of typical usability/user research metrics (completion rates, completion times, satisfaction) and situations (estimating values, comparison with benchmarks, comparing two interfaces), and will also address two other important topics in user research: correlation and simple linear regression.

1. Introduction: Brief review of the most common ways to measure usability with lead into fundamental concepts of statistics and terms
   1. Statistical concepts, terms and misconceptions
   2. Understanding the role of chance in interpreting data
2. Objective: Clearly understand the capabilities and limits of small sample usability data through the use of confidence intervals
   1. Interval and point estimation
   2. Confidence interval around a completion rate
   3. Confidence interval for completion times
   4. Confidence interval for correlations
   5. Confidence intervals for simple linear regression
3. Sample Size Estimation (Formative and Summative)
   1. Do you only need to test with five users?
   2. Sample size and precision
   3. Sample size for comparisons
   4. Sample size for correlations
   5. Sample size for simple linear regression
4. Wrapping up

Target Audience:

The tutorial will be of particular value to practitioners and researchers with an interest in quantitative usability tests and other types of user research. Participants should be familiar with the process of conducting usability tests as well as basic descriptive statistics such as the mean, median and standard deviation. Although not mandatory, attendees should plan to bring laptops (Mac or PC) to get the most out of the planned exercises.

Bio Sketch of Presenter:

Dr. James R. (Jim) Lewis, Ph.D., CHFP

James R. (Jim) Lewis graduated with an M.A. in Engineering Psychology in 1982 from New Mexico State University, and received his Ph.D. in Experimental Psychology (Psycholinguistics) from Florida Atlantic University in 1996. He has worked as a human factors engineer and usability practitioner at IBM since 1981. He has published influential research on the measurement of usability satisfaction, use of confidence intervals, and sample size estimation for usability studies. He is on the editorial board of the International Journal of Human-Computer Interaction and the Journal of Usability Studies, and wrote the chapter on usability testing for the 3rd and 4th editions of the Handbook of Human Factors and Ergonomics (2006/2012). He is the author of "Practical Speech User Interface Design" (2011) and co-author (with Jeff Sauro) of "Quantifying the User Experience" (2012). He is a BCPE Certified Human Factors Professional, an IBM Master Inventor, a member of UXPA, HFES, ACM SIGCHI, and past-president of AVIxD.