Monday, 18 July 2016, 09:00 - 12:30

James R. Lewis (short bio)
IBM Corporation, United States

Objectives:

Standardized usability questionnaires are questionnaires designed for the assessment of perceived usability, typically with a specific set of questions presented in a specified order using a specified format with specific rules for producing scores based on the answers of respondents. For usability testing, standardized questionnaires are available for assessment of a product at the end of a study (post-study – for example, QUIS, SUMI, PSSUQ, SUS, and, most recently, UMUX and UMUX-LITE) and after each task in a study (post-task – for example, ASQ, Expectation Ratings, SEQ, SMEQ, and Usability Magnitude Estimation). Standardized questionnaires are also available for the general assessment of website usability (for example, WAMMI and SUPR-Q).

All of these questionnaires have undergone psychometric qualification, including assessment of reliability, validity, and sensitivity, making them valuable tools for usability practitioners. The purpose of this course is to provide an introduction to standardized usability measurement, an inventory of currently available instruments (including their psychometric properties) for after-task and after-study measurement, and discussion of recent research in the important characteristics of standardized usability questionnaires (including their interrelationships and statistical connections to other metrics).

At the end of the course, attendees will know about the key properties and uses for the currently available standardized usability questionnaires.

Content and Benefits:

The course starts with coverage of the basic psychometric properties of standardized measurement – reliability, validity, and sensitivity. The next topic is the inventory of post-study questionnaires (such as the QUIS, SUMI, PSSUQ, SUS, and, most recently, UMUX and UMUX-LITE) – descriptions of the instruments, summaries of their known psychometric properties, and key recent research findings. The SUS will receive special attention due to the relatively large amount of normative and usage data that has become available for it in the past five years. Following that we will cover the available after-task questions/questionnaires (such as ASQ, Expectation Ratings, SEQ, SMEQ, and Usability Magnitude Estimation), with descriptions, summaries, and recent research findings. We will end with a discussion of other standardized usability questionnaires of interest to the HCI community (e.g., some specifically for website evaluation and some key questionnaires from the marketing research literature, such as the TAM and Net Promoter Score).

1. Introduction to standardized usability measurement
   1. What is a standardized questionnaire?
   2. Advantages of standardized usability questionnaires
   3. What standardized usability questionnaires are available?
   4. Assessing the quality of standardized questionnaires
2. Post-study questionnaires
   1. QUIS (Questionnaire for User Interaction Satisfaction)
   2. SUMI (Software Usability Measurement Inventory)
   3. PSSUQ (Post-Study System Usability Questionnaire)
   4. SUS (System Usability Scale)
   5. UMUX and UMUX-LITE (Usability Metric for User Experience)
   6. Experimental comparisons of post-study questionnaires
3. Post-task questionnaires
   1. ASQ (After-Scenario Questionnaire)
   2. SEQ (Single Ease Question)
   3. SMEQ (Subjective Mental Effort Questionnaire)
   4. ER (Expectation Ratings)
   5. UME (Usability Magnitude Estimation)
   6. Experimental comparisons of post-task questionnaires
4. Questionnaires for assessing the perceived usability of websites
   1. WAMMI (Website Analysis and Measurement Inventory)
   2. SUPR-Q (Standardized Universal Percentile Rank Questionnaire)
   3. Other questionnaires for assessing websites
5. Other questionnaires of interest
   1. CSUQ (Computer System Usability Questionnaire)
   2. USE (Usefulness, Satisfaction, and Ease of Use)
   3. HQ (Hedonic Quality)
   4. EMO (Emotional Metric Outcomes)
   5. ACSI (American Customer Satisfaction Index)
   6. NPS (Net Promoter Score)
   7. CxPi (Forrester Customer Experience Index)
   8. TAM (Technology Acceptance Model)
6. Wrapping up

Target Audience:

The course will likely be of interest to a wide variety of attendees, but will be especially useful to those usability practitioners and HCI researchers who currently use or plan to use standardized usability questionnaires.

Bio Sketch of Presenter:

Dr. James R. (Jim) Lewis, Ph.D., CHFP

James R. (Jim) Lewis graduated with an M.A. in Engineering Psychology in 1982 from New Mexico State University, and received his Ph.D. in Experimental Psychology (Psycholinguistics) from Florida Atlantic University in 1996. He has worked as a human factors engineer and usability practitioner at IBM since 1981. He has published influential research on the measurement of usability satisfaction, use of confidence intervals, and sample size estimation for usability studies. He is on the editorial board of the International Journal of Human-Computer Interaction and the Journal of Usability Studies, and wrote the chapter on usability testing for the 3rd and 4th editions of the Handbook of Human Factors and Ergonomics (2006/2012). He is the author of "Practical Speech User Interface Design" (2011) and co-author (with Jeff Sauro) of "Quantifying the User Experience" (2012). He is a BCPE Certified Human Factors Professional, an IBM Master Inventor, a member of UXPA, HFES, ACM SIGCHI, and past-president of AVIxD.