We can do everything we can to create the perfect survey and it can still fail. This is why pretesting is critical. Unfortunately, pretesting can be costly—both in terms of time and money. However, if you want to ensure the data you collect is reliable and valid, then this is a critical step in the survey process. There are three typical methods for pretesting: using a panel of experts, pilot testing, and cognitive interviews.

Panel of experts

One strategy for pretesting surveys is to convent a panel of experts on the topic to assess the face validity of the survey. This is useful to ensure you are measuring the construct of interest and not some other construct, and that you are measuring the entirety of the construct of interest. Typically, what is considered in this sort of pretesting are researchers with tons of knowledge and years of experience in the specific topic. However, in evaluation, the panel of experts could also be your evaluation advisory board or program staff, particularly when questions are geared towards the program itself.

Pilot testing

A panel of experts does not guarantee that youth will still understand and respond to the survey appropriately, though. Pilot testing can be one method for bringing youth voice into pretesting of the survey instrument. This involves giving the survey to a small sample of participants to collect data on the survey and do preliminary analyses to ensure the survey is being responded to in the intended manner. You can test for things like (a) Cronbach’s alpha for internal consistency, which typically increases as age/grade increases, (b) item statistics like difficulty (i.e., percentage of participants who got the item “correct” or answered a certain way) and discrimination (i.e., the item-total correlation), and (c) factor analysis to determine if the factor structure is as intended and whether the factor structure holds across multiple age groups[1]. You can also provide space for comments throughout the survey (Survey Gizmo has this feature built in, which is quite useful) or at the end of the survey.

Cognitive interviews

The most thorough method for pretesting your survey is to conduct cognitive interviews with your survey and participants. This involves sitting with one or a few participants and asking them questions to determine whether participants understand the question in the way intended. You can either approach this process through “think-alouds”, where you ask participants to “think aloud” or verbalize their thought processes as they answer survey questions, or through verbal probing, where you ask participants specific questions or “probes” to gain more information about their thought process on the question.[1] However, you may not have the sample size to assess the factor structure or to use a multigroup CFA to test the factor structure across age groups.