Questionnaires

Validity and Reliability

Questionnaires serve to measure constructs that can be abstract and not directly observable such as perceptions, attitudes, beliefs, and knowledge. As such, it is critical that survey and questionnaire measures have adequate construct validity. [Page 1799]This construct validity determines that researchers may draw accurate inferences from the characteristics assessed in the measure. The measure must cover the range of concepts that might represent the construct yet exclude those that might represent adjacent but differing constructs. For example, a researcher examining eating disorders who would like to assess body image would use a survey measure that includes items about the participant’s perception of self but not questions pertaining to eating behaviors, although these may be related. In addition, it is important that questionnaires be validated for understanding, in that the respondents are interpreting and answering the questions with the intended meaning (face validity). Construct validity is usually developed by surveying the related concepts, creating candidate items, and small-scale preliminary investigations for understanding and scaling properties (i.e., the distribution of scores across the range). Sensitivity of items can also be of interest. Depending on the theory of scaling (e.g., items represent an underlying dimension without order or degree or are ordered along a dimension), factor analysis or item-response latent-trait analyses may be applied to test dimensionality and item inclusion.

In addition, questionnaire measures should possess discriminant validity, which determines that a measure assesses the construct of interest and not other similar constructs. For example, a questionnaire measure that assesses self-esteem should differentiate levels of positive regard and also be differentiated (i.e., not highly correlated) with a measure of particular preferences for activities (not dependent on esteem level).

Questionnaire measures can be qualitative (i.e., an open-ended answer is provided) or quantitative (e.g., participants indicate a value along a rating scale). Measures are worded the same for all respondents, except for developmental and language variations. In those cases, equivalency psychometric analyses are needed to ensure the tests are measuring the same constructs with the same psychometric features. Alternatively, scores can be normed for different groups and then standardized to permit cross-group comparisons, like the process used with intelligence tests.

Questionnaire measures also need to be reliable or likely to render the same score for the same person under the same conditions. This reliability is usually calculated as internal consistency or the extent to which items intercorrelate within respondents or test–retest reliability (the consistency of scores over time). A theory about the stability of the construct of interest is important for proper interpretation of reliability. That is to say, if a construct is expected to fluctuate over time, this should be taken into consideration when assessing test–retest reliability.

...