Job Knowledge Testing

How is Job Knowledge Measured?

Although job knowledge is sometimes assessed using ratings (e.g., made by interviewers or supervisors), it is typically measured more directly and objectively with multiple-choice tests. Such tests are developed to [Page 400]be content valid (i.e., to cover knowledge areas proportionately to their importance to the job as determined through job analysis). Many strategies can help ensure the quality of such tests. For example, a test blueprint (based on a job analysis) is developed to specify test content. The blueprint reflects the appropriate weighting of knowledge areas. Item-writing guidelines improve the readability and clarity of test items and help prevent “test-wise” examinees from performing inappropriately well on the test. It is also good practice to develop test questions that go beyond simple recall and definitions, instead requiring some amount of analysis or reasoning to answer the question. Some test developers use Bloom's taxonomy as a framework to accomplish this. Another strategy is to use visual aids (e.g., illustrations, photos, graphics) to make the questions look more job-relevant and to limit the degree to which test scores depend on reading ability. Job experts also should review items for accuracy and collect judgments about the relevance and importance of each item to help document the content validity of the test.

Developers are increasingly using item formats other than traditional multiple choice because they can be easily administered and scored by computer. Such formats include multiple-response (e.g., check all that apply), matching, drag-and-drop, and ranking. Some of these formats efficiently cover more content than do traditional item formats, and varying the formats can make the test more engaging for examinees. It is important, however, to consider how to combine scores from different types of items so that the resulting total test score appropriately weights them. For example, how do you combine the score on a five-part matching item (in which examinees may be given partial credit for getting some, but not all, parts right) with the scores from several multiple-choice items (scored one point each) so that the reliability and validity of the total score are maximized? The answer might vary depending on the primary testing goal (e.g., maximizing content validity or correlations with other measures).

A job knowledge test can be developed, scored, and evaluated using classical test theory (CTT) and item response theory (IRT) strategies. Of these, CTT strategies have the advantage of being particularly useful for providing diagnostic information about items (e.g., percentage of examinees selecting each response option and option–total score correlations) that can be used to improve them through rewriting. Because they provide a common underlying metric, IRT strategies are particularly useful if the test uses several item formats or if multiple forms of the test are required, but they require larger sample sizes to yield reliable information. If sample sizes permit, it is good practice to use both types of analytic strategies.

...