To attract and retain top talent in a competitive labor market, federal organizations are adopting skill-based assessment strategies. Here are three ways we're using AI to create and improve them efficiently.
Non-verbal, puzzle-like items
Our team developed an automatic test item generator to rapidly produce measures of deductive reasoning, spatial visualization, and other similar abilities. Previously, item writers spent upwards of two months and over 300 hours to develop roughly 200 valid items. With this new automated tool, puzzles can be generated in seconds, with rule-based checks built in to ensure each puzzle has just one correct answer. Our experts now spend closer to 30 hours reviewing and validating outputs for accuracy, a 90%-time reduction.
Analogies challenge respondents to infer the relationship between two words and apply the same relationship to another pair of words. For instance, the pairings of a) chef and kitchen and b) executive and office both reflect the relationship between a type of worker and their workplace.
It can be labor-intensive to create unique analogies at varying levels of difficulty, all with multiple response options and “distractor” items that seem correct at first glance but are not. However, our team has engineered Gen AI prompts to produce dozens of analogy test items almost instantly. As expected, 60-80% of our outputs for the Army were deemed usable, which provided a significant head start for our experts, who were then able to refocus their energies on hand-picking the highest quality items and improving distractor responses to ensure the final product met our high standards.
Critical thinking scenarios
This method invites respondents to evaluate a given scenario. For instance, they might identify underlying assumptions, or weigh alternate courses of action given a set of conditions. Developing these scenarios requires creativity and a unique blend of expertise from industrial organizational psychologists who know how to construct these items, and industry experts who ensure the scenarios are realistic. This combination of expertise rarely resides in a single person, requiring teamwork and/or significant research to ensure the quality of each item.
To streamline this process, we’re leveraging Gen AI and prompt engineering to produce scenarios relevant to the job and context in which Army officers work. Experts then refine the resulting outputs, tweaking the style, vernacular, and other details as needed. By enlisting the support of AI as a team member, our team has reduced the average number of review cycles for these scenarios from two or three to just one, resulting in considerable time savings.
Reconsidering skills assessment for the future
These are just a few examples of how we’re leveraging automation and AI to efficiently create and improve skill-based workforce assessments that can support hiring, internal mobility, promotion, and/or learning and development, provided the appropriate experts are engaged to ensure reliability and validity.
If your organization has considered leveraging assessments like these in the past, but felt it might not be possible, the emergence of these innovative technologies may help. For more information about how to improve talent outcomes, please explore our additional resources.