Identification of Sample Comparability Issues During the Iterative Design of Game-Based Cognitive Assessments


Imbellus develops cognitive assessments designed to evaluate the skills and abilities of its users within the context of game-based simulations. This paper describes work from our ongoing collaboration with a best-in-class management consulting firm, McKinsey & Company, to build a simulation-based assessment that evaluates applicants’ cognitive skills and abilities. A major challenge inherent in producing a game-based cognitive assessment in an industry setting is to balance the need for iterative design improvements and the need for validation with large samples. The work presented here describes a four-step process for assessing the impact of a given design change on the comparability of pre-change and post-change samples. We also discuss implications and future work.

Proceedings of the 9th International Conference Learning Analytics & Knowledge Conference