FRAUD DETECTION (COPY)
In order to strengthen the methodologies for detecting copying at the State standardized tests, Quantil supported ICFES (the government institution in charge of standardized testing in Colombia) in the development and implementation of statistic techniques, based on the state of the art implemented in other countries. This work happened in two phases; one in 2009 and the other in 2010.
During the first phase, we developed an index for the detection of possible causes of copying in multiple choice tests (this was done for two tests, Saber 11 and Docentes). The scheme started from the construction of the kappa index, widely used in the literature, by authors such as Sotaridona & Meijer, Angoff and Belleza & Belleza; our main contribution was the use of a re-codification technique of the answers that made the massive calculation of the index easier. Basically, for each pair of subjects that presents an exam in the same room at the same time, distinguishing one individual as the source and the other as the subject of the copied evaluation, we calculate the probability of both of them presenting the observed responses, assuming an invariant distribution through questions.
The results were promising and posed opportunities for improvement, which were approached during the second stage. Specifically, we extended the possibility of changing the joint distribution of possible answers according to the question in the exam, and the treatment of the individuals was strengthened, modeling the differences in their abilities. The index constructed in this stage was based on the omega index, also described in the literature, and improved the Type I error (that is, the error of accusing an innocent of copying) against the kappa index, maintaining a similar Type II error (that is, the error of not detecting those guilty of copying).
Several functionalities were implemented with the index, such as the calculation of the proportion of individuals suspected of copying, of couples ordered suspected of copying, of rooms where it is suspected that copying may have occurred, and of conference rooms where it is suspected that massive copying has occurred.
These models were implemented by ICFES. Subsequently, the work was published by three members quantileros in the Journal of Educational and Behavioral Statistics in August 2015, under the title "On the Optimality of Answer-Copying Indices: Theory and Practice.” Click to read the article here.