Options
2015
Conference Paper
Titel
Evaluation of statistical methods for the evaluation of observer trials for the assessment of the effectiveness of signature measures
Abstract
The statistical methods discussed in this paper are drawn from the area of machine learning or data mining as well as from descriptive statistics. These techniques are discussed with focus on their applicability to the results of observer trials in order to evaluate the effectiveness of signature measures. Signature measures aim at the change of the apparent signature of an object, e.g. a vehicle. So signature measures can be camouflage against infrared sensory, or they can be used for deception reasons. In order to evaluate the effectiveness of signature measures, observer trials provide an efficient method. The department of Signatorics of Fraunhofer IOSB developed a software tool named CARPET (Computer Aided inteRactive Performance Evaluation Tool) for the realization of observer trials. The benefit of this system is the reproducibility and uniformity of trials for every observer. The results from this system consist of marks, that were placed at particular times, as well as computer mouse positions recorded for each human observer. Based on the information gathered from these marks together with the known target object positions the statistical treatment can be done. For the statistics it has to be known to which target object the marks belong. The first problem considered in this paper concentrates on the correct labeling of the marks according to the target objects. The labeling is done using an expectation maximization scheme with the k-means clustering algorithm. The next step involves a second labeling. In this step a linear discriminant is used to decide whether a mark should be considered a hit or miss for every particular target object. After these decisions, a receiver-operating characteristics (ROC) analysis is performed in order to evaluate the detectability of each target object. Furthermore the sample mean and sample covariance formulas are used on the so called hit sets in order to approximate Gaussian distributions for every hit set. These Gaussians facilitate the evaluation of the accuracy and the precision of the hit sets. Accuracy and precision offer information about the quality of the marks set by the observers.