Dr. rer. nat.

Cheng, Chih-Hong

Filters

3 2
Reset filters

Settings
Now showing 1 - 5 of 5
  • Publication
    Statistical Property Testing for Generative Models
    ( 2023)
    Seferis, Emmanouil
    ;
    Burton, Simon
    ;
    Cheng, Chih-Hong
    Generative models that produce images, text, or other types of data are recently be equipped with more powerful capabilities. Nevertheless, in some use cases of the generated data (e.g., using it for model training), one must ensure that the synthetic data points satisfy some properties that make them suitable for the intended use. Towards this goal, we present a simple framework to statistically check if the data produced by a generative model satisfy some property with a given confidence level. We apply our methodology to standard image and text-to-image generative models.
  • Publication
    Statistical Guarantees for Safe 2D Object Detection Post-processing
    ( 2023)
    Seferis, Emmanouil
    ;
    Burton, Simon
    ;
    Kollias, Stefanos
    ;
    Cheng, Chih-Hong
    Safe and reliable object detection is essential for safetycritical applications of machine learning, such as autonomous driving. However, standard object detection methods cannot guarantee their performance during operation. In this work, we leverage conformal prediction in order to provide statistical guarantees for back-box object detection models. Extending prior work, we present a postprocessing methodology that can cover the entire object detection problem (localization, classification, false negatives, detection in videos, etc.), while offering sound safety guarantees on its error rates. We apply our method on state-of-the-art 2D object detection models and measure its efficacy in practice. Moreover, we investigate what happens as the acceptable error rates are pushed towards high safety levels. Overall, the presented methodology offers a practical approach towards safety-aware object detection, and we hope it can pave the way for further research in this area.
  • Publication
    Can Conformal Prediction Obtain Meaningful Safety Guarantees for ML Models?
    ( 2023)
    Seferis, Emmanouil
    ;
    Burton, Simon
    ;
    Cheng, Chih-Hong
    Conformal Prediction (CP) has been recently proposed as a methodology to calibrate the predictions of Machine Learning (ML) models so that they can output rigorous quantification of their uncertainties. For example, one can calibrate the predictions of an ML model into prediction sets, that guarantee to cover the ground truth class with a probability larger than a specified threshold. In this paper, we study whether CP can provide strong statistical guarantees that would be required in safety-critical applications. Our evaluation on the ImageNet demonstrates that using CP over state-of-the-art models fails to deliver the required guarantees. We corroborate our results by deriving a simple connection between the CP prediction sets and top-k accuracy.
  • Publication
    Prioritizing Corners in OoD Detectors via Symbolic String Manipulation
    ( 2022-10)
    Cheng, Chih-Hong
    ;
    Changshun, Wu
    ;
    Seferis, Emmanouil
    ;
    Bensalem, Saddek
    For safety assurance of deep neural networks (DNNs), out-of-distribution (OoD) monitoring techniques are essential as they filter spurious input that is distant from the training dataset. This paper studies the problem of systematically testing OoD monitors to avoid cases where an input data point is tested as in-distribution by the monitor, but the DNN produces spurious output predictions. We consider the definition of "in-distribution" characterized in the feature space by a union of hyperrectangles learned from the training dataset. Thus the testing is reduced to finding corners in hyperrectangles distant from the available training data in the feature space. Concretely, we encode the abstract location of every data point as a finite-length binary string, and the union of all binary strings is stored compactly using binary decision diagrams (BDDs). We demonstrate how to use BDDs to symbolically extract corners distant from all data points within the training set. Apart from test case generation, we explain how to use the proposed corners to fine-tune the DNN to ensure that it does not predict overly confidently. The result is evaluated over examples such as number and traffic sign recognition.
  • Publication
    Formally Compensating Performance Limitations for Imprecise 2D Object Detection
    ( 2022-08-25)
    Schuster, Tobias
    ;
    Seferis, Emmanouil
    ;
    Burton, Simon
    ;
    Cheng, Chih-Hong
    In this paper, we consider the imperfection within machine learning-based 2D object detection and its impact on safety. We address a special sub-type of performance limitations related to the misalignment of bounding-box predictions to the ground truth: the prediction bounding box cannot be perfectly aligned with the ground truth. We formally prove the minimum required bounding box enlargement factor to cover the ground truth. We then demonstrate that this factor can be mathematically adjusted to a smaller value, provided that the motion planner uses a fixed-length buffer in making its decisions. Finally, observing the difference between an empirically measured enlargement factor and our formally derived worst-case enlargement factor offers an interesting connection between quantitative evidence (demonstrated by statistics) and qualitative evidence (demonstrated by worst-case analysis) when arguing safety-relevant properties of machine learning functions.