Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

UISketch: A Large-Scale Dataset of UI Element Sketches

: Pandian, Vinoth; Suleri, Sarah; Jarke, Matthias


Kitamura, Y. ; Association for Computing Machinery -ACM-; Association for Computing Machinery -ACM-, Special Interest Group on Computer and Human Interaction -SIGCHI-:
CHI 2021, Conference on Human Factors in Computing Systems. Proceedings : May 8-13, 2021, Online Virtual Conference, (originally Yokohama, Japan)
New York: ACM, 2021
ISBN: 978-1-4503-8096-6
Art. 319, 14 S.
Conference on Human Factors in Computing Systems (CHI) <2021, Online>
Fraunhofer FIT ()

This paper contributes the first large-scale dataset of 17,979 hand-drawn sketches of 21 UI element categories collected from 967 participants, including UI/UX designers, front-end developers, HCI, and CS grad students, from 10 different countries. We performed a perceptual study with this dataset and found out that UI/UX designers can recognize the UI element sketches with ~96% accuracy. To compare human performance against computational recognition methods, we trained the state-of-the-art DNN-based image classification models to recognize the UI elements sketches. This study revealed that the ResNet-152 model outperforms other classification networks and detects unknown UI element sketches with 91.77% accuracy (chance is 4.76%). We have open-sourced the entire dataset of UI element sketches to the community intending to pave the way for further research in utilizing AI to assist the conversion of lo-fi UI sketches to higher fidelities.