Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

Syn: Synthetic Dataset for Training UI Element Detector from Lo-Fi Sketches

: Pandian, Vinoth; Pandian, Sermuga; Suleri, Sarah; Jarke, Matthias


Association for Computing Machinery -ACM-; Association for Computing Machinery -ACM-, Special Interest Group on Computer and Human Interaction -SIGCHI-:
IUI 2020, 25th International Conference on Intelligent User Interfaces. Proceedings : March 17-20, 2020, Cagliari Italy
New York: ACM, 2020
ISBN: 978-1-4503-7513-9
International Conference on Intelligent User Interfaces (IUI) <25, 2020, Cagliari>
Conference Paper
Fraunhofer FIT ()

User Interface design is an iterative process that progresses through low-, medium-, and high-fidelity prototypes. A few research projects use deep learning to automate this process by transforming low fidelity (lo-fi) sketches into front-end code. However, these research projects lack a large scale dataset of lo-fi sketches to train detection models. As a solution, we created Syn, a synthetic dataset containing 125,000 lo-fi sketches. These lo-fi sketches were synthetically generated using our UISketch dataset containing 5,917 sketches of 19 UI elements drawn by 350 participants. To realize the usage of Syn, we used it to train a UI element detector, Meta-Morph. It detects UI elements from a lo-fi sketch with 84.9% mAP and 72.7% AR. This work aims to support future research on UI element sketch detection and automating prototype fidelity transformation.