Applying force fields to black-box GUIs using computer vision

While new input modalities emerge, many established applications have not been adjusted to these new possibilities of interaction. Assisting technologies like force fields have been shown to enhance interaction, but rely on context information that is not provided by existing applications. We present a computer vision based approach that can automatically locate user interface elements to apply assisting technologies to any application without modification. In a user study we show that user interface elements can be detected reliably and allows for automatic placement of force fields that improve interaction in both speed and accuracy. Our approach makes new interaction modalities like pointing gestures usable with existing applications without need for modification or adaptation.