A Context-aware Conversational Interface for Controlling Internet-of-Things Devices

This paper presents a demonstration of a context-aware conversational interface for interacting with multiple IoT devices using multi-turn natural language commands. Our system facilitates device discovery and identification through a Knowledge Graph (KG) and addresses the interoperability issue by using the Web of Things (WoT) specifications as an intermediate abstraction layer. The proposed system comprises four major components: a context-aware multi-turn dialogue interface, a device discovery and identification module using a domain-specific KG, a text-to-code module to parse natural language commands into an executable code format, and a customized code execution engine. We demonstrate our system using two Philips Hue smart lamps and one Elgato Stream Deck controller. The Philips Hue smart lamp device can be controlled standalone via natural language, allowing users to adjust the power state, the color, and the brightness. Also, the lamp can be integrated with the Elgato Stream Deck Controller to enable user-defined automation rules following the Trigger Action programming (TAP) paradigm.