Uncertain yet Rational - Uncertainty as an Evaluation Measure of Rational Privacy Decision-Making in Conversational AI

In today’s connected world, privacy decision-making is crucial for people to maintain control over their personal information and effectively manage their privacy. However, people’s decisions on privacy are likely to be subject to bias and can lead to frustration and regret. Privacy strategies in Conversational AI can aim at debiasing peoples’ choices by drawing from dual-process theory and triggering a more rational thinking process. Previous research on evaluation measures for such strategies has focused on minimizing regret or aligning user behaviour with their attitudes. In this paper, we propose a subjective measure of uncertainty to evaluate the effectiveness of debiasing strategies in a Conversational AI privacy scenario. We investigate two different scales of uncertainty - an adapted privacy uncertainty scale consisting of four subscales and the PANAS-X scale on the affective state of fear. We find that only one of the adapted subscales and the scale on fear showed sufficient reliability and validity results. Moreover, we did not find differences in uncertainty between our tested strategies. Finally, we propose alternative measures to investigate uncertainty and evaluate privacy strategies that promote rational thinking in the future.