System Design and Usability Evaluation of an LLM-based Voice Assistant for Controlling Digital Environments

Abstract: Recent advances in artificial intelligence have significantly improved computers’ ability to understand natural language, enabling more intuitive and natural interactions. In this context, this paper introduces a local-first, speaker-aware voice assistant for controlling digital environments that integrates speech recognition, speaker identification, large language model reasoning, automation across digital environments, and text-to-speech within a modular multi-process architecture. To support safe interaction, the voice assistant incorporates a risk-aware validation layer that classifies commands according to their potential impact and requires explicit user confirmation before executing potentially harmful actions. We evaluated the proposed voice assistant through a pilot usability study involving seven participants performing representative desktop tasks. Preliminary results indicate positive usability, a favorable user experience, and a 95.5% overall tool invocation success rate, suggesting that local-first voice assistants can provide an effective, accessible, and privacypreserving approach to digital interaction. Based on these findings, we discuss future application opportunities in accessible computing and assistive technology.

Authors: Alexandru Gheorghiu, Laura-Bianca Bilius.

Conference: ICUSI ’26 – Proceedings of ICUSI ’26, the 2026 International Conference on User-System Interaction, 133-139.

Link: https://rochi.utcluj.ro/icusi/proceedings/articles/3/25.pdf