In some cases, like XAML, most of the heavy lifting is done by the framework, greatly reducing the work required to support Narrator and VES.įor more info on UI Automation see UI Automation Fundamentals. As the UI changes in the app, VES will react to UIA update events and re-parse the updated UI Automation tree to find all the actionable items, using this information to build a speech recognition grammar.Īll UWP apps have access to the UI Automation framework and can expose information about the UI independent of which graphics framework they are built upon (XAML, DirectX/Direct3D, Xamarin, etc.). UI Automation enables programmatic access to user interface elements, including the name of the control, its type and what control patterns it implements. This is the same infrastructure already being used by the Narrator feature on Windows platforms. VES is a UI Automation client and relies on information exposed by the app through its UI Automation providers. User: "Hey Cortana, make a selection" or "Hey Cortana, select" Later on, a new active listening session can be started with: To exit active listening mode, user tells Xbox to stop listening: Now that it's clear what to say, the app can be launched: User asks for voice tip labels to be displayed: User sees the box art for the app they are interested in but forgot the name. User can now switch to view apps and scroll through the app list: User is left in Active Listening Mode (ALM), meaning the console is now listening for the user to invoke a control that’s visible on the screen, without needing to say, “Hey Cortana” each time. User: "Hey Cortana, open My Games and Apps" User turns on the Xbox console and wants to browse through their apps to find something of interest: The following is an overview of what a user would experience when using VES on Xbox, and it should help set the context before diving into the details of how VES works. To achieve this, it leverages the Microsoft Speech Platform and the UI Automation (UIA) framework. VES strives to provide a common end-to-end see-it-say-it experience on all Windows Shells and devices, with minimum effort required from app developers. Voice Enabled Shell (VES) is an extension to the Windows Speech Platform that enables a first-class speech experience inside apps, allowing users to use speech for invoking on-screen controls and to insert text via dictation.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |