Using Xamarin.UITest with Accessible Xamarin.Forms Apps

Millions of people around the world rely on assistive technology to computers and smartphones every day. While making sure your app is usable by as many people as possible seems like a good idea anyway, accessibility is becoming the law in a growing number of circumstances.

What’s the problem?

Xamarin.Forms provides a class of AutomationProperties that map to the native accessibility API on Android, iOS and Windows. These allow you to set the description of user interface elements so that they may be read aloud by a screen reader, a common form of assistive technology that is essential for many people with visual impairments.

Testing is also important, and Xamarin.UITest is a popular solution for automating UI-driven testing scenarios. Unfortunately, both Xamarin.UITest and AutomationProperties on Android use the same native ContentDescription property, which leads to apps that are either accessible or usable with Xamarin.UITest, but not both — and this is a longstanding pain point. Support for accessibility has been improving in Xamarin.Forms, particularly in Xamarin.Forms 5.0, however open issues remain.

Just setting AutomationId can cause accessibility issues with screen readers. We encountered one such issue when we were using Android’s TalkBack screen reader with a Label like the one below:

<Label AutomationId="WelcomeLabel"
       Text="Please set a login and password for this app."/>

When this Label comes into focus, the TalkBack screen reader announces it as “WelcomeLabel”. Only the AutomationId is read by the screen reader – the crucial Text is ignored! A screen reader user would not get to hear the instructions written in the Text property, so it would be difficult for them to understand and interact with this app’s content.

From an accessibility perspective, the fastest solution would be to remove the AutomationId completely and let the screen reader announce the rest of the element’s ContentDescription, including any AutomationProperties.Name or HelpText.

But since automated UI testing is required, developers still need to set the AutomationId to identify specific elements. Here is the other big conflict between UI testing and accessibility – the UI test will fail if it searches specifically for the AutomationId property of an element that also has AutomationProperties.

The sample UI test method below uses AppQuery.Property() to find the Entry element by its AutomationId. However because it also sets AutomationProperties.HelpText, it will fail to find it on Android:

MainPage.Xaml
<Entry AutomationId="UsernameEntryId" Text="{Binding Username}"
       AutomationProperties.IsInAccessibleTree="True"
       AutomationProperties.HelpText="Enter your name"/>

Tests.cs
// UI test fails - no element found
var entry = app.Query(c => c.All().Property("AutomationId",        "UsernameEntryId"));

Given this conflict, how can we balance the need for accessibility and UI testing without having a different build for each purpose or resorting to custom Android renderers?

Support for both UITest and Accessibility

We found a platform-independent way to set the AutomationId for UI tests while keeping it null in the during normal app execution so that screen readers can still read the Text and the AutomationProperties of the element. There are two parts to our solution: 1) a custom XAML markup extension and 2) a hidden button on the app startup page.

The markup extension has a flag that is set to true in the code-behind to signal switching to UI test mode. When this flag is set to true, the extension returns the Id string, otherwise it returns null. Here’s the code for the extension:

public class UITestModeExtension : IMarkupExtension
{
     public string Id { get; set; }
     public static bool IsInUiTestMode { get; set; } = false;    
     
     object IMarkupExtension.ProvideValue(IServiceProvider
      serviceProvider)
     {
        if (IsInUiTestMode)
            return Id;
        return null;
     }
}

Include the local namespace for the project in the XAML file, and then the extension can be used like the example below:

<Entry AutomationId="{ext:UITestMode Id=UsernameEntryId}"
       Text="{Binding Username}"
       AutomationProperties.IsInAccessibleTree="True"
       AutomationProperties.HelpText="Enter your name"/>

On the initial page of the app, have a hidden Button with a clicked event that sets IsInUiTestMode to true and creates a new instance of the page (code below). This will force the XAML elements to get recreated with the AutomationId now set, and all subsequent pages will also have the AutomationId set on their XAML elements.

<Grid RowDefinitions="1" ColumnDefinitions="1,*">
    <Button AutomationId="ToggleTestMode"
            TextColor="Transparent" BackgroundColor="Transparent"
            Clicked="ToggleTestModeButton_Clicked"
            AutomationProperties.IsInAccessibleTree="False"/>
</Grid>

private void ToggleTestModeButton_Clicked(
               object sender, EventArgs e)
{
    UITestModeExtension.IsInUiTestMode =
              !UITestModeExtension.IsInUiTestMode;
    Application.Current.MainPage = new MainPage();
}

Now in the UI test files, you can simulate Tap events on the Button to trigger the event that sets the AutomationId on the elements. After the Button is tapped, you can begin automated UI testing by querying for elements with an AutomationId.

An important thing to note here is to use the AppQuery.Marked() method to search for the element, and not AppQuery.Property() to access the AutomationId directly.

[Test]
public void CanSwitchAutomationIdMode()
{
    // 1: verify that entry is not found yet
    var entry = app.Query(c => c.All().Marked("UsernameEntryId"));
    Assert.AreEqual(entry.Count(), 0);
    // 2: tap the ToggleTestMode button
    app.Tap(c => c.Marked("ToggleTestMode"));
    // 3: wait for ToggleTestMode to reappear
    var button = app.WaitForElement(c =>
                 c.All().Marked("ToggleTestMode"));
    // 4: query for UsernameEntryId again
    var entrySecondQuery = app.Query(c =>
                           c.Marked("UsernameEntryId"));
    // 5: verify that UsernameEntryId is found
    Assert.AreEqual(entrySecondQuery.Count(), 1);

Normally, the code to toggle the button into test mode would be in a test setup method so it wouldn’t have to be repeated on every test.

Note that the “hidden” Button on our XAML page isn’t truly hidden. Setting IsVisible=”False” or otherwise making the Button truly not available in the visual tree means that the button can’t ever be tapped, even programmatically with UI Test. However a 1 pixel by 1 pixel button is good enough to be in the visual tree, available to UI Test, and in practice is not visible or able to be tapped by users.

To test accessibility, try your test device’s screen reader on the page with the Button not clicked (no AutomationId). Even though the page’s Xaml file sets the AutomationId with the custom extension, the screen reader should not read them until the Button is clicked. Keep in mind that the exact behavior of a screen reader varies depending on the platform and how it is used, and it is encouraged that you always perform manual tests for accessibility.

The app we used to demonstrate this solution is available here on GitHub: https://github.com/criticalhittech/UITestWithA11y.

We hope you find this guide useful as you consider accessibility features in your app!