Include a UI testing framework in Project Reunion

[harvinders]

Proposal: Include a testing framework in Project Reunion

Summary

Project reunion is an overarching project to bring different UI stacks (WPF and UWP) using WinUI 3. However, the recommended testing framework (WinAppDriver) doesn't seems to maintained any longer.

Rationale

• WinAppDriver is not maintained and we would like an officially sponsored project for testing needs.
• WinAppDriver relies on selenium protocol, which is geared towards web, this means certain things demands lot of efforts to get it right. Say, we need to click an item in CommandBar. It would require that the script first checks if the item is shown, otherwise click on the overflow icon ... and find if it present in the flyout menu and then click.
• UIAutomation exposes much richer set of APIs then WinAppDriver, a good testing framework would help leverage those APIs.

Scope

Capability	Priority
Ability to run from remote machine, just like what we are able to do with WinAppDriver	Must
Expose richer UIAutomation API capabilities	Must
Conditional retry of actions	Must
Ability to install/uninstall application packages (msix, appx)	Should
Ability to reset application	Must
Conditional retry on the server end, selenium makes it client responsibility	Could

Important Notes

• Ability to find elements using AutomationId as well as text displayed on the UI.
• Ideally you would like to keep retrying till a condition is met or an alternate condition is met or timeout. Not all test framework provide it or make it simple to use. Say, checking for a message until timeout or error message is displayed.
• Resetting the application to first use is also very important, as test cases tend to change the state. Installing and uninstalling application adds significant amount of time to testing, sometime more than running the scenario.
• Working with lists/grid is also challenging, ideally one would like to be able to pick and listview/gridview item from the control and able to perform action on a subcontrol present. ItemsRepeater doesn't even have equivalent of ListViewItem exposed. Though it can be added by the app writer.
• Working with different snap points is also sometimes challenging. Take for example the CommandBar depending on the size of the application some items maybe in the overflow menu. Though one can argue there should be separate test case.
• Picking item from combobox containing lot of items is also challenging. Probably same as scrolling.

Open Questions

[stevewri]

@jevansaks - are you aware of anything like this for WinUI3?