Question: What are the plans for supporting testing of projects using WinUI3?

[mrlacey]

I'm interested to know the broad plans for testing support of apps built using WinUI3. This doesn't appear to have been documented or discussed publicly previously.

Specific questions to help prompt a broad answer:

• Will there be a test project option available specifically tailored to WinUI3 projects? (Currently an MSTest project--targetting either .NET Framework or .NET Core--cannot reference a WinUI class library)
• Will it be possible to create Mocks for all the WinUI APIs?
• Will it be possible to generate Fakes for all the WinUI APIs?
• Will any specific guidance, support, or tooling, be provided to help with UI testing of WinUI3 projects?
• Are there any other details of what can be expected and associated timelines?

[mrlacey]

X-Ref: microsoft/WindowsAppSDK#382

[mrlacey]

X-Ref: microsoft/testfx#708

[mrlacey]

Some more background:

• Testing of applications is often seen by developers as an afterthought and not a priority. (I realize this is a broad generalization, if you're reading this and thinking that's not you then I'm not talking about you. I'm speaking based on many years experience talking to and working with hundreds of different developers.)
• Writing automated tests helps create software that is easier and faster to maintain, modify, and extend over time.
• Developers are less likely to create tests if it's not possible to easily create tests for that platform. (A theory)
• If WinUI3 apps do not have a way to enable automated testing, developers will assume they don't need to do so. (A theory)
• If no testing capabilities are provided, it would be easy to assume that the WinUI team, and by proxy Microsoft as a whole, don't care about helping developers produce quality software to run on their platform. (A theory--based on previous developer reactions to windows app development changes.)

Yes, for performing automated testing of applications built with WinForms and WPF, it has mostly been necessary to rely on tooling from 3rd parties. However, UWP (and WinRT, & Windows Phone before it) had automated UI test projects (previously "Coded UI" tests) that made testing of code in application projects possible.

While WinUI3 is seen as the future for Win32 and UWP apps, if it has less functionality than what is currently available for UWP (WinUI2) applications it will be seen as a step backwards.

Yes, it's possible to put application logic in a library project and run tests against the library code, but this is not a perfect solution.
This is particularly an issue as the WinUI3 templates available do not encourage this or make it easy.
Yes, Windows Template Studio does a better job of this but WinTS is not how most developers create their projects.

• It's possible to create a .NET(5) Windows Forms app project and reference it from an MSTest (.NET Core) project.
• It's possible to create a .NET(5) WPF app project and reference it from an MSTest (.NET Core) project. (With a little bit if work to make it run with STA threading)
• It's NOT possible to reference a WinUI3 desktop or UWP app project from an MSTest (.NET5) project. (Due to target framework and dependency incompatibilities.)

Yes, it's possible to create automated UI tests for WinUI3 apps by using WinAppDriver, however:
- It is poorly supported (look at all those open issues)
- It's not easy to get started with.
- It's not easy to use / poorly documented.
- It's fragile and challenging to include in automated/CI processes.
- It's got no public roadmap and no indication that it will be improved. (Plus, the code isn't open so it's not possible to workaround problems or contribute fixes.)

What I'd like:

• WHEN 3.0 SHIPS, have a way to create a test project that can reference a WinUI3 app project and run tests against it.

Yes, my desired list is just one thing. This is the absolute minimum I deem to be acceptable.

For a first version I'll even accept if that can't run headless. Headless would be better so it can easily run on a CI server but I can wait for vNext for that.

I have lots of other ideas for things that could improve or make testing easier/better but those discussions seem much more appropriate once testing is possible.

In summary:

• I want to create great, reliable, well-tested apps.
• I don't want to have to jump through lots of seemingly unnecessary hoops to test the code I write and the apps I create.
• I want this for others too.
• I want Microsoft to be seen as making testing easy and encouraging good testing practices as an ecosystem of low-quality, untested apps hurts everyone in the ecosystem.

[mrlacey]

As WinUI is used by parts of the windows Shell, can details of how these are tested be shared?

The WinUI codebase uses both MSTest2 and TAEF as test frameworks. Without further advice, it might be easy to assume that both must be used to test any all that uses WinUI.

The WinUI codebase uses Tests that use the Microsoft.Windows.Apps.Test (MWAT) library.
MWAT does not seem to be under active development. WinAppDriver has previously been pitched as a more usable way to test with MWAT. Without details of why WinUI uses MWAT directly it could easily be seen as the way all WinUI apps should be tested.

[mrlacey]

In the spirit of Phrasing Positively:

I want to create high quality, well tested apps.
I want all developed apps to be thoroughly tested and of high quality.
The ecosystem suffers when app quality is low.
Testing apps makes it easier to find and fix bugs.
Currently there is no easy solution to test WinUI3 apps.
I want an easy way to set apps created with WInUI3. (Or if not easy, at least possible.)
If Microsoft/the WinUI team do not provide a way to test WinUI3 apps it can be interpreted as implying they don't think testing is important. This will lead to fewer people testing and the general quality of apps in the ecosystem (and the ecosystem as a whole) suffering.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[mrlacey]

Don't give up on requirements for a "proper" (& comprehensive) testing solution for WinUI3 apps.