Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use fakes to surround vstest core and test it #3347

Merged
merged 37 commits into from
Feb 21, 2022

Conversation

nohwnd
Copy link
Member

@nohwnd nohwnd commented Feb 10, 2022

Adds a set of fakes (not Microsoft.Fakes), that replicate the basics of functionality that we need complete a test run. This is a first step towards having the ability to write "integration" tests that run in memory, and inject different kinds of failures. If you look at UnitTests1.cs, the code looks terribly complicated right now. But it will clean up when more relationships are clear. Even though there are many Fake implementations, just 3 or 4 are crucial. FakeTestRuntimeProvider, because it is an extension point, and it manages all aspects of setting up and tearing down a testhost. FakeCommunicationChannel, because that provides the way for posting and receiving messages that would normally go via sockets, and it allows me to receive and send messages.

Only a single test is written here, and it depends on having just a single "testhost" and just a single communication channel to it. But it shows that the idea is possible.

When moving forward with this I want to keep these rules in mind:

  • Keep automatic behavior to minimum. E.g. ProcessHelper.LaunchProcess should not try to simulate starting any process it is given, but rather find the process object in a list that was injected from test, or throw.
  • Keep fake relationships as minimal as possible. The fake code should not need debugging in most cases.
  • Record everything. Fakes try to record everything that they did, especially all events that were produced, that are waiting to be processed, and that were processed. So you can stop in a test and look at the state easily.
  • Pass all errors to ErrorAggregator so we can assert on them in the test.
  • Where possible use the real Fake* type rather than some interface, to avoid confusion about which dependencies are real an which are fake.
  • Review where the boundaries are. E.g. AssemblyMetadataProvider is better place to cut off, because that boils down to just few properties, than trying to replicate filesystem, and loading an assembly binary content from virtual file system.
  • Document where boundaries are and find out what is the limitation of this approach.
  • Not planned: Add ways to test other pieces in isolation, like testhost as a whole, the pieces that we actually cut of (e.g. DotnetTestHostProvider, AssemblyMetadataProvider), and possibly add way to tie two sides together, so we test Sender <-> Handler interaction.
  • Avoid breaking changes, and make this possible to merge.

@nohwnd nohwnd marked this pull request as ready for review February 18, 2022 15:31
Copy link
Member

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will finish the review later, my brain is dead now.

Copy link
Member

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments from me but otherwise LGTM! Thanks for this awesome improvement!

// TODO: In various places TestRequest sender is instantiated, and we can't easily inject the factory, so this is last
// resort of getting the dependency into the execution flow.
_communicationEndpoint = communicationEndPoint
#if DEBUG
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure, when in debug we want to have the 3 possible paths, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in debug we want to try the service locator before setting the channel.

test/Intent/ConsoleLogger.cs Outdated Show resolved Hide resolved
test/Intent/Extensions.cs Outdated Show resolved Hide resolved
@nohwnd nohwnd merged commit c91e284 into microsoft:main Feb 21, 2022
@nohwnd nohwnd deleted the fakes-over-mocks branch February 21, 2022 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants