Run tests only for test projects

[smadala]

Description

• Avoid obvious failure for non test projects

Related issue

#1129
#705
#411

[smadala]

/cc @dasMulli Thanks for idea 👍 #411 (comment)

[dasMulli]

Btw I was protoyping a different idea where:

each project exhibits a target to collect test assemblies.

<Project>
  <Target Name="CollectBuildTestOutput"
    Condition="'$(TargetFramework)' != '' or '$(TargetPath)' != ''"
    DependsOnTargets="Build"
    Returns="@(TestAssembly)">
    <ItemGroup>
      <TestAssembly Include="$([System.IO.Path]::GetFullPath($(TargetPath)))" TargetFrameworkMoniker="$(TargetFrameworkMoniker)" />
    </ItemGroup>
  </Target>
  
  <Target Name="CollectInnerBuildTestOutput"
    Condition="'$(TargetFramework)' == '' and '$(TargetFrameworks)' != ''">
    <ItemGroup>
      <_InnerTestFramework Include="$(TargetFrameworks)" />
    </ItemGroup>
    <MSBuild Projects="$(MSBuildProjectFile)" Targets="CollectBuildTestOutput" Properties="TargetFramework=%(_InnerTestFramework.Identity)">
      <Output TaskParameter="TargetOutputs" ItemName="TestAssembly" />
    </MSBuild>
  </Target>

  <Target Name="CollectTestBuildOutput"
     Condition="'$(IsTestProject)' == 'true'"
     DependsOnTargets="CollectBuildTestOutput;CollectInnerBuildTestOutput"
     Returns="@(TestAssembly)" />
</Project>

then the solution collects the assemblies and calls dotnet vstest (since the VSTest task only accepts a single assembly but this could be done using solution extensions targets that make an updated task available)

  <Target Name="VSTest">
    <MSBuild Projects="@(ProjectReference)" Targets="CollectTestBuildOutput">
      <Output TaskParameter="TargetOutputs" ItemName="TestAssembly" />
    </MSBuild>
    <Exec Command="dotnet vstest --Framework:%(TestAssembly.TargetFrameworkMoniker) @(TestAssembly,' ')"
    StandardOutputImportance="high" />
  </Target>

[dasMulli]

(updated to fix it for multi-targeting projects)

I have no idea how this behaves with multiple test runners or different runners in the same solution (or different versions of xunit etc.).
I guess there is some logic to handle this in the VS "run all" experience already?
But currently this works quite well when run using dotnet msbuild -restore -t:VSTest and all test projects using the same version of xunit.

[dasMulli]

(sry for spamming)

I'm fine with however this is resolved to make typing dotnet test work.
I see it as a dev experience and my tests are usually factored well enough to not have weird dependency issues.
Some users however prefer having a single trx output. (but I don't mind since VSTS can merge multiple results anyway.. need this for JS + .NET tests anyway)

[smadala]

I have no idea how this behaves with multiple test runners or different runners in the same solution (or different versions of xunit etc.)

Different test frameworks(NUnit, xUnit, MSTest V2,etc..) and version will be handled by vstest.console.

[smadala]

I'm planning to address below problems in next few days. I will create an issue for each task very soon.

[smadala]

<Target Name="CollectBuildTestOutput"
    Condition="'$(TargetFramework)' != '' or '$(TargetPath)' != ''"
    DependsOnTargets="Build"
    Returns="@(TestAssembly)">
    <ItemGroup>
      <TestAssembly Include="$([System.IO.Path]::GetFullPath($(TargetPath)))" TargetFrameworkMoniker="$(TargetFrameworkMoniker)" />
    </ItemGroup>
  </Target>

I'm just wondering can we avoid build (DependsOnTargets="Build") every time?

[dasMulli]

I think there's a NoBuild property already, so DependsOnTargets="Build" => DependsOnTargets="BuildForTestIfNecessary".
plus a BuildForTestIfNecessary target that conditions a depends on or a CallTarget on Build based on the property passed in.

Alternatively: Build the solution form the outer VSTest target and then remove the DependsOnTargets completely. The TargetPath property should be available regardless.
This will build non-test projects as well but is probably what users would expect.

[dasMulli]

@smadala contact me if you want any feedback, testing etc. (twitter, mail)

[smadala]

@dasMulli Sure 😄

[gitfool]

@smadala when and how will this be available? Ideally it’s in the next drop of the 2.1.x SDK. Failing that, the next 2.2.x SDK preview. Finally, more out of curiosity, is there a way to drop in a nightly build of just MS Build Core?