--list-tests and --list-tests-xml CLI options do not report error when data provider method throws exception

[harentius]

Q	A
PHPUnit version	11.2.8 10.5.28 9.6.20
PHP version	8.2.7
Installation Method	Composer

Summary

list-tests (list-tests-xml) command doesn't report any error if dataProvider throws an exception and just return smaller list of tests
As this command(s) often used for CI environment and parallel runs, skipping this exception leads to false positive CI result.

Current behavior

How to reproduce

1. The test (based on the documentation example):

<?php declare(strict_types=1);
use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\TestCase;

final class NumericDataSetsTest extends TestCase
{
    public static function additionProvider(): array
    {
       throw new \Exception();
       return [
            [0, 0, 0],
            [0, 1, 1],
        ];
    }

    #[DataProvider('additionProvider')]
    public function testAdd(int $a, int $b, int $expected): void
    {
        $this->assertSame($expected, $a + $b);
    }

    public function testAnother(): void
    {
        $this->assertSame(0, 0);
    }
}

2. Run
   vendor/bin/phpunit tests --list-tests

Expected behavior

I would expect displaying an error and do not return 0 code.

[sebastianbergmann]

Simplified reproducing example:

Issue5908Test.php

<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;
use PHPUnit\Framework\Attributes\DataProvider;

final class Issue5908Test extends TestCase
{
    public static function provider(): array
    {
        throw new Exception('message');
    }

    #[DataProvider('provider')]
    public function testOne(int $value): void
    {
    }
}

$ ./phpunit --list-tests ~/Issue5908Test.php
PHPUnit 10.5.28-18-g98cbd0f7fc by Sebastian Bergmann and contributors.

Available test(s):

$ ./phpunit ~/Issue5908Test.php
PHPUnit 10.5.28-18-g98cbd0f7fc by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.9
Configuration: /usr/local/src/phpunit/phpunit.xml

There was 1 PHPUnit error:

1) Issue5908Test::testOne
The data provider specified for Issue5908Test::testOne is invalid
message
/usr/local/src/phpunit/src/Metadata/Api/DataProvider.php:174
/usr/local/src/phpunit/src/Metadata/Api/DataProvider.php:67
/usr/local/src/phpunit/src/Framework/TestBuilder.php:40
/usr/local/src/phpunit/src/Framework/TestSuite.php:517
/usr/local/src/phpunit/src/Framework/TestSuite.php:139
/usr/local/src/phpunit/src/TextUI/Configuration/TestSuiteBuilder.php:122
/usr/local/src/phpunit/src/TextUI/Configuration/TestSuiteBuilder.php:58
/usr/local/src/phpunit/src/TextUI/Application.php:373
/usr/local/src/phpunit/src/TextUI/Application.php:112

/home/sb/Issue5908Test.php:13

--

There was 1 PHPUnit test runner warning:

1) No tests found in class "Issue5908Test".

No tests executed!

[harentius]

Thank you for solving it! 🙂