Skip to content

Commit

Permalink
[SPARK-44377][BUILD] Exclude Junit5 dependencies from `jersey-test-fr…
Browse files Browse the repository at this point in the history
…amework-provider-simple`

### What changes were proposed in this pull request?
The following situation has led to Java tests no longer be executed when performing maven tests on the core module:

1.  SPARK-44316 upgrade Jersey from 2.36 to 2.40.
2. Jersey 2.38 start to use [Junit5 instead of Junit4](eclipse-ee4j/jersey#5123).
3. Spark core module uses `org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-simple:2.40:test`, which cascades and introduces the dependencies of Junit5.

So this pr manual exclude `Junit5` dependencies from `org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-simple:2.40:test` to make Java tests of core module can be tested using Maven.

Additionally, this pr leaves a TODO to remove this exclusion after Spark  migrates to Junit5 for Java testing:

- [SPARK-44170](https://issues.apache.org/jira/browse/SPARK-44170): Migrating Junit4 to Junit5

### Why are the changes needed?
Make Java tests of core module can be tested using Maven after SPARK-44316. This issue only affects Maven, not SBT.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass Git Hub Action
- Manual test:

run `mvn clean install -pl core -am -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest`

**Before**

The core module is not running any Java tests

```
[INFO] --- maven-surefire-plugin:3.1.2:test (default-test)  spark-core_2.12 ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:3.1.2:test (test)  spark-core_2.12 ---
[INFO] Skipping execution of surefire because it has already been run for this configuration
```

**After**

The Java test run of the core module was successful

```
[INFO] --- maven-surefire-plugin:3.1.2:test (default-test)  spark-core_2.12 ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running test.org.apache.spark.Java8RDDAPISuite
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.845 s -- in test.org.apache.spark.Java8RDDAPISuite
[INFO] Running test.org.apache.spark.JavaAPISuite
[INFO] Tests run: 92, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.40 s -- in test.org.apache.spark.JavaAPISuite
[INFO] Running test.org.apache.spark.JavaSparkContextSuite
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.587 s -- in test.org.apache.spark.JavaSparkContextSuite
[INFO] Running org.apache.spark.launcher.SparkLauncherSuite
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.977 s -- in org.apache.spark.launcher.SparkLauncherSuite
[INFO] Running org.apache.spark.JavaJdbcRDDSuite
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.491 s -- in org.apache.spark.JavaJdbcRDDSuite
[INFO] Running org.apache.spark.shuffle.sort.PackedRecordPointerSuite
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s -- in org.apache.spark.shuffle.sort.PackedRecordPointerSuite
[INFO] Running org.apache.spark.shuffle.sort.ShuffleInMemoryRadixSorterSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s -- in org.apache.spark.shuffle.sort.ShuffleInMemoryRadixSorterSuite
[INFO] Running org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite
[INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.829 s -- in org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite
[INFO] Running org.apache.spark.shuffle.sort.ShuffleInMemorySorterSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.052 s -- in org.apache.spark.shuffle.sort.ShuffleInMemorySorterSuite
[INFO] Running org.apache.spark.memory.TaskMemoryManagerSuite
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.memory.TaskMemoryManagerSuite
[INFO] Running org.apache.spark.util.SerializableConfigurationSuite
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 s -- in org.apache.spark.util.SerializableConfigurationSuite
[INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterRadixSortSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterRadixSortSuite
[INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterSuite
[INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterSuite
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.488 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterSuite
[INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterRadixSortSuite
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.433 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterRadixSortSuite
[INFO] Running org.apache.spark.io.ReadAheadInputStreamSuite
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.145 s -- in org.apache.spark.io.ReadAheadInputStreamSuite
[INFO] Running org.apache.spark.io.NioBufferedInputStreamSuite
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.540 s -- in org.apache.spark.io.NioBufferedInputStreamSuite
[INFO] Running org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.771 s -- in org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite
[INFO] Running org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.647 s -- in org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite
[INFO] Running org.apache.spark.api.java.OptionalSuite
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.api.java.OptionalSuite
[INFO] Running org.apache.spark.resource.JavaResourceProfileSuite
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.resource.JavaResourceProfileSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 264, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:3.1.2:test (test)  spark-core_2.12 ---
[INFO] Skipping execution of surefire because it has already been run for this configuration
...

Run completed in 21 minutes, 45 seconds.
Total number of tests run: 3196
Suites: completed 300, aborted 0
Tests: succeeded 3196, failed 0, canceled 5, ignored 10, pending 0
All tests passed.
```

Closes #41944 from LuciferYang/SPARK-44377.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
  • Loading branch information
LuciferYang committed Jul 12, 2023
1 parent 054f94a commit 443b49e
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1038,6 +1038,15 @@
<groupId>org.glassfish.jersey.test-framework.providers</groupId>
<artifactId>jersey-test-framework-provider-simple</artifactId>
<version>${jersey.version}</version>
<!--
TODO(SPARK-44170): Remove the exclusion of jupiter after Spark migrates to Junit5.
-->
<exclusions>
<exclusion>
<groupId>org.junit.jupiter</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
<scope>test</scope>
</dependency>
<dependency>
Expand Down

0 comments on commit 443b49e

Please sign in to comment.