Update to Spark 3.0 and Java 11 (or 17)

[droazen]

No description provided.

[Abby3017]

Hey, just want to know that does GATK 4.1.8.1 support spark 3.0.1 ?
If it doesn't support, do you guys have timeline to support Spark 3.x ?
One more suggestion, if you guys mention version of spark it support for each version would be helpful.
Thank you

[droazen]

@Abby3017 We are hoping to add support for Spark 3.x within the next few months. The currently-supported Spark version as of GATK 4.1.9.0 is 2.4.5. I agree that it would be helpful to make the supported Spark version easier to discover -- I've opened a ticket to modify GATK to print the Spark version on startup: #7027

[droazen]

@lbergelson There is currently an intermittent segfault in the GKL on travis in Java 11 that might pose an obstacle for this ticket:

#6649
Intel-HLS/GKL#133

[lbergelson]

Yes, this is not ideal. It's weird that this just started. I wonder what's changed in the environment.

[droazen]

Blocked waiting for the GKL upgrade PR

[calvertj]

Hello,

Wondering what the status of this is? It seems like the java 11 issue is closed, so that blocker is gone correct?
We would really like spark 3.0.1 + for compatibility with other tools, thanks.

[calvertj]

@droazen , any updates here?

[droazen]

@lbergelson Could you comment on what's holding this migration up when you get a chance?

[lbergelson]

What's holding us up is me being very and busy with other things. I don't think there are any technical problems that we know about. The newest disq releases support 3+. We just need to upgrade and test to find out what we broke.

[natalie-23-gill]

Upgrading to Spark 3.3 (switches to log4j 2.x) when it's released would be ideal since it would solve the security issue mentioned here

[droazen]

@cmnbroad is currently working on this -- we're hoping to update GATK to Java 17, if possible

[cmnbroad]

Closed via #8035.