Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark 3.3.0 - WIP. #7920

Closed
wants to merge 104 commits into from
Closed

Spark 3.3.0 - WIP. #7920

wants to merge 104 commits into from

Commits on Jun 28, 2022

  1. Attempt to build with Java 17 - WIP.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    ec07eea View commit details
    Browse the repository at this point in the history

  2. Upgrade gradle to 7.4.2.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    2770318 View commit details
    Browse the repository at this point in the history

  3. Remove doclet code.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    02a15f8 View commit details
    Browse the repository at this point in the history

  4. Oh yeah remove the doclet tests too.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    9e3c89c View commit details
    Browse the repository at this point in the history

  5. Temporarily pretend we don't need SecurityManager.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    a9e252e View commit details
    Browse the repository at this point in the history

  6. Spark 3.3.0.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    9b044c6 View commit details
    Browse the repository at this point in the history

  7. Suppress more deprecation warnings.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    866d91e View commit details
    Browse the repository at this point in the history

  8. Remove guava lists.

    @cmnbroad
    cmnbroad committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    5c049e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2022

  1. Upgrade Scala - WIP.

    @cmnbroad
    cmnbroad committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    098c6ce View commit details
    Browse the repository at this point in the history

  2. More exports/opens clauses.

    @cmnbroad
    cmnbroad committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    3168672 View commit details
    Browse the repository at this point in the history

  3. Revert problematic Scala upgrade.

    @cmnbroad
    cmnbroad committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    5ec48fc View commit details
    Browse the repository at this point in the history

  4. Open java.lang.invoke.

    @cmnbroad
    cmnbroad committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    7a95da7 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2022

  1. Open sun.nio.fs.

    @cmnbroad
    cmnbroad committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    7285665 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2022

  1. Support changd avro API.

    @cmnbroad
    cmnbroad committed Jul 11, 2022
    Configuration menu
    Copy the full SHA
    f41eee5 View commit details
    Browse the repository at this point in the history

  2. Upgrade jacoco for Java 17.

    @cmnbroad
    cmnbroad committed Jul 11, 2022
    Configuration menu
    Copy the full SHA
    d792d48 View commit details
    Browse the repository at this point in the history

  3. Add opens for jacoco.

    @cmnbroad
    cmnbroad committed Jul 11, 2022
    Configuration menu
    Copy the full SHA
    068c493 View commit details
    Browse the repository at this point in the history

  4. Try running jacoco with --daemon.

    @cmnbroad
    cmnbroad committed Jul 11, 2022
    Configuration menu
    Copy the full SHA
    014e101 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2022

  1. Temporarily update .seg files for ModelSegmentsIntegrationTest.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    294b10c View commit details
    Browse the repository at this point in the history

  2. Remove unused PosteriorSummary and friends.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    3741c42 View commit details
    Browse the repository at this point in the history

  3. I feel lucky.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    92b7f16 View commit details
    Browse the repository at this point in the history

  4. Remove unnecessary github action job matrix entries.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    56c8a09 View commit details
    Browse the repository at this point in the history

  5. Update job matrix for remaining tasks.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    65de87d View commit details
    Browse the repository at this point in the history

  6. Fix expired google cloud key.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    a8fdca2 View commit details
    Browse the repository at this point in the history

  7. You have to update ALL the job matrix entires.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    e1ad86e View commit details
    Browse the repository at this point in the history

  8. Update Docker file for google key.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    3f97219 View commit details
    Browse the repository at this point in the history

  9. Fix statement order.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    789f5bd View commit details
    Browse the repository at this point in the history

  10. Not sudo.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    5cff122 View commit details
    Browse the repository at this point in the history

  11. Pin commons-math3 to 3.5 and revert changes to expected files for Mod… 

    …elSegmentsIntegrationTest.
    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    e5bd651 View commit details
    Browse the repository at this point in the history

  12. Check for failures with scala 11.

    @cmnbroad
    cmnbroad committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    0df59c4 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2022

  1. Revert "Check for failures with scala 11." 

    This reverts commit 0df59c4.
    @cmnbroad
    cmnbroad committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    c2640b6 View commit details
    Browse the repository at this point in the history

  2. Update ModelSegmentsIntegration test using values resulting for commo… 

    …ns-math3-2.5 pinned.
    @cmnbroad
    cmnbroad committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    ec3ea0c View commit details
    Browse the repository at this point in the history

  3. Fix Docker build.

    @cmnbroad
    cmnbroad committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    fb2a649 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2022

  1. Temporarily disable all gatkDoc dependencies.

    @cmnbroad
    cmnbroad committed Jul 18, 2022
    Configuration menu
    Copy the full SHA
    5c5204c View commit details
    Browse the repository at this point in the history

  2. Fix docker java path, fix gradle dependency prereqs.

    @cmnbroad
    cmnbroad committed Jul 18, 2022
    Configuration menu
    Copy the full SHA
    0f9111b View commit details
    Browse the repository at this point in the history

  3. Fix docker second stage.

    @cmnbroad
    cmnbroad committed Jul 18, 2022
    Configuration menu
    Copy the full SHA
    b133ea4 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2022

  1. Update order.

    @cmnbroad
    cmnbroad committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    bc4a917 View commit details
    Browse the repository at this point in the history

  2. Don't apt-upgrade again.

    @cmnbroad
    cmnbroad committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    f73c9cf View commit details
    Browse the repository at this point in the history

  3. Update gatk script.

    @cmnbroad
    cmnbroad committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    182fa95 View commit details
    Browse the repository at this point in the history

  4. Oh there is another gradle file too.

    @cmnbroad
    cmnbroad committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    91a20c0 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2022

  1. Remove unnecessary and problematic non-test closure.

    @cmnbroad
    cmnbroad committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    a612ee7 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2022

  1. Update jvmargs.

    @cmnbroad
    cmnbroad committed Jul 30, 2022
    Configuration menu
    Copy the full SHA
    0dffd93 View commit details
    Browse the repository at this point in the history

  2. Revert "Update jvmargs." 

    This reverts commit 0dffd93.
    @cmnbroad
    cmnbroad committed Jul 30, 2022
    Configuration menu
    Copy the full SHA
    132ea7b View commit details
    Browse the repository at this point in the history

  3. Move jmvargs.

    @cmnbroad
    cmnbroad committed Jul 30, 2022
    Configuration menu
    Copy the full SHA
    d6b7745 View commit details
    Browse the repository at this point in the history

  4. Use open rather than export (--add-opens vs --add-export) for sun.nio.ch

    @cmnbroad
    cmnbroad committed Jul 30, 2022
    Configuration menu
    Copy the full SHA
    b18936e View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2022

  1. Use folder of classes rather than separate jars for the docker tests.

    @cmnbroad
    cmnbroad committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    5157c99 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2022

  1. Update docker build script.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    47818b7 View commit details
    Browse the repository at this point in the history

  2. Add docker trace.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    b195ea2 View commit details
    Browse the repository at this point in the history

  3. Set file permissions.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    e8a713a View commit details
    Browse the repository at this point in the history

  4. TEMP change to pre-create testclass dir.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    f7c0c96 View commit details
    Browse the repository at this point in the history

  5. Fix ls command.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    9607336 View commit details
    Browse the repository at this point in the history

  6. Update file perms.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    c307253 View commit details
    Browse the repository at this point in the history

  7. More permissions.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    305fbb8 View commit details
    Browse the repository at this point in the history

  8. More updates.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    0bf0737 View commit details
    Browse the repository at this point in the history

  9. Move test classes dir so it is onwed by root.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    789e797 View commit details
    Browse the repository at this point in the history

  10. Fix bad command.

    @cmnbroad
    cmnbroad committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    34b112d View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2022

  1. Check files post extract.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    c83795d View commit details
    Browse the repository at this point in the history

  2. Smarter logging.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    a0e2dbd View commit details
    Browse the repository at this point in the history

  3. Update file logging command.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    6ee5f2f View commit details
    Browse the repository at this point in the history

  4. Spill cromwell logs.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    9be9332 View commit details
    Browse the repository at this point in the history

  5. Fix last commit.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    a3026d7 View commit details
    Browse the repository at this point in the history

  6. Fix the fix.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    5438ef2 View commit details
    Browse the repository at this point in the history

  7. update find.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    237a80b View commit details
    Browse the repository at this point in the history

  8. Try running the docker tests using an uber test jar.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    b370a50 View commit details
    Browse the repository at this point in the history

  9. Revert "Try running the docker tests using an uber test jar." 

    This reverts commit b370a50.
    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    9dca49d View commit details
    Browse the repository at this point in the history

  10. Try utf-8 encoding with flat classes.

    @cmnbroad
    cmnbroad committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    d97c108 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2022

  1. Try filtering out the shapeless class entirely.

    @cmnbroad
    cmnbroad committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    02b1b0f View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2022

  1. Set fileMode.

    @cmnbroad
    cmnbroad committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    d238283 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2022

  1. Try an alternative file copy task.

    @cmnbroad
    cmnbroad committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    de037cd View commit details
    Browse the repository at this point in the history

  2. Exclude META_INF.

    @cmnbroad
    cmnbroad committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    b9f09f4 View commit details
    Browse the repository at this point in the history

  3. Set duplicatesStrategy.

    @cmnbroad
    cmnbroad committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    56d75d6 View commit details
    Browse the repository at this point in the history

  4. Upgrade hadoop.

    @cmnbroad
    cmnbroad committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    c286883 View commit details
    Browse the repository at this point in the history

  5. Add the shapeless filter.

    @cmnbroad
    cmnbroad committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    27ecf1a View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2022

  1. Filter spire.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    56c3fb7 View commit details
    Browse the repository at this point in the history

  2. Update messages.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    fbd8cfc View commit details
    Browse the repository at this point in the history

  3. Use the correct classpath.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    8e3b14a View commit details
    Browse the repository at this point in the history

  4. Restore (incorrect) isJavaNN call.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    6ad4ee5 View commit details
    Browse the repository at this point in the history

  5. Add TODO comment.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    7e46c0b View commit details
    Browse the repository at this point in the history

  6. Revert to multiple jars.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    d4d86f0 View commit details
    Browse the repository at this point in the history

  7. Update CP_DIR.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    77d5b16 View commit details
    Browse the repository at this point in the history

  8. Try not caching.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    df915ed View commit details
    Browse the repository at this point in the history

  9. Revert "Try not caching." 

    This reverts commit df915ed.
    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    7e15c4c View commit details
    Browse the repository at this point in the history

  10. Temporarily revert missing io redirection fixes.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    65e2d89 View commit details
    Browse the repository at this point in the history

  11. Revert unpackTestJars.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    69a747e View commit details
    Browse the repository at this point in the history

  12. Export not open.

    @cmnbroad
    cmnbroad committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    c0a7ee2 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2022

  1. Use --add-opens instead of --add-exports.

    @cmnbroad
    cmnbroad committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    1972474 View commit details
    Browse the repository at this point in the history

  2. More opens for spark test code.

    @cmnbroad
    cmnbroad committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    f399f5a View commit details
    Browse the repository at this point in the history

  3. Fix opens.

    @cmnbroad
    cmnbroad committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    c06cbed View commit details
    Browse the repository at this point in the history

  4. More opens.

    @cmnbroad
    cmnbroad committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    9c273c4 View commit details
    Browse the repository at this point in the history

  5. Upgrade kryo.

    @cmnbroad
    cmnbroad committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    fec786a View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2022

  1. Use older kryo.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    df9c068 View commit details
    Browse the repository at this point in the history

  2. More opens.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    7949651 View commit details
    Browse the repository at this point in the history

  3. More opens.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    d5dddbb View commit details
    Browse the repository at this point in the history

  4. Display jvm args.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    50a8766 View commit details
    Browse the repository at this point in the history

  5. Set args directly.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    8109c70 View commit details
    Browse the repository at this point in the history

  6. Remove jacoco jvm args.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    389ba86 View commit details
    Browse the repository at this point in the history

  7. Fix opens.

    @cmnbroad
    cmnbroad committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    63abfc6 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2022

  1. More opens.

    @cmnbroad
    cmnbroad committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    5e0adff View commit details
    Browse the repository at this point in the history

  2. Open java.lang.module.

    @cmnbroad
    cmnbroad committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    b101656 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2022

  1. Lambda serialization, try forcing jetty.

    @cmnbroad
    cmnbroad committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    21ebfae View commit details
    Browse the repository at this point in the history

  2. Upgrade gradle to 7.5.1.

    @cmnbroad
    cmnbroad committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    949cd5d View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2022

  1. Use gradle daemon.

    @cmnbroad
    cmnbroad committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    d5cf9d7 View commit details
    Browse the repository at this point in the history

  2. Use newest jetty.

    @cmnbroad
    cmnbroad committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    cb66210 View commit details
    Browse the repository at this point in the history

  3. Update opens.

    @cmnbroad
    cmnbroad committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    34c96c4 View commit details
    Browse the repository at this point in the history

  4. Revert jetty.

    @cmnbroad
    cmnbroad committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    6977dd8 View commit details
    Browse the repository at this point in the history