59

59 Release Notes

Bug Fixes

• Fixed a pair of bugs that could cause workflows to fail unexpectedly with the errors "413 Request Entity Too Large"
  and "java.net.SocketTimeoutException: Read timed out" when accessing Google Cloud Storage.

58

58 Release Notes

Internal CI-related changes only.

57

57 Release Notes

Breaking configuration change to reference disk support on PAPI v2

Beginning with Cromwell 57, reference disk manifests are now specified completely within Cromwell configuration
rather than through a level of indirection to a manifest file stored in GCS. More details can be found
here.

56

56 Release Notes

Retry with More Memory as workflow option

The experimental memory retry feature gains per-workflow customization and includes breaking changes:

• The per-backend configuration key <backend>.config.memory-retry.error-keys has been removed and replaced
  with global key system.memory-retry-error-keys
• The per-backend configuration key <backend>.config.memory-retry.multiplier has been replaced with workflow option
  memory_retry_multiplier

More details can be found here.

Bug Fixes

• Fixed a bug that caused Cromwell to mark workflows as failed after a single 500, 503, or 504 error from Google Cloud Storage.
  • Cromwell will now retry these errors as designed.
  • The default retry count is 5 and may be customized with system.io.number-of-attempts.

55

55 Release Notes

Apple Silicon support statement

Users with access to the new Mac hardware should review important information provided here.

Bug Fixes

• Fixed a bug that prevented read_json() from working with arrays and primitives. The function now works as expected for all valid JSON data inputs.
  More information on JSON Type to WDL Type conversion can be found here.

• Now retries HTTP 408 responses as well as HTTP 429 responses during DOS/DRS resolution requests.

• Fixed a bug that prevented the call caching diff endpoint from working with scatters in workflows with archived metadata.

New Features

Reference disk support on PAPI v2

Cromwell now offers support for the use of reference disks on the PAPI v2 backend as an alternative to localizing
reference inputs. More details here.

Docker image cache support on PAPI v2 lifesciences beta

Cromwell now offers support for the use of Docker image caches on the PAPI v2 lifesciences beta backend. More details here.

Preemptible Recovery via Checkpointing

• Cromwell can now help tasks recover from preemption by allowing them to specify a 'checkpoint' file which will be restored
  to the worker VM on the next attempt if the task is interrupted. More details here

54

54 Release Notes

Bug Fixes

• Fixed a bug where write_json() failed for Array[_] inputs. It should now work for Boolean, String, Integer, Float,
  Pair[_, _], Object, Map[_, _] and Array[_] (including array of objects) type inputs. More information on WDL Type to JSON Type
  conversion can be found here.

Spark backend support removal

Spark backend was not widely used and it was decided to remove it from the codebase in order to narrow the scope of Cromwell code.

Improved DRS Localizer logging

Error logging while localizing a DRS URI should now be more clear especially when there is a Requester Pays bucket involved.

Per-backend hog factors

Cromwell now allows overriding system-level log factors on back-end level. First, Cromwell will try to use hog-factor
defined in the backend config, and if it is not defined, it will default to using system-wide hog factor.

backend {
  providers {
    PAPIv2 {
      config {
        hog-factor: 2
      }
    }
  }
}

For more information about hog factors please see this page.

martha_v2 Support Removed

Cromwell now only supports resolving DOS or DRS URIs through Martha's most
recent metadata endpoint martha_v3, dropping support for Martha's previous metadata endpoint martha_v2. To switch to
the new version of Martha's metadata endpoint, update the martha.url found in the filesystems
config to point to /martha_v3. More
information on Martha's martha_v3 request and response schema can be found
here.

DOS/DRS localization_optional Support

When running on a backend that supports localization_optional: true any DOS or DRS File values in the generated
command line will be substituted with the gsUri returned from Martha's martha_v3 endpoint. More information on
localization_optional can be found here.

DOS/DRS metadata retrieval retried by default

Attempts to retrieve DOS/DRS metadata from Martha will be retried by default. More information can be found
here.

53.1

Fixes the womtool crash-on-launch regression introduced in 53. We apologize for the inconvenience.

53

53 Release Notes

Martha v3 Support

Cromwell now supports resolving DRS URIs through Martha v3 (in addition to Martha v2). To switch to the new version of Martha, update the martha.url found in the filesystems config to
point to /martha_v3. More information on Martha v3 request and response schema can be found here.

Support for custom entrypoints on Docker images

Cromwell can now support docker images which have custom entrypoints in the PAPIv2 alpha and beta backends.

Alpha support for WDL optional outputs on PAPI v2

• Alpha support for WDL optional output files on the PAPI v2 backend has been added, please see the
  documentation
  for known limitations.

Monitoring Image Script

• Cromwell now supports an optional monitoring_image_script workflow option in addition to the existing
  monitoring_script and monitoring_image options. For more information see the Google Pipelines API Workflow Options
  documentation.

52

52 Release Notes

Documentation

Information on how to properly use the Singularity cache with Cromwell is now
provided in the Cromwell Singularity documentation.

Google library upgrade (#5565)

All previous versions of Cromwell shipped with Google Cloud Storage (GCS) libraries that are now deprecated and will stop working in August 2020. This release adopts updated libraries to ensure uninterrupted operation. The only user action required is upgrading Cromwell.

Bug fixes

• Fixed a bug that required Cromwell to be restarted in order to pick up DNS changes.
  • By default, the JVM caches DNS records with a TTL of infinity.
  • Cromwell now configures its JVM with a 3-minute TTL. This value can be customized by setting system.dns-cache-ttl.
• Clarified an error message that Cromwell emits when the compute backend terminates a job of its own volition (as opposed to termination in response to an abort request from Cromwell)
  • Previously, the error read The job was aborted from outside Cromwell
  • The new error reads The compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute instance, or exceeding the backend's maximum job duration.

51

51 Release Notes

Changes and Warnings

The configuration format for call cache blacklisting has been updated, please see the call caching documentation for details.

Bug fixes

• Fixed a bug where the size(...) function did not work correctly on files
  from a shared filesystem if size(...) was called in the input section on a
  relative path.

• Fixed a bug where the use_relative_output_paths option would not preserve intermediate folders.

New functionality

Call caching blacklisting improvements

Cromwell previously supported blacklisting GCS buckets containing cache hits which could not be copied for permissions
reasons. Cromwell now adds support for blacklisting individual cache hits which could not be copied for any reason,
as well as grouping blacklist caches according to a workflow option key. More information available in the
call caching documentation.

new xxh64 and fingerprint strategies for call caching

Existing call cache strategies path and path+modtime don't work when using docker on shared filesystems
(SFS backend, i.e. not in cloud storage). The file (md5sum) strategy works, but uses a lot of resources.
Two faster strategies have been added for this use case: xxh64 and
fingerprint. xxh64 is a lightweight hashing algorithm, fingerprint is a strategy designed to be very
lightweight. Read more about it in the call caching documentation.