jenkins stage doesn't fail on error in ingest-dataset

[peggynewman]

An ERROR in the logs on the UUID and the Index stages didn't fail the stage and the job kept going.

Ingest dataset job: http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/87/console

UUID stage: http://aws-spark-quoll-master.ala:9194/job/UUID-dataset/605/console

Found this looking for a loaded record count.

[javier-molina]

@djtfmartin can I leave this with you?

[djtfmartin]

This is expected behaviour - but obviously we can change if there's a better solution. There maybe something we can do on the jenkins side.

The way it works is the Indexing pipeline validates that other pipelines have completed successfully for a dataset.
it checks

• Interpretation has been ran dataset
• UUIDs have been run and have been run after interpretation (so we are in-sync)
• SDS has been run after interpretation

If all these validations pass then we generate the IndexRecords for a dataset ready for SOLR indexing. If any fail, we don't and the dataset does make it into the SOLR index.

We've done it this way so that we avoid the situation that a single problem dataset breaks our processes for all datasets e.g. "Uuid-all". We can regenerate all UUIDs and skip over the dataset that's causing an issue.

We could change the end of each pipeline so that we throw an exception if it fails. Whether we throw the exception could be determined by config and hence could be a commandline/jenkins parameter for the job.

[peggynewman]

Hi Dave, after discussing with @patkyn this sounds like it would be very useful to use config to set whether the UUID job fails, but that's not high priority.