You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is expected behaviour - but obviously we can change if there's a better solution. There maybe something we can do on the jenkins side.
The way it works is the Indexing pipeline validates that other pipelines have completed successfully for a dataset.
it checks
Interpretation has been ran dataset
UUIDs have been run and have been run after interpretation (so we are in-sync)
SDS has been run after interpretation
If all these validations pass then we generate the IndexRecords for a dataset ready for SOLR indexing. If any fail, we don't and the dataset does make it into the SOLR index.
We've done it this way so that we avoid the situation that a single problem dataset breaks our processes for all datasets e.g. "Uuid-all". We can regenerate all UUIDs and skip over the dataset that's causing an issue.
We could change the end of each pipeline so that we throw an exception if it fails. Whether we throw the exception could be determined by config and hence could be a commandline/jenkins parameter for the job.
Hi Dave, after discussing with @patkyn this sounds like it would be very useful to use config to set whether the UUID job fails, but that's not high priority.
An ERROR in the logs on the UUID and the Index stages didn't fail the stage and the job kept going.
Ingest dataset job: http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/87/console
UUID stage: http://aws-spark-quoll-master.ala:9194/job/UUID-dataset/605/console
Found this looking for a loaded record count.
The text was updated successfully, but these errors were encountered: