[MPP] Running slow with large number of observations

[oyvindeide]

We are testing the job with a large number of realizations (~30000s), but only a low number of observations (100s). It has been reported that the MisfitPreprocessor is slow with a large number of observations (equinor/flownet#217)

We should:

• Create a test case that has a large number of observations and a large number of realizations to test this
• I suspect the culprit might be the exporting of data from ert, we should profile some of the integration tests to see
  where the bottleneck is.

[oyvindeide]

After initial testing the main problem lies in the export of data from ert. My test case had 10 000 (gen) observations and 100 realizations. The reading of data took more than 5 times longer than running the scaling.

My assumption was wrong, the problem lies in the linkage performed before the cluster analysis.