Add command for dumping an Elasticsearch instance in the PaaS

[Kuruyia]

What does this PR do?

This adds the paas:elasticsearch:dump command that allows PaaS clients to easily dump the indexes and documents from an Elasticsearch instance of a PaaS application. This PR follows the addition of the application/storage controller in the PaaS console.

This command asks for the PaaS environment name and application ID (a project name can also be specified), and a directory where to dump the Elasticsearch data.
The directory structure is automatically created.

First, the indexes are dumped using the application/storage:getIndexes action, and are stored in the <dump_dir>/indexes.json file.

Then, the documents are dumped usind the application/storage:dumpDocuments action, and are stored in the <dump_dir>/documents.jsonl file. This file is a Newline delimited JSON file.
Because there could be a huge amount of documents totaling a big size for the response, the PaaS console only allows to dump at most a certain amount of documents at a time. Kourou makes multiple requests to the PaaS console until there are no more documents to dump to get them all. The amount of documents dumped can be set with the batch-size flag, but keep in mind that the PaaS console may also limit the maximum amount of documents that can be returned at once.

Finally, the dump session is cleanly terminated and the user is informed where to find their dump files.

Other changes

Types for Node.js were updated to Node 18.

[rolljee]

LGTM, but there are some modifications

[rolljee]

Classname seems odd

[Kuruyia]

Yup, that's a copy-paste error! 😅

[rolljee]

does option { recursive true } works as mkdirp ?

[Kuruyia]

Yes, it works the same as mkdir -p

[Aschen]

Why using multiple files instead of a single file in JSONL format?
We use this format with a stream for the index/collection dump to avoid high memory usage.

I see some potentials problems with the actual design:

• having multiple file can end up with literally thousands of small files which is not efficient
• 10 documents limit is really low and the overhead of a network request should be considered, for instance if you have 100000 documents then you will need 10 thousands requests. This limit should be configurable

Actually most of the issues you will encounter here were already discussed and solved for the index/collection export commands. You can have a look how it was done

[Aschen]

What about the finishDump method if the dump crash in between two pages of documents, should it be called?

[Kuruyia]

Good catch, yes it should totally be called.

[Kuruyia]

Why using multiple files instead of a single file in JSONL format?

I didn't know about that file format, I will look at that 😄

• 10 documents limit is really low and the overhead of a network request should be considered, for instance if you have 100000 documents then you will need 10 thousands requests. This limit should be configurable

You're right, I think I just forgot to make this limit configurable 😅

Actually most of the issues you will encounter here were already discussed and solved for the index/collection export commands. You can have a look how it was done

👍