Port GrimoireLab to Open Distro

[GeorgLink]

This issue is an idea for the Open Distro Hack Day at All Things Open on October 14.

GrimoireLab currently works with Elasticsearch.
The idea is to:

1. deploy GrimoireLab with Open Distro for Elasticsearch, matching the same version of Elasticsearch
2. port GrimoireLab to the latest version of Open Distro.

[valeriocos]

AFAIK porting grimoirelab to open distro is part of the roadmap, but doing it in 3 days sounds a bit difficult :)

The main reasons of being unable to do this task in the short-term are:

• Migrating all current data to the latest open distro version (and make sure that the data is correct).
• Handle breaking changes between the current ES version used in GrimoireLab and the target one.

@sduenas please share your thoughts

[GeorgLink]

part of the roadmap

That's good to know. I had searched the issues but didn't see anything.

Anyways, it may be more challenging for a one-day project. Any suggestions on smaller steps that can move us in the right direction and earn an easy win for the Hack Day participants?

[valeriocos]

chaoss/grimoirelab-sirmordred#330
The corresponding internal ticket is in the backlog of the roadmap kanban.

I'll think about suggestions and let you know.

[valeriocos]

An idea could be to:

• a. have a docker compose with OD and Kibana
• b. install Perceval
• c1. execute Perceval against Git/GitHub/GitLab/Meetup/... repo
• c2. take the JSON documents returned and save them to OD
• d. create visualizations via Kibana

Points c1 and c2 would require the participants to write a Python script combining Perceval and ElasticSearch (or requests) library. In this way, they will use part of GrimoireLab and play with OD. WDYT?

[sduenas]

What do you mean to port GrimoireLab to OpenDistro? GrimoireLab (especifically Mordred and ELK) only need a ElastickSearch and Kibana endpoints to work. As far as I know, that's what OpenDistro provides.

So configuring SirModred would be more than enough if you have running OpenDistro. Am I missing something?

[valeriocos]

You're right @sduenas, however there are some considerations to take into account.

• deploy GrimoireLab with Open Distro for Elasticsearch, matching the same version of Elasticsearch

  Our stack is Elasticsearch and Kibiter 6.1.x, the first OD version available is 6.5.4 (https://opendistro.github.io/for-elasticsearch-docs/version-history/). Switching to 6.5.4 it might work, but we've never tested it.

• port GrimoireLab to the latest version of Open Distro.

  The last OD version is based on ElasticSearch 7.x, and there is at least a breaking change to handle, reported by 2 issues in mordred:

  • Error with mappings in OpenDistro for Elasticsearch 1.1.0 grimoirelab-sirmordred#330
  • Error: grimoire_elk.elastic - ERROR - Error creating ES mappings grimoirelab-sirmordred#318

[sduenas]

@dlumbrer was able to use GrimoireLab with OpenDistro. Probably he can give us more info.

[valeriocos]

that's true, I forgot about the evaluation of @dlumbrer

[valeriocos]

While waiting for @dlumbrer feedback, we got the one from @jjmerchante (please, feel free to join the discussion) who is using OD 0.9.0 for Cauldron. The platform seems to work fine at least for the following data sources: git, gitlab, github, meetup. He also reported that there are some minor changes to apply on the docker-compose.

For the next time @GeorgLink, it would be great to start this kind of discussions with more than 3 days in advance to be able to help you.

[jsmanrique]

FYI, there is already possible to use core GrimoireLab with Open Distro for Elasticsearch (ODFE), but not with the latest version of ODFE (check chaoss/grimoirelab-sirmordred#330 and chaoss/grimoirelab-sirmordred#318). But if you wanna play I've created a custom docker-compose.yml file for testing ODFE 0.9.0:

https://gitlab.com/jsmanrique/grimoirelab-for-demos#bonus

The main differences with current GrimoireLab full deployment are:

• that everything related with panels is deactivated in the custom setp.cfg I've created, so:
  • There is no default Kibana index pattern (you need to configure it manually in Kibana)
  • There is no navigation menu to access to the dashboards.
  • There are pre-loaded dashboards / panels
• @dlumbrer plugins are not pre-loaded, so you would need to load them manually

Having said that, perhaps a good project for the hackathon would be:

• update GrimoireLab to use latest version of ODFE, with the minimum requirement of updating GrimoireLab data mappings.
• test and demo it with current GrimoireLab for demos or analytics demo.

But, IMHO, I am not sure if that should be something for a GrimoireLab hackathon instead of a ODFE hackathon, but I would let the organizers to decide.

Additional extra "outcomes" could be:

• create an ODFE docker image with kbn_network and kbn_dotplot already installed to be used in docker-compose.yml in GrimoireLab for demos or analytics demo.

And perhaps things could be easier if chaoss/grimoirelab-sirmordred#84 gets solved sometime :-)

[jsmanrique]

@dlumbrer was able to use GrimoireLab with OpenDistro. Probably he can give us more info.

He only tested some data dumps with mini mordred and ODFE 0.9.0. A couple of weeks ago, we tested with the latest ODFE and it was when we realized about the mapping issues mentioned before.

[dlumbrer]

@dlumbrer was able to use GrimoireLab with OpenDistro. Probably he can give us more info.

He only tested some data dumps with mini mordred and ODFE 0.9.0. A couple of weeks ago, we tested with the latest ODFE and it was when we realized about the mapping issues mentioned before.

That's right, I want to add that we can modify the Kibana provided in Open Distro in order to have our Kibiter (that includes menu, colors and plugins installation), because the docker images are available and we can add the features on it.

[GeorgLink]

For the next time @GeorgLink, it would be great to start this kind of discussions with more than 3 days in advance to be able to help you.

Sorry about the short notice. I strive to give everyone as much time as possible. I posted the issue as soon as I knew that we needed an issue. With the short time left, I was not expecting the level of detail that everyone provided, but I very much appreciate all of the input here. I think it is great that we have this much information that will be valuable to hackathon participants.

[valeriocos]

no worries @GeorgLink :) good luck with the hackathon!

[GeorgLink]

Thank you @valeriocos @GeorgLink @jsmanrique @dlumbrer for the discussion and pointing out the potential pitfalls and steps required for resolving this issue. This information was very helpful for me when I was asked about this issue during the Hackathon. Due to the complexity of this issue and the limited time during the Hack day, no one went for this issue. However, someone started and got fairly far with porting the radio chart to open distro for kibana. 👍

[valeriocos]

An initial support for ODFE is ready for review:

• [ELK] Support ODFE grimoirelab-elk#716
• [travis.yml] Support for tests using ODFE grimoirelab-sirmordred#362 (the previous PR should be accepted to avoid Travis to fail)

Note that uploading panels to Kibana is currently not supported for ODFE

Please @sduenas have a look

[sduenas]

GrimoireLab is compatible with ODFE 0.9.0 now.

Closing it.