Wind Tunnel results analysis

[ThetaSinner]

Wind Tunnel reports its results to InfluxDB and there are dashboards for the InfluxDB UI that give you some insight about what is happening during scenario execution. As a reporting and analysis mechanism though this is limited in value. Rather than a graph that gives an overview of all the data, we need a summary that captures key results and can be compared across versions of Holochain.

In a way that is as automated as possible we need to build a report that:

• Presents key metrics for Holochain
• Calculates and presents the % and absolute improvement or regression against a given previous report
• Keeps reports separate by hardware/target environment so that, for example, two reports for HPs and HP+s are treated as different

We should publish reports alongside Holochain releases so that users get an insight into the changes they can expect. For minor version releases (e.g. 0.3.0) we should produce a diff report against the previous relevant version (e.g. 0.2.6) so that people can understand the differences when upgrading across these larger boundaries than just patch releases or weekly releases.

AC:

• measure hard-coded key metrics
• save in a report file
• upload the report file
• create a diff to previous report of same version line
• upload the diff report
• automate this process

[ThetaSinner]

Best estimate for this issue is 1 week.

[jost-s]

Questions:

• In what format should the report be written?
• Where should it be stored and uploaded to?
• How automated is first pass?

[jost-s]

There is some analysis capability in InfluxDB that we may use to query for the relevant data.

Export reports to JSON.

It would be nice for the community to be able to see performance reports at a public location.

Automate the report and diff report creation.

Take key metrics from https://hackmd.io/PeSYrCYaTeGW3Zz8ju77ug?view