-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update benchmark docs #860
Comments
What about storing images externally? I know that git-lfs is often used to store binary files. And supposedly GitHub support it. But I've never used it myself. |
git-lfs is an idea. However there is a consideration that the boost documentation is built by automated tools https://github.com/boostorg/release-tools and also sometimes by end users. It introduces a new requirement to have git-lfs installed in all those cases. |
Oh, right! We actually could just link to external images from the docs! But for that we need a stable link to latest release's benchmark results. |
The latest release was commit 31b10c1. Copied to: https://benchmark.cppalliance.org/jsonbenchmarks-releases/latest.html |
I can't figure out how to integrate it in the docs. There doesn't seem to be a way to generate an iframe in a quickbook document. iframe also looks a bit wierd (second scroll bar). Can we generate the benchmark results as images? SVG should be good enough, I think. |
Yes. There are currently 28 images. The suggestion I was proposing is to show perhaps 1 or 2 images and then refer the reader externally to the other website, not embed the images in an iframe. |
I don't see the point of showing only a few images, because they represent different tests. Maybe we should only show images for one target (e.g. clang). But if the images are generated any way, why not show all of them? |
It's possible. The code has been written. But refer to the original post in this issue about at least a potential "why not". |
Is it against the rules to use javascript in the final html doc output? If docs are ultimately written to .html files, and those are rendered by browsers, and browsers support javascript. The benchmarks are originating from https://github.com/boostorg/json/blob/develop/bench/results.html . That file uses a compact format that renders images dynamically, without png image files. What if the doc/Jamfile copied |
To be honest, I have no idea how to do that. QuickBook doesn't have an option to pass through to HTML, because it produces docbook. And I couldn't find any kind of pass through tag for docbook. |
Not sure either. What if the benchmark docs page is edited to replace the entire section of images and titles with the word "benchmarksplaceholder". After quickbook and docbook are finished, the resulting page with be mostly correct, except the graphics will be missing. The page will be installed to the location /html/json/benchmarks.html . Then, is it possible to have the Jamfile, b2, which is still executing, to run a post-processing step, where it does a find-and-replace such as one-liner bash command UPDATE bash isn't necessarily cross-platform compatible so better choices for the find-and-replace step could be native b2 (if it can do that) or python. |
I discovered a way to embed HTML into the page after all. In #870 I'm using it to show data copied from |
That is fantastic! It solves the main issue. If you manually copy the benchmark results from https://github.com/sdarwin/json/blob/0485ab8af94bd68fa7df92e8864208338b2a3cf7/bench/results.html , those were recently generated, on a fast server. At that point, everything is working. It may not be necessary to add any more features.
The format that's already available are these "basicresults" files. Such as https://benchmark.cppalliance.org/jsonbenchmarks-intel/2023-03-01-12-28-40-4d4d5fc/basicresults/2023-03-08-05-00-02 They are comma delimited and hopefully easy to parse. Using those has pros and cons. They are more up-to-date. It creates a dependency on the server. A static html file in the docs is guaranteed to work. Both choices are options. |
I've just realised that these files lack msvc benchmarks. And as far as I can see, benchmarking hasn't been done with msvc recently. |
The Windows operating system has many proprietary systems processes, that can't be quiesced, and benchmarks showed random variations, green and red, that made the results appear unreliable. so they were removed from the pull request emails. In terms of daily benchmarks, that requires keeping a windows benchmark server running full time in the data center. It was done, for some time, before. |
I have figured out a script to "update the docs" for json benchmarks. https://github.com/sdarwin/json/tree/update-docs-2 . However, now hesitant to submit a PR because of how the system is functioning. Currently, in the json/doc/images/ directory are benchmark images consuming 1.3MB. An update to those images will increase the size of the git repository by 1MB, and thus boost by 1MB. Each time the images are updated it happens again. Maybe json boost docs should display fewer benchmark images?
The text was updated successfully, but these errors were encountered: