Generate documentation for third-party packages similar to https://docs.rs/

[yorickpeterse]

Description

With #333 completed, we now have the means to generate documentation for a project.

To make it easier to find documentation for existing projects, we should set up something similar to https://docs.rs/ that generates documentation for a list of projects.

Ideally this tool uses the list of packages found here, such that users don't need to submit their package to two different repositories.

A few challenges/questions present themselves when building this:

• Packages may use syntax/features/etc not supported by idoc. This means we might need a way to specify a minimum version to generate the documentation for
• Cloning each repository, iterating all tags and generating documentation probably won't scale well
• If we upload to Cloudflare R2, we need to set up the necessary caching rules, similar to what we do for the manual and standard library documentation. Using Cloudflare Pages means we have to check-in the generated files into Git, which I'm not a fan of
• If I'm not mistaken, GitHub Actions applies some restrictions in terms of what you can use recurring workflows for. I'm not sure updating the website periodically is actually allowed, since it's technically not used to drive any work on GitHub (e.g. by creating issues)

I feel a push model would be better instead of pulling, but this requires a mechanism by which users can submit a request to generate the documentation for a certain package.

One way we could do this is by tying it into the package list of the website: when the website generator updates the package metadata, it also triggers a CI build to generate the documentation for that project. This way the documentation portal doesn't have to maintain any state. The downside is that we currently only update the package list once a week, which may not be fast enough in the future, though we could allow users to manually submit a request of some sort at some point.

TODO

• Ensure all packages listed at https://inko-lang.org/packages/ compile with 0.15.0, updating them where necessary
• Ensure all these packages use the correct Markdown formatting
• Automatically create an issue if building the documentation fails

Related work

No response

[yorickpeterse]

Regarding the use of recurring workflows, GitHub automatically disables recurring workflows if no repository activity is detected for 60 days: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#disabling-and-enabling-workflows

[yorickpeterse]

Today I spent some time looking into this. One question that comes up is do we actually need this? For example, in the Ruby community it's common to just self-host the documentation (e.g. using GitHub Pages). While RubyDoc exists, I recall (back then I still actively used Ruby) that it wasn't always consistent in offering documentation (e.g. versions might be skipped due to build failures). Even then, many projects don't link to it as their main source of documentation.

One option is that instead we publish a GitHub Action used for generating documentation and uploading it to GitHub Pages. This removes the need for a background service periodically checking what to update, and also gives package authors greater control (e.g. documentation is generated immediately after a release, not a few days later).

[yorickpeterse]

Deploying with GitHub Pages is definitely possible using a workflow like this.

The downside is that GitHub Pages doesn't support deploying to a directory, nor does it support incremental builds. This means that if you want to document multiple versions, you have to re-generate and re-deploy the data for each version every time. This isn't realistic though, as you may need different Inko versions and build steps for different version of a package.

Artifacts and caches aren't persistent either, so neither can be used to reliably store the build output to allow for incremental builds. Tracking generated data in a separate branch (similar to how GitHub Pages used to work) isn't a viable option either, as Git isn't really meant for that sort of thing, and it requires a more complicated CI setup.

This is the same problem as using Cloudflare Pages, hence we deploy to Cloudflare R2 for the manual and standard library documentation.

[yorickpeterse]

Using https://github.com/JamesIves/github-pages-deploy-action it seems we could make it easier to commit changes to a separate branch, then deploy the aggregated result. The downside is that having a fat gh-pages branch means that e.g. inko pkg sync also clones the branch when synchronising packages, even though this is completely redundant.

[yorickpeterse]

Actually, the presence of a gh-pages branch could be dealt with easily: instead of running git clone when a package hasn't been cloned yet, we'd use git init, then we just git fetch the tag as usual. This way we only include the things we actually need.

[yorickpeterse]

Per https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#troubleshooting-publishing-from-a-branch, it seems that having GitHub Actions push changes to a gh-pages branch likely won't trigger the build, as the commit isn't from an actual user. This means package authors have to integrate this step into their (local) release process somehow. In the case of my own packages, that means adding some extra steps to the Makefile.

While this is all very much doable, it feels a bit clunky to say the least.

[yorickpeterse]

To further add: when publishing documentation, we also need some sort of "latest" tag similar to what we do with the standard library and manual. In addition, publishing a backport release shouldn't update such a tag (or the website as a whole in case of the current GitHub Pages setup).

[yorickpeterse]

I think for simple libraries like inko-markdown, just using GitHub Pages to only host the latest version of the documentation is good enough. Most users are likely to use the most recent version of a library and thus its documentation. If one needs the documentation for the exact version used by their project, they can always generate their project's documentation locally and use idoc --dependencies to also include the dependencies.

At some point we may provide a hub of sorts, or maybe provide a better integration (e.g. some tool that uploads to S3/R2/etc), but for now I think we should just document a simple setup in the manual.

[yorickpeterse]

I refactored the page about generating documentation in 1773c9e. As part of this I included an example workflow one can use to deploy documentation to GitHub Pages. This should be good enough for the time being.