Proposal: Add support for pipeline-based content processing

[digitalcraftsman]

@spf13 proposed in his roadmap for Hugo (see #3205) the introduction of pipelines to process content files, assets and other media. Based on predefined triggers the pipelined media get processed by a command associated with the triggers. Pipelines feature a queue that buffers incoming media until they're ready to be processed.

Potential usecases are:

• image cropping, resizing and compression via a special filename format
• compiling stylesheets and JS (i.e. Less, Sass and CoffeeScript)
• minification and concatenation of stylesheets and JS

Such applications are likely to require external tools and could lay the foundation for plugins (which will not be focused in this issue) that can be registered as a command for a trigger.

Related issues:

• assets concatenation (#432)
• support for preprocessors (#16)
• Support for minification of generated HTML files (#1251)

Note: this issue just outlines the concept around pipelines and potential applications. A more formal specification has still to be defined. Leave a comment with your ideas to shape this draft further.

[n10v]

Is it out of responsibility of Hugo? Theme developers can minify their themes, if they want. But built-in HTML minification and image compression would be really great and it also isn't hard to implement.
I also can't imagine, how users can configure pipelines.

[tech4him1]

Pipelines might add a lot more complexity to Hugo, versus just something like "pre" and "post" hooks to allow Hugo to hook into other workflows more easily.

[rdwatters]

@digitalcraftsman Maybe we can get some more feedback if you expand a bit on the differences between, say, a pipeline, a plugin, and an external helper?

[rdwatters]

Ah, finally found it. My thoughts about Hugo's purpose and scope is basically the same as when I first started using the project 16 months ago:

https://discuss.gohugo.io/t/roadmap-to-hugo-v1-0/2278/2?u=rdwatters

In other words, I agree with @BoGeM. The best thing that Hugo can do is focus on content creation and content management, which already seems to be the direction that things are heading:

1. Write to markup, JSON, ical, etc (i.e., multiple output formats).
2. Keep things fast, or better, faster (many thanks and kudos to @bep and all the awesome devs for making an SSG that was recently described as "Hugo is so fast it builds my site faster than my browser can reload it")
3. Don't necessarily couple a CMS layer to Hugo but start laying the infrastructure for other's to create multiple drop-in SPA CMSs (seems like this is already happening with Netlify, Forestry, et al).
4. Terser, easier template functions for the most common use cases (this is already improved so much since I first started using the tool).
5. Similar to (3), focus more on how to make the framework easier for theme developers and avoid painting ourselves into a corner with a rigid Wordpress-like everything-is-a-post-or-page data model.
6. Focus on other content-related pieces that are part and parcel of iterative content management: e.g., image cropping/resizing/optimizing/, link checking/management, etc.

I think the JS/NPM community already has all CSS/SCSS/concat/minification covered in spades. That said, if you think there is a way to bridge the two without overburdening Hugo, that's awesome and I defer to you.

All said, I think the coolest piece from @spf13 's presentation is the content bundling piece, so I'm excited to see what the community says about that as well. Thanks, @digitalcraftsman !

[Jos512]

I think the JS/NPM community already has all CSS/SCSS/concat/minification covered in spades. That said, if you think there is a way to bridge the two without overburdening Hugo, that's awesome and I defer to you.

That's true @rdwatters . I also doubt how easy this will be; a minifier like html-minifier already has 30+ configurable options. We might not need all of those options in the Hugo use case, but likely still need more than expected. That gives additional code and docs to maintain, tests to develop and possible conflicts to resolve, as well as forum support queries to attend to.

Perhaps content processing by Hugo natively is something for when the Hugo team is bigger?

[ghost]

Pipelined content is a really cool idea but it's probably too much here. Just a pre and post processing hook that integrates with LiveReload is enough.

If Hugo can fetch additionnal parameters from a preprocessing hook to use it in the templates, it's perfect for me. Maybe by parsing the hook stdout as toml/yaml/json. And if hooks are executed by lexical order, it's like a pipelined processing, but out of Hugo itself.

This can be done right now, using a simple Makefile/Gruntfile/Gulpfile/Rakefile/AnythingYouWant but that do not integrate with LiveReload and I'm sure it is what we want.

[andyfleming]

I mostly agree that taking on the whole world of npm, gulp, webpack, javascript, css, and all that might be unnecessary.

However, in first using Hugo, I felt like there wasn't a clear workflow or defaults for where to put javascript and css, especially when starting a new site/theme. It quickly becomes obvious that there are some flexible options, but it seems like it should have a more obvious starting point.

I think it is the responsibility of hugo to have clear defaults for where CSS and JS end up (though it should maintain the flexibility to do whatever you want), but it doesn't need to handle extra pre-processing. Once there are more clear defaults, it may actually unify how others approach layering additional tools on top of hugo.

[tdewolff]

@rdwatters Not to brag, but I think my minify library would not overburden Hugo, it is also build with high performance in mind and typical small pages would minify in under 1 ms. It is possible to use NPM tooling for this, which are surely more battle-tested, but they are usually two orders of magnitude slower. I'd be willing to help integrate it if needed.

EDIT: I just noticed that there is a PR already at #2664

[galopin]

@digitalcraftsman @rdwatters Everyone should be compressing their images efficiently. There are many third-party solutions out there that do offer comprehensive automated image optimization solutions. They can work out-of-the box with SSGs, especially Cloudinary.

We should all be automating our image compression.
-- Essential Image Optimization, Addy Osmani

[bep]

I have spent considerable amount of time thinking about "page bundles" (see #3651), which also includes basic image processing (scaling, resizing etc.) -- and in my head the "assets pipeline" model only works for global assets (examples would be the main stylesheet, all PNG images etc.).

For page or template local assets, I find a reactive/lazy/on-demand model easier to reason about and to get up to the scale and performance Hugo users expect.

I have gotten some gray hair wrapping my head around how to handle changes in these bundles. But now, watching the browser reload with updated thumbnails after an edit of the original in Photoshop or similar after I change the thumbnail settings in the template, I can safely say this: That would not have been possible at this speed without some kind of controlled content dependency graph.

[regisphilibert]

Now that .Resources will be available to any kind of file. (#4381)
Wouldn't it make sense to rely on them for this content processing feature?

{{ $script := .Site.Resources.GetMatch "js/app.js" }}
<script type="text/javascript" src="{{ $script.Minify }}?{{ $style.FingerPrint }}"></script>

Would make sense.

PS: On some very public repos, you'd rather have your minified files committed so your users don't have to npm install/grunt when they start using your theme or upgrading it. This means merging conflict hell! Having hugo do at least the minification would solve that!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[Jos512]

I don't think this issue should be closed, mr. Stale Bot. 🙂

I still love to have HTML, JavaScript, and CSS minification + finger printing in Hugo. But not just me, I think the Hugo community will also benefit from it. Particular people that come from platforms like WordPress, and don't know Gulp/Grunt or any other asset processing tool. For them Hugo will become easier to use when that kind of minification is provided out of the box.

And with the SASS support coming in the future, this might be something that's easier to implement.

[hanzei]

I think hugo 0.43 and hugo piper added everythink needed for this issue. See http://gohugo.io/hugo-pipes/minification/

[bep]

@hanzei really appreciate the work you do on issue tracking. I guess I got a little burned out when I finished implementing all the asset handling stuff.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.