Preview size

[pulsejet]

Currently the preview size is fixed to 1024 unlike viewer, which is responsive

[keesfluitman]

preview sizes of the "thumbnails" in your timeline are quite weird as well. Just a list of what came up out of a 100 or so requests.
278x278
310x310
339x339
340x340
344x344
395x395
454x454
why can't all be the same? most were 340x340 though. sometimes a lot of 339

[pulsejet]

That seems correct. The timeline needs to load lower-res thumbs to be performant. But the boundaries at which these exist aren't necessarily the same.

[keesfluitman]

yes. But why so many different sized ones? If I use preview generator to generate 340x340 for all my photos, would it then only use them?

[pulsejet]

Yes. This is a workaround for some weirdness in Nextcloud behavior. The sizes of the returned images match the pre-generated ones.

[keesfluitman]

It's maybe not totally part of this "bug". But I still struggle to find a proper way to pre-generate previews. Mostly because my users have different phones, browsers, etc. For instance I pre-generated, then tested with firefox. Firefox-Memories requests 383... I have 340 generated... So all in all the preview folder is getting bigger and bigger. This does not only concern you of course.

[pulsejet]

It's maybe not totally part of this "bug". But I still struggle to find a proper way to pre-generate previews. Mostly because my users have different phones, browsers, etc. For instance I pre-generated, then tested with firefox. Firefox-Memories requests 383... I have 340 generated... So all in all the preview folder is getting bigger and bigger. This does not only concern you of course.

I assume you're using preview generator to pre-generate.

Nextcloud uses a power-of-2 scheme to serve previews. So when e.g. memories requests 383, the returned preview is not necessarily 383 but whatever is the closest. So this has nothing to do with the preview folder getting bigger, since the sizes of the previews are pre-determined.

[pulsejet]

As an example: the size of the requested and returned images aren't the same:

[rhatguy]

When this is implemented it would be great to get a bit of detail around the recommended configuration (along with the preview generator app) for most people. I've struggled with preview sizes in Nextcloud for years. :(

[keesfluitman]

When this is implemented it would be great to get a bit of detail around the recommended configuration (along with the preview generator app) for most people. I've struggled with preview sizes in Nextcloud for years. :( <

I think this is the best way to set previews. You will have 3 and no more will be created. Memories doesn't request the max version yet though. That kind off annoys me still. It still requests a smaller version. (1024x1024) That's why I won't use memories yet. Since it's max quality is bad and it takes more preview space.

occ config:app:set previewgenerator squareSizes --value="32 256"
occ config:app:set previewgenerator widthSizes --value="256 384"
occ config:app:set previewgenerator heightSizes --value="256"
occ config:system:set preview_max_x --value 2048
occ config:system:set preview_max_y --value 2048
occ config:system:set jpeg_quality --value 60
occ config:app:set preview jpeg_quality --value="60"

I currently have
Total occupied space: 16.8 GB
in 198,490 folders and 155,951 files
for about 30k pictures.

http://web.archive.org/web/20200513043150/https://ownyourbits.com/2019/06/29/understanding-and-improving-nextcloud-previews/

@pulsejet Could you set it to request the max preview (or the original, maybe by a button or config option) at any time for the normal picture? Viewer also requests the max and this way we'd only need 3 types of preview files and have more control over it. (but i'd prefer that anytime I click a picture I get the original. Since I have 10gbit internet, that's fast enough)

Sorry for all the edits:
I must say, although it's so beautiful to have it all integrated. Apps like Pigallery, Photoprism and Photoview do it so much more efficient and clear. Nextcloud has a clear performance issue when a lot of requests are done. And it hurts the server.

I think this boils down to an architectural design choice, which has been at debate in a Github Request and explained here:
http://web.archive.org/web/20200513040858/https://ownyourbits.com/2019/10/16/a-new-architecture-for-nextcloud/

I sometimes consider going back to Owncloud, just for basic cloud functions, and using external apps for other functions. But I'll give it a bit of time :).

[pulsejet]

@keesfluitman this fix is on the roadmap for v4.9

I must say, although it's so beautiful to have it all integrated. Apps like Pigallery, Photoprism and Photoview do it so much more efficient and clear. Nextcloud has a clear performance issue when a lot of requests are done. And it hurts the server.

I'm surprised you've performance issues. I've a similar number of files (~100000) and ~25k photos totaling 100GB and Nextcloud is very fast for me (and I don't run particularly fast hardware, just a Pentium). I used to work with another setup with ~20k users and that had no performance issues either.

BTW a JPEG quality of 60 isn't the most optimal IMO. Seems too low.

[pulsejet]

I think this boils down to an architectural design choice, which has been at debate in a Github Request and explained here: http://web.archive.org/web/20200513040858/https://ownyourbits.com/2019/10/16/a-new-architecture-for-nextcloud/

Btw some of this isn't so valid anymore. With PHP 8 JIT and PHP-FPM + nginx, you don't need to warm up everything for every request; instead there's a thread pool of workers. And PHP got very fast in the last couple of years. If you're running (say) 7.4 with Apache I do expect it'll be much slower.

[keesfluitman]

@keesfluitman this fix is on the roadmap for v4.9

I must say, although it's so beautiful to have it all integrated. Apps like Pigallery, Photoprism and Photoview do it so much more efficient and clear. Nextcloud has a clear performance issue when a lot of requests are done. And it hurts the server.

I'm surprised you've performance issues. I've a similar number of files (~100000) and ~25k photos totaling 100GB and Nextcloud is very fast for me (and I don't run particularly fast hardware, just a Pentium). I used to work with another setup with ~20k users and that had no performance issues either.

BTW a JPEG quality of 60 isn't the most optimal IMO. Seems too low.

It isn't that slow of course. But it's a bit inefficient. I find the other apps have more options (loading of originals), less preview files and faster respond times. I sometimes see nextcloud not requesting previews, not loading the previews faster than when I would on the fly load them. It just seems a bit buggy. Do you have a place where I can learn a bit about how it works (the php 8 JIT, PHP-FPM and nginx? Im trying to learn a bit as well.

Don't know about the jpeg quality. It seems fine atm. Im no expert on that. I just followed the guide on that.

[pulsejet]

Do you have a place where I can learn a bit about how it works (the php 8 JIT, PHP-FPM and nginx? Im trying to learn a bit as well.

I just trust good old Google for these things. There are many nice blogs where people have analyzed this in great detail.

[keesfluitman]

Do you have a place where I can learn a bit about how it works (the php 8 JIT, PHP-FPM and nginx? Im trying to learn a bit as well.

I just trust good old Google for these things. There are many nice blogs where people have analyzed this in great detail.

On loading memories or photos CPU spikes all the time. Don't know why it would need to spike when all it needs to do is send the pictures.

[pulsejet]

On loading memories or photos CPU spikes all the time. Don't know why it would need to spike when all it needs to do is send the pictures.

Access control. For every photo/thumbnail you request, a check is needed if you're allowed to access this photo. The bane of Nextcloud is its flexibility -- there are many different ways the file could be accessible (e.g. directly owned by you, shared by someone else, on an external storage etc.).

[keesfluitman]

ah ye now i remember reading something about that once. So not to bother you too much. But with a sufficient code that loads bulks of pictures that will come next (and unload old ones), wouldn't it be possible that it would simply always request 10-30 previews at once? That is, in the timeline view? So one access request would download that bulk of pictures?

I deleted most of the configs (swag and nextcloud) to let the defaults load again (they weren't updated it seems either). Then I changed the minimal necessary php conf settings and now it seems a bit better. Still loading one by one of course.

[pulsejet]

That's an interesting thought! Another general idea is to offload preview serving to something else, e.g. using an auth token. Just not sure how something like this could be integrated without a lot of extra configuration.

[pulsejet]

Still loading one by one of course.

It definitely should not be that slow. I run a Pentium on the server and all previews load as soon as I can scroll.

[keesfluitman]

Still loading one by one of course.

It definitely should not be that slow. I run a Pentium on the server and all previews load as soon as I can scroll.

no I mean still separate GET requests. It loads quick enough :). Of course after a while it's cached anyway, but with my defaults loaded, it seems much better again.

Im now having fun trying to figure out a nice script that would always change pictures to their proper date taken (give them exif data if they are whatsapp pictures) and change their names to the date it was taken/uploaded. Depending on the task.
Sometimes I get beautiful (but low quality) whatsapp pictures of my kid and friends. So as soon as I save them, I want their info to be changed, so I won't lose this information of when it happened/i got the picture.

[pulsejet]

Im now having fun trying to figure out a nice script that would always change pictures to their proper date taken (give them exif data if they are whatsapp pictures) and change their names to the date it was taken/uploaded. Depending on the task. Sometimes I get beautiful (but low quality) whatsapp pictures of my kid and friends. So as soon as I save them, I want their info to be changed, so I won't lose this information of when it happened/i got the picture.

You could probably make that into a Nextcloud app. PHP has a basic exif extension that works for JPEG, so you don't have to depend on exiftool (maybe leverage it optionally if memories is installed haha). There are hooks for file creation that you could use to (maybe) pattern match the names and update the Exif data.

[major-mayer]

occ config:app:set preview jpeg_quality --value="60"

Can someone confirm that this value is still used nowadays?
I briefly scanned the Nextcloud code a few days ago and couldn't find a reference execpt from the deprecated OC_Image class (which might be not used anymore?).
The newer preview generation process that uses Imaginary also doesn't submit the JPEG quality via HTTP.

Apart from that: Great work with your app, and I am really looking forward to seeing the preview size fixed in the next version 👍
Currently, my previews are all really pixelated.

[pulsejet]

Currently, my previews are all really pixelated.

Hmm, they should not be very pixelated. Is this the problem after opening the photo or on the timeline? After opening it should currently load 1024px, which should look okayish unless you zoom in. Is this not the case?

[major-mayer]

Maybe on smaller FHD displays, but not on a 32'' 4k display. The switch from 1024 to 2048 makes a really big difference there.

[pulsejet]

Ah yes, of course. I was just concerned if it wasn't loading the 1024 image at all (then it'll be unusably pixelated).

[major-mayer]

Ah okay, that's not the case.

[keesfluitman]

Hey great work. So as i see. Now it takes a higher quality image preview on high pixel count monitors? But still takes the max when it's bigger than the max set?

[pulsejet]

On high res monitors it'll take whatever is the highest nextcloud preview setting (same behavior as Photos, and every other non-nextcloud photo app really).

Showing the original image is a more complex problem. See #266

[pulsejet]

Also: https://github.com/pulsejet/memories/wiki/Configuration#preview-storage-considerations

[keesfluitman]

hey. thanks. So I have a full hd monitor. But it still takes the 1024/1024 previews. is this update not online yet? Only until the next big upgrade?

[pulsejet]

Yep. Almost there now: v4.9.0

[keesfluitman]

i love you :)

[pulsejet]

with a sufficient code that loads bulks of pictures that will come next (and unload old ones), wouldn't it be possible that it would simply always request 10-30 previews at once? That is, in the timeline view? So one access request would download that bulk of pictures?

@keesfluitman v4.9.0 uses an optimization like this, and CPU usage (in my measurements) is 5-10x lower on a multi-core machine :)
Basically now it uses a single thread to do the access control once, then serves multiple previews at once. Hopefully this won't cause new bugs; there's no way I can test this enough.

[major-mayer]

Wow, that's amazing ❤️
The development pace of this app is really incredible and by no means comparable with the original Nextcloud photos/files app.
I hope that at some point your work can be upstreamed to improve the situation for all users, even those who are unaware of this app.

I see my donation is well-invested 🙂

[pulsejet]

The development pace of this app is really incredible and by no means comparable with the original Nextcloud photos/files app.

If you're interested in more context: #146 (comment)

I hope that at some point your work can be upstreamed to improve the situation for all users, even those who are unaware of this app.

Likewise: #183 #12

I see my donation is well-invested 🙂

Thank you! 😄