diff --git a/docs/meeting-notes/2024-06-27.md b/docs/meeting-notes/2024-06-27.md new file mode 100644 index 0000000..01c932b --- /dev/null +++ b/docs/meeting-notes/2024-06-27.md @@ -0,0 +1,47 @@ +# 2024-06-27 + +Attendees: edef, hexa, vcunat, zimbatm + +## Round table + +- hexa + - Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting Indices (https://github.com/NixOS/infra/issues/446) + + - Actionables: + 1. Setup rsync.net account, so we can have a proper backup, and help hexa's pipe + 2. Try lighter compression with lz4 because we are seeing CPU load bottlenecking + 3. https://github.com/NixOS/infra/pull/447 + - Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade path not clear because of incompatible DB versions. Might need a fresh instance after talking to the marketing team. + +- vcunat: + - Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup target + - DB crashed due to full disk and would stop Hydra from working + +- edef: + - Discussed with tomberek and jonas with getting the Glacier copy started. For only large objects to keep it simple. + - The release bucket traffic has grown again? + - edef: it doesn't seem that sizable based on the graphs I am watching + - hexa: did you see the chart Eelco posted? they looked worrying + - edef: to the fastly endpoint + - hexa: AWS + - edef: (looking the AWS Price explorer) looks like 1000 USD/month (30 USD/day), not exploded + - 2000/2010 style infra team + - We get this software thrown over and shall run it + - How can Hydra be made future-proof? + - Who maintains Hydra? Who makes sure the software works for the infra stack we can provide? + - hexa: Only Ericson updates Hydra to new Nix versions, probably for CA derivations, not much else is happening + - vcunat: Scale has increased much over the years since Hydra was written, and it hasn't kept up + - edef: too few people to commit and cover stuff + - biggest issues: + - queue-runner cannot compute runnables faster than they are getting consumed + - hydra kept busy with expensive xz compression of all results it gets + +- jonas: + - requester pay on the release S3 bucket? + - last rollout resulted in 404 (silent 403s) + - we use the same code as for the cache + - edef: I tried the fastly code for the cache bucket. Tried it on a separate deployment. It doesn't appear to experience the same issues. Doesn't require a privileged token. Not sure how to further debug that. + - could talk about tigris data + - edef: let's get stuff in there + - edef: need to talk to AWS for free egress + - jonas: just the release bucket for now, because we have issues with it