meeting-notes: add 2024-06-07

docs/meeting-notes/2024-06-27.md

@@ -0,0 +1,47 @@
+# 2024-06-27
+
+Attendees: edef, hexa, vcunat, zimbatm
+
+## Round table
+
+- hexa
+    - Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting Indices (https://github.com/NixOS/infra/issues/446)
+
+    - Actionables:
+      1. Setup rsync.net account, so we can have a proper backup, and help hexa's pipe
+      2. Try lighter compression with lz4 because we are seeing CPU load bottlenecking
+      3. https://github.com/NixOS/infra/pull/447
+    - Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade path not clear because of incompatible DB versions. Might need a fresh instance after talking to the marketing team.
+
+- vcunat:
+    - Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup target
+        - DB crashed due to full disk and would stop Hydra from working
+
+- edef:
+    - Discussed with tomberek and jonas with getting the Glacier copy started. For only large objects to keep it simple.
+    - The release bucket traffic has grown again?
+        - edef: it doesn't seem that sizable based on the graphs I am watching
+        - hexa: did you see the chart Eelco posted? they looked worrying
+        - edef: to the fastly endpoint
+        - hexa: AWS
+        - edef: (looking the AWS Price explorer) looks like 1000 USD/month (30 USD/day), not exploded
+    - 2000/2010 style infra team
+        - We get this software thrown over and shall run it
+        - How can Hydra be made future-proof?
+        - Who maintains Hydra? Who makes sure the software works for the infra stack we can provide?
+        - hexa: Only Ericson updates Hydra to new Nix versions, probably for CA derivations, not much else is happening
+        - vcunat: Scale has increased much over the years since Hydra was written, and it hasn't kept up
+        - edef: too few people to commit and cover stuff
+        - biggest issues:
+            - queue-runner cannot compute runnables faster than they are getting consumed
+            - hydra kept busy with expensive xz compression of all results it gets
+
+- jonas:
+    - requester pay on the release S3 bucket?
+        - last rollout resulted in 404 (silent 403s)
+        - we use the same code as for the cache
+        - edef: I tried the fastly code for the cache bucket. Tried it on a separate deployment. It doesn't appear to experience the same issues. Doesn't require a privileged token. Not sure how to further debug that.
+    - could talk about tigris data
+        - edef: let's get stuff in there
+        - edef: need to talk to AWS for free egress
+        - jonas: just the release bucket for now, because we have issues with it