Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nixos/invidious: cleanup, allow for easy scaling and load balancing, add http3-ytproxy #265857

Merged
merged 8 commits into from
Dec 18, 2023
Merged

nixos/invidious: cleanup, allow for easy scaling and load balancing, add http3-ytproxy #265857

merged 8 commits into from
Dec 18, 2023

Conversation

999eagle
Copy link
Contributor

@999eagle 999eagle commented Nov 6, 2023 •
edited
Loading

Description of changes

This PR contains mainly three changes:

Tests have also been fixed by switching to PostgreSQL 14 (see #216989) and new tests for the scaled Invidious deployment and http3-ytproxy have been added.

I've deployed these changes to my own Invidious instance at iv.catgirl.cloud for testing.

CC: @sbruder @GaetanLepage @infinisil

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@999eagle 999eagle changed the title nixos/invidious: cleanup, allow for easy scaling and load balancing nixos/invidious: cleanup, allow for easy scaling and load balancing, add http3-ytproxy Nov 6, 2023
Ma27 added a commit to Ma27/nixpkgs that referenced this pull request Nov 8, 2023
@Ma27
nixos/postgresql: drop ensurePermissions, fix ensureUsers for postgre…
65fe7b2 
…sql15

Closes NixOS#216989

First of all, a bit of context: in PostgreSQL, newly created users don't
have the CREATE privilege on the public schema of a database even with
`ALL PRIVILEGES` granted via `ensurePermissions` which is how most of
the DB users are currently set up "declaratively"[1]. This means e.g. a
freshly deployed Nextcloud service will break early because Nextcloud
itself cannot CREATE any tables in the public schema anymore.

The other issue here is that `ensurePermissions` is a mere hack. It's
effectively a mixture of SQL code (e.g. `DATABASE foo` is relying on how
a value is substituted in a query. You'd have to parse a subset of SQL
to actually know which object are permissions granted to for a user).

After analyzing the existing modules I realized that in every case with
a single exception[2] the UNIX system user is equal to the db user is
equal to the db name and I don't see a compelling reason why people
would change that in 99% of the cases. In fact, some modules would even
break if you'd change that because the declarations of the system user &
the db user are mixed up[3].

So I decided to go with something new which restricts the ways to use
`ensure*` options rather than expanding those[4]. Effectively this means
that

* The DB user _must_ be equal to the DB name.
* Permissions are granted via `ensureDBOwnerhip` for an attribute-set in
  `ensureUsers`. That way, the user is actually the owner and can
  perform `CREATE`.
* For such a postgres user, a database must be declared in
  `ensureDatabases`.

For anything else, a custom state management should be implemented. This
can either be `initialScript`, doing it manual, outside of the module or
by implementing proper state management for postgresql[5], but the
current state of `ensure*` isn't even declarative, but a convergent tool
which is what Nix actually claims to _not_ do.

Regarding existing setups: there are effectively two options:

* Leave everything as-is (assuming that system user == db user == db
  name): then the DB user will automatically become the DB owner and
  everything else stays the same.

* Drop the `createDatabase = true;` declarations: nothing will change
  because a removal of `ensure*` statements is ignored, so it doesn't
  matter at all whether this option is kept after the first deploy (and
  later on you'd usually restore from backups anyways).

  The DB user isn't the owner of the DB then, but for an existing setup
  this is irrelevant because CREATE on the public schema isn't revoked
  from existing users (only not granted for new users).

[1] not really declarative though because removals of these statements
    are simply ignored for instance: NixOS#206467
[2] `services.invidious`: I removed the `ensure*` part temporarily
    because it IMHO falls into the category "manage the state on your
    own" (see the commit message). See also
    NixOS#265857
[3] e.g. roundcube had `"DATABASE ${cfg.database.username}" = "ALL PRIVILEGES";`
[4] As opposed to other changes that are considered a potential fix, but
    also add more things like collation for DBs or passwords that are
    _never_ touched again when changing those.
[5] As suggested in e.g. NixOS#206467
@Ma27 Ma27 mentioned this pull request Nov 8, 2023
Merged
7 tasks
Ma27 added a commit to Ma27/nixpkgs that referenced this pull request Nov 9, 2023
@Ma27
nixos/postgresql: drop ensurePermissions, fix ensureUsers for postgre…
02ae527 
…sql15

Closes NixOS#216989

First of all, a bit of context: in PostgreSQL, newly created users don't
have the CREATE privilege on the public schema of a database even with
`ALL PRIVILEGES` granted via `ensurePermissions` which is how most of
the DB users are currently set up "declaratively"[1]. This means e.g. a
freshly deployed Nextcloud service will break early because Nextcloud
itself cannot CREATE any tables in the public schema anymore.

The other issue here is that `ensurePermissions` is a mere hack. It's
effectively a mixture of SQL code (e.g. `DATABASE foo` is relying on how
a value is substituted in a query. You'd have to parse a subset of SQL
to actually know which object are permissions granted to for a user).

After analyzing the existing modules I realized that in every case with
a single exception[2] the UNIX system user is equal to the db user is
equal to the db name and I don't see a compelling reason why people
would change that in 99% of the cases. In fact, some modules would even
break if you'd change that because the declarations of the system user &
the db user are mixed up[3].

So I decided to go with something new which restricts the ways to use
`ensure*` options rather than expanding those[4]. Effectively this means
that

* The DB user _must_ be equal to the DB name.
* Permissions are granted via `ensureDBOwnerhip` for an attribute-set in
  `ensureUsers`. That way, the user is actually the owner and can
  perform `CREATE`.
* For such a postgres user, a database must be declared in
  `ensureDatabases`.

For anything else, a custom state management should be implemented. This
can either be `initialScript`, doing it manual, outside of the module or
by implementing proper state management for postgresql[5], but the
current state of `ensure*` isn't even declarative, but a convergent tool
which is what Nix actually claims to _not_ do.

Regarding existing setups: there are effectively two options:

* Leave everything as-is (assuming that system user == db user == db
  name): then the DB user will automatically become the DB owner and
  everything else stays the same.

* Drop the `createDatabase = true;` declarations: nothing will change
  because a removal of `ensure*` statements is ignored, so it doesn't
  matter at all whether this option is kept after the first deploy (and
  later on you'd usually restore from backups anyways).

  The DB user isn't the owner of the DB then, but for an existing setup
  this is irrelevant because CREATE on the public schema isn't revoked
  from existing users (only not granted for new users).

[1] not really declarative though because removals of these statements
    are simply ignored for instance: NixOS#206467
[2] `services.invidious`: I removed the `ensure*` part temporarily
    because it IMHO falls into the category "manage the state on your
    own" (see the commit message). See also
    NixOS#265857
[3] e.g. roundcube had `"DATABASE ${cfg.database.username}" = "ALL PRIVILEGES";`
[4] As opposed to other changes that are considered a potential fix, but
    also add more things like collation for DBs or passwords that are
    _never_ touched again when changing those.
[5] As suggested in e.g. NixOS#206467
@infinisil
Copy link
Member

Generally looks good! I don't have the time for a detailed review, but ping me before it gets stuck too long :)

RaitoBezarius pushed a commit to Ma27/nixpkgs that referenced this pull request Nov 13, 2023
@Ma27 @RaitoBezarius
nixos/postgresql: drop ensurePermissions, fix ensureUsers for postgre…
e16f94a 
…sql15

Closes NixOS#216989

First of all, a bit of context: in PostgreSQL, newly created users don't
have the CREATE privilege on the public schema of a database even with
`ALL PRIVILEGES` granted via `ensurePermissions` which is how most of
the DB users are currently set up "declaratively"[1]. This means e.g. a
freshly deployed Nextcloud service will break early because Nextcloud
itself cannot CREATE any tables in the public schema anymore.

The other issue here is that `ensurePermissions` is a mere hack. It's
effectively a mixture of SQL code (e.g. `DATABASE foo` is relying on how
a value is substituted in a query. You'd have to parse a subset of SQL
to actually know which object are permissions granted to for a user).

After analyzing the existing modules I realized that in every case with
a single exception[2] the UNIX system user is equal to the db user is
equal to the db name and I don't see a compelling reason why people
would change that in 99% of the cases. In fact, some modules would even
break if you'd change that because the declarations of the system user &
the db user are mixed up[3].

So I decided to go with something new which restricts the ways to use
`ensure*` options rather than expanding those[4]. Effectively this means
that

* The DB user _must_ be equal to the DB name.
* Permissions are granted via `ensureDBOwnerhip` for an attribute-set in
  `ensureUsers`. That way, the user is actually the owner and can
  perform `CREATE`.
* For such a postgres user, a database must be declared in
  `ensureDatabases`.

For anything else, a custom state management should be implemented. This
can either be `initialScript`, doing it manual, outside of the module or
by implementing proper state management for postgresql[5], but the
current state of `ensure*` isn't even declarative, but a convergent tool
which is what Nix actually claims to _not_ do.

Regarding existing setups: there are effectively two options:

* Leave everything as-is (assuming that system user == db user == db
  name): then the DB user will automatically become the DB owner and
  everything else stays the same.

* Drop the `createDatabase = true;` declarations: nothing will change
  because a removal of `ensure*` statements is ignored, so it doesn't
  matter at all whether this option is kept after the first deploy (and
  later on you'd usually restore from backups anyways).

  The DB user isn't the owner of the DB then, but for an existing setup
  this is irrelevant because CREATE on the public schema isn't revoked
  from existing users (only not granted for new users).

[1] not really declarative though because removals of these statements
    are simply ignored for instance: NixOS#206467
[2] `services.invidious`: I removed the `ensure*` part temporarily
    because it IMHO falls into the category "manage the state on your
    own" (see the commit message). See also
    NixOS#265857
[3] e.g. roundcube had `"DATABASE ${cfg.database.username}" = "ALL PRIVILEGES";`
[4] As opposed to other changes that are considered a potential fix, but
    also add more things like collation for DBs or passwords that are
    _never_ touched again when changing those.
[5] As suggested in e.g. NixOS#206467
RaitoBezarius pushed a commit to Ma27/nixpkgs that referenced this pull request Nov 13, 2023
@Ma27 @RaitoBezarius
nixos/postgresql: drop ensurePermissions, fix ensureUsers for postgre…
4845956 
…sql15

Closes NixOS#216989

First of all, a bit of context: in PostgreSQL, newly created users don't
have the CREATE privilege on the public schema of a database even with
`ALL PRIVILEGES` granted via `ensurePermissions` which is how most of
the DB users are currently set up "declaratively"[1]. This means e.g. a
freshly deployed Nextcloud service will break early because Nextcloud
itself cannot CREATE any tables in the public schema anymore.

The other issue here is that `ensurePermissions` is a mere hack. It's
effectively a mixture of SQL code (e.g. `DATABASE foo` is relying on how
a value is substituted in a query. You'd have to parse a subset of SQL
to actually know which object are permissions granted to for a user).

After analyzing the existing modules I realized that in every case with
a single exception[2] the UNIX system user is equal to the db user is
equal to the db name and I don't see a compelling reason why people
would change that in 99% of the cases. In fact, some modules would even
break if you'd change that because the declarations of the system user &
the db user are mixed up[3].

So I decided to go with something new which restricts the ways to use
`ensure*` options rather than expanding those[4]. Effectively this means
that

* The DB user _must_ be equal to the DB name.
* Permissions are granted via `ensureDBOwnerhip` for an attribute-set in
  `ensureUsers`. That way, the user is actually the owner and can
  perform `CREATE`.
* For such a postgres user, a database must be declared in
  `ensureDatabases`.

For anything else, a custom state management should be implemented. This
can either be `initialScript`, doing it manual, outside of the module or
by implementing proper state management for postgresql[5], but the
current state of `ensure*` isn't even declarative, but a convergent tool
which is what Nix actually claims to _not_ do.

Regarding existing setups: there are effectively two options:

* Leave everything as-is (assuming that system user == db user == db
  name): then the DB user will automatically become the DB owner and
  everything else stays the same.

* Drop the `createDatabase = true;` declarations: nothing will change
  because a removal of `ensure*` statements is ignored, so it doesn't
  matter at all whether this option is kept after the first deploy (and
  later on you'd usually restore from backups anyways).

  The DB user isn't the owner of the DB then, but for an existing setup
  this is irrelevant because CREATE on the public schema isn't revoked
  from existing users (only not granted for new users).

[1] not really declarative though because removals of these statements
    are simply ignored for instance: NixOS#206467
[2] `services.invidious`: I removed the `ensure*` part temporarily
    because it IMHO falls into the category "manage the state on your
    own" (see the commit message). See also
    NixOS#265857
[3] e.g. roundcube had `"DATABASE ${cfg.database.username}" = "ALL PRIVILEGES";`
[4] As opposed to other changes that are considered a potential fix, but
    also add more things like collation for DBs or passwords that are
    _never_ touched again when changing those.
[5] As suggested in e.g. NixOS#206467
@999eagle
Copy link
Contributor Author

@RaitoBezarius I've changed this module to use ensureDBOwnership = true now as was requested in #266270.

@infinisil it'd be great if you could review this :)

@RaitoBezarius
Copy link
Member

@RaitoBezarius I've changed this module to use ensureDBOwnership = true now as was requested in #266270.

@infinisil it'd be great if you could review this :)

Thank you so much!!

@999eagle
Copy link
Contributor Author

Updated so the database changes now only apply by default for system.stateVersion >= 24.05.

@GaetanLepage
Copy link
Contributor

GaetanLepage commented Nov 24, 2023 •
edited
Loading

Updated so the database changes now only apply by default for system.stateVersion >= 24.05.

What if a user is following nixos-unstable ?
He might still have stateVersion = 23.11 in his config but at the same time he would be running a recent version of invidious.
Wouldn't it complain about not finding the database ?

@999eagle
Copy link
Contributor Author

If stateVersion is anything less than 24.05, the default database user stays kemal to not break compatibility with externally provisioned databases. For automatically provisioned databases the assertion is triggered to prompt users to manually set the database user to invidious which will then create the new user in postgres and change the owner of the database to the new user. I think this should be enough to not break compatibility.

@GaetanLepage
Copy link
Contributor

If stateVersion is anything less than 24.05, the default database user stays kemal to not break compatibility with externally provisioned databases. For automatically provisioned databases the assertion is triggered to prompt users to manually set the database user to invidious which will then create the new user in postgres and change the owner of the database to the new user. I think this should be enough to not break compatibility.

Oh I see ! I missed the warning part. Thanks :)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1290

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1318

@RaitoBezarius
Copy link
Member

@999eagle Can I ask you to resolve the comments and resolve the conflict? Then, we can go for the merge.

@999eagle 999eagle force-pushed the feat/invidious-scale branch 2 times, most recently from 30da208 to 2a2cf92 Compare December 17, 2023 18:44
@999eagle
nixos/invidious: remove obsolete database maintenance service
4a7faea 
See the note on https://docs.invidious.io/db-maintenance/. Manual maintenance
is no longer required.
@999eagle
nixos/tests/invidious: move postgres-tcp into second machine and fix …
460e34b 
…tests

Using PostgreSQL 15 without the init script fails due to
NixOS#216989.
@999eagle
nixos/invidious: add option to run more invidious instances
45bd4b1
@999eagle
nixos/tests/invidious: add test for scaled invidious
65e8f8a
@999eagle
nixos/invidious: bind to 127.0.0.1 instead of 0.0.0.0 if nginx is used
d41706b
@999eagle
http3-ytproxy: init at unstable-2022-07-03
5e320db
@999eagle
nixos/invidious: add option to configure http3-ytproxy for invidious
ac5c188
@999eagle
nixos/invidious: change default database user to invidious
24e561f 
This makes sure we don't need any workarounds for running Invidious with a local
PostgreSQL database.
Changing the default user should be fine as the new init script for PostgreSQL automatically
creates the new user and changes the existing database's owner to the new user. The old user
will still linger and must be removed manually.
See also: NixOS#266270
@999eagle
Copy link
Contributor Author

The conflict is in the release notes, so as long as the release notes for 24.05 are relatively short, it will conflict again very quickly. I've rebased to fix the conflict now.
I was still waiting on more feedback on some of the open comment threads but if you're fine with it I can just resolve them.

@RaitoBezarius
Copy link
Member

The conflict is in the release notes, so as long as the release notes for 24.05 are relatively short, it will conflict again very quickly. I've rebased to fix the conflict now. I was still waiting on more feedback on some of the open comment threads but if you're fine with it I can just resolve them.

Don't hesitate to resolve them otherwise you get stuck in this "everyone is waiting on each other" hell and we can still improve further your stuff later on. :)

@RaitoBezarius RaitoBezarius merged commit 17c3ebd into NixOS:master Dec 18, 2023
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SuperSandro2000 SuperSandro2000 SuperSandro2000 approved these changes

@infinisil infinisil Awaiting requested review from infinisil

@GaetanLepage GaetanLepage Awaiting requested review from GaetanLepage

@RaitoBezarius RaitoBezarius Awaiting requested review from RaitoBezarius

Assignees
No one assigned
Labels
6.topic: nixos 8.has: changelog 8.has: documentation 8.has: module (update) 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@999eagle @infinisil @RaitoBezarius @GaetanLepage @nixos-discourse @SuperSandro2000