Update logging: bump lib and introduce pebble log forwarding

[sed-i]

Issue

• The charm is using an outdated loki_push_api charm lib.
• In air-gapped deployments, promtail is not readily available for download.

Solution

• Bump loki_push_api to v1.
• Since air-gapped deployments are Juju 3.4+ anyway, use pebble log forwarding when possible.

Related to:

• Airgap deployment not possible because of promtail grafana-agent-k8s-operator#300
• Add pebble log forwarding support check to jujuversion operator#1230

Testing

Deploy the bundle:

bundle: kubernetes
applications:
  loki:
    charm: loki-k8s
    channel: latest/edge
    base: [email protected]/stable
    resources:
      loki-image: 95
      node-exporter-image: 1
    scale: 1
    trust: true
  pg:
    charm: ./postgresql-k8s_ubuntu-22.04-amd64.charm
    resources:
      postgresql-image: ghcr.io/canonical/charmed-postgresql@sha256:31cf150b4523481202c1ff9b7b5d7f0b36729edad89d61242d8f1eb56b2912c0
    scale: 1
    trust: true
relations:
- - loki:logging
  - pg:logging

Confirm the pebble plan has a log-targets section:

juju exec --unit pg/0 -- PEBBLE_SOCKET=/charm/containers/postgresql/pebble.socket pebble plan

# ...
log-targets:
    loki/0:
        type: loki
        location: http://loki-0.loki-endpoints.pg.svc.cluster.local:3100/loki/api/v1/push
        services:
            - all
        override: replace
        labels:
            charm: postgresql-k8s
            juju_application: pg
            juju_model: pg
            juju_model_uuid: 31887e0e-4c62-48df-85db-7f529bc25177
            juju_unit: pg/0
            product: Juju

Confirms logs are in loki:

LOKI=$(juju status --format=json | jq -r '.applications.loki.units."loki/0".address')

curl -sG $LOKI:3100/loki/api/v1/labels
# {"status":"success","data":["charm","juju_application","juju_model","juju_model_uuid","juju_unit","pebble_service","product"]}

curl -sG $LOKI:3100/loki/api/v1/label/juju_unit/values
# {"status":"success","data":["pg/0"]}

curl -sG $LOKI:3100/loki/api/v1/query_range --data-urlencode 'query={pebble_service="postgresql"}' | jq '.data.result[0]'

Trimmed down output:

{
  "stream": {
    "charm": "postgresql-k8s",
    "juju_application": "pg",
    "juju_model": "pg",
    "juju_model_uuid": "31887e0e-4c62-48df-85db-7f529bc25177",
    "juju_unit": "pg/0",
    "pebble_service": "postgresql",
    "product": "Juju"
  },
  "values": [
    [
      "1716835072419000000",
      "server signaled"
    ],
    [
      "1716834778918000000",
      "2024-05-27 18:32:58 UTC [113]: user=,db=,app=,client=,line=2 HINT:  Future log output will appear in directory \"/var/log/postgresql\"."
    ],
    [
      "1716834777421000000",
      "performing post-bootstrap initialization ... ok"
    ],
    [
      "1716834777317000000",
      "The files belonging to this database system will be owned by user \"postgres\"."
    ]
  ]
}

[sed-i]

Note that, while LogProxyConsumer is streaming everything from

POSTGRES_LOG_FILES = [
    "/var/log/pgbackrest/*",
    "/var/log/postgresql/patroni.log",
    "/var/log/postgresql/postgresql*.log",
]

using LogForwarder streams the stdout of all pebble services.

The two may differ. If critical, then could add a pebble service analogue to tail -f the files.

[phvalguima]

@marceloneppel @shayancanonical @dragomirp, wdyt?

[sed-i]

It seems like there's a way to config pg to log to stdout, but I'm not clear on the syntax.
https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOGGING-COLLECTOR

[dragomirp]

I don't think a tailing service or stdout will work for Pgbackrest out of the box. IIRC it creates separate log files for various commands. I don't see an option to configure the logfile names. Maybe we can use --log-level-console and run the backup commands as a oneshot pebble service?

[marceloneppel]

We probably can also use something like --log-level-console along with container.exec([pgbackrest-command])wait_output().

[lucabello]

Looks good to me!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.76%. Comparing base (a5af248) to head (01a5e94).
Report is 13 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #486      +/-   ##
==========================================
+ Coverage   70.71%   70.76%   +0.04%     
==========================================
  Files          11       11              
  Lines        2926     2931       +5     
  Branches      511      511              
==========================================
+ Hits         2069     2074       +5     
  Misses        748      748              
  Partials      109      109              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dragomirp]

Hi, @sed-i, sorry for the long delay. I've merged current main to the branch and created #627 and #628 to track the outstanding log forwarding issues. I think we can merge this.