external: add support for rados namespace for rbd ec pools

[parth-gr]

currently addded the support for rados namesapce
for rbd ec pools upstream

closes: #13633

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[parth-gr]

@bauerjs1 any feedback?

[bauerjs1]

@parth-gr the "missing rados namespace" error is gone! However I am still experiencing another error:

The provided rgw Endpoint, 'rgw.example.org:443', is invalid.

I added some debugging output to that location in the script and it turns out that this request just gets an HTTP 403 response. Not sure if that is somehow related to #13856 (comment)

Another thing that looks suspicious to me are some of the client caps, e.g.:

Execute: 'ceph auth get-or-create client.csi-rbd-node-staging... mon profile rbd, allow command 'osd blocklist' osd profile rbd pool=ceph-hdd-block namespace=staging'
Execute: 'ceph auth get-or-create client.csi-rbd-provisioner-staging... mon profile rbd, allow command 'osd blocklist' mgr allow rw osd profile rbd pool=ceph-hdd-block namespace=staging'

(from the dry-run output)

Since pool=ceph-hdd-block is an erasure-coded pool, there will be no namespace=staging. The namespace is only present in the metadata pool.

[parth-gr]

@parth-gr the "missing rados namespace" error is gone! However I am still experiencing another error:

The provided rgw Endpoint, 'rgw.example.org:443', is invalid.

I added some debugging output to that location in the script and it turns out that this request just gets an HTTP 403 response. Not sure if that is somehow related to #13856 (comment)

Another thing that looks suspicious to me are some of the client caps, e.g.:

Execute: 'ceph auth get-or-create client.csi-rbd-node-staging... mon profile rbd, allow command 'osd blocklist' osd profile rbd pool=ceph-hdd-block namespace=staging'
Execute: 'ceph auth get-or-create client.csi-rbd-provisioner-staging... mon profile rbd, allow command 'osd blocklist' mgr allow rw osd profile rbd pool=ceph-hdd-block namespace=staging'

(from the dry-run output)

Since pool=ceph-hdd-block is an erasure-coded pool, there will be no namespace=staging. The namespace is only present in the metadata pool.

Can you paste the exact command ou used to run the script and the exact output,

And I believe the original issue is fixed and we are good with the pr

[bauerjs1]

Can you paste the exact command ou used to run the script and the exact output

Command:

python3 create-external-cluster-resources.py
    --format bash
    --namespace storage
    --v2-port-enable
    --restricted-auth-permission true
    --run-as-user=client.staging
    --k8s-cluster-name staging
    --rgw-endpoint rgw.example.org:443
    --rgw-skip-tls true
    --rbd-data-pool-name ceph-hdd-block
    --rbd-metadata-ec-pool-name ceph-hdd-block-metadata
    --rados-namespace staging
    --cephfs-filesystem-name ceph-hdd-file
    --cephfs-data-pool-name ceph-hdd-file-erasure-coded
    --subvolume-group ceph-hdd-file-csi-staging

Output:

WARNING: Multiple data pools detected: ['ceph-hdd-file-default', 'ceph-hdd-file-erasure-coded']
Using the data-pool: 'ceph-hdd-file-erasure-coded'

'v2' address type not present, and 'v2-port-enable' flag is provided/usr/lib/python3.6/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/lib/python3.6/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

# The debug output that I added to the script:
debug: Request URL: https://rgw.example.org:443/admin/info?format=json
debug: Response: <Response [403]>

export NAMESPACE=storage
export ROOK_EXTERNAL_FSID=49b5d4a6-a0bb-4c8f-9736-1f57ba3a5425
export ROOK_EXTERNAL_USERNAME=client.staging
export ROOK_EXTERNAL_CEPH_MON_DATA=a=10.233.5.247:3300
export ROOK_EXTERNAL_USER_SECRET=AQACM/xlRrZJARAA1+sA/lrZYoK9fKWcYM6MRA==
export ROOK_EXTERNAL_DASHBOARD_LINK=http://10.233.73.100:7000/
export CSI_RBD_NODE_SECRET=AQACM/xl5MYzAhAAPXv+iKqJ1yBwWeVYbQU5MA==
export CSI_RBD_NODE_SECRET_NAME=csi-rbd-node-staging-ceph-hdd-block-staging
export CSI_RBD_PROVISIONER_SECRET=AQACM/xlCwAPAxAA752AxL04JsnSnNMk+PVHpQ==
export CSI_RBD_PROVISIONER_SECRET_NAME=csi-rbd-provisioner-staging-ceph-hdd-block-staging
export CEPHFS_POOL_NAME=ceph-hdd-file-erasure-coded
export CEPHFS_METADATA_POOL_NAME=ceph-hdd-file-metadata
export CEPHFS_FS_NAME=ceph-hdd-file
export RESTRICTED_AUTH_PERMISSION=true
export RADOS_NAMESPACE=staging
export SUBVOLUME_GROUP=ceph-hdd-file-csi-staging
export CSI_CEPHFS_NODE_SECRET=AQACM/xl2Su0AxAAAEBURv17/ApLEXXJn0j6wQ==
export CSI_CEPHFS_PROVISIONER_SECRET=AQACM/xlxps6BBAAuGxcXnfRDpLQDV93rTMhAQ==
export CSI_CEPHFS_NODE_SECRET_NAME=csi-cephfs-node-staging-ceph-hdd-file
export CSI_CEPHFS_PROVISIONER_SECRET_NAME=csi-cephfs-provisioner-staging-ceph-hdd-file
export MONITORING_ENDPOINT=10.233.73.100
export MONITORING_ENDPOINT_PORT=9283
export RBD_POOL_NAME=ceph-hdd-block
export RBD_METADATA_EC_POOL_NAME=ceph-hdd-block-metadata
export RGW_POOL_PREFIX=default
export RGW_ADMIN_OPS_USER_ACCESS_KEY=9B0XF2WEFHH2H44TUI5Q
export RGW_ADMIN_OPS_USER_SECRET_KEY=5hD4IiScUERel5LUunUJQFmiGcAq5x7zOSiZ0P4V

The provided rgw Endpoint, 'rgw.example.org:443', is invalid.

Of course rgw.example.org is not the real FQDN I used here 😉

Additional output when using the --dry-run flag:

Execute: 'ceph fs ls'
Execute: 'ceph fsid'
Execute: 'ceph quorum_status'
Execute: 'ceph auth get-or-create client.staging mon allow r, allow command quorum_status, allow command version mgr allow command config osd profile rbd-read-only, allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index'
Execute: 'ceph mgr services'
Execute: 'ceph auth get-or-create client.csi-rbd-node-staging-ceph-hdd-block-staging mon profile rbd, allow command 'osd blocklist' osd profile rbd pool=ceph-hdd-block namespace=staging'
Execute: 'ceph auth get-or-create client.csi-rbd-provisioner-staging-ceph-hdd-block-staging mon profile rbd, allow command 'osd blocklist' mgr allow rw osd profile rbd pool=ceph-hdd-block namespace=staging'
Execute: 'ceph auth get-or-create client.csi-cephfs-node-staging-ceph-hdd-file mon allow r, allow command 'osd blocklist' mgr allow rw osd allow rw tag cephfs *=ceph-hdd-file mds allow rw'
Execute: 'ceph auth get-or-create client.csi-cephfs-provisioner-staging-ceph-hdd-file mon allow r, allow command 'osd blocklist' mgr allow rw osd allow rw tag cephfs metadata=ceph-hdd-file'
Execute: 'ceph status'
Execute: 'ceph radosgw-admin user create --uid rgw-admin-ops-user --display-name Rook RGW Admin Ops user --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm  --rgw-zonegroup  --rgw-zone '

[parth-gr]

@bauerjs1 we did the fix #13982 thanks for reporting, would you like to try the rados namespace now?

[parth-gr]

@bauerjs1 checking again

[parth-gr]

@travisn I believe we are good with this fix.