Merge pull request #9041 from kobergj/ImproveCleanCommand

[full-ci] Assimilate clean command into sessions
owncloud · May 3, 2024 · b1418fe · b1418fe
2 parents acc4b26 + 62b1286
commit b1418fe
Show file tree

Hide file tree

Showing 19 changed files with 1,285 additions and 140 deletions.
diff --git a/changelog/unreleased/add-virus-filter.md b/changelog/unreleased/add-virus-filter.md
@@ -0,0 +1,5 @@
+Enhancement: Add virus filter to sessions command
+
+Allow filtering upload session by virus status (has-virus=true/false)
+
+https://github.com/owncloud/ocis/pull/9041
diff --git a/changelog/unreleased/assimilate-clean-into-sessions.md b/changelog/unreleased/assimilate-clean-into-sessions.md
@@ -0,0 +1,5 @@
+Enhancement: Assimilate `clean` into `sessions` command
+
+We deprecated `ocis storage-user uploads clean` and added the same logic to `ocis storage-users uploads session --clean`
+
+https://github.com/owncloud/ocis/pull/9041
diff --git a/changelog/unreleased/bump-reva.md b/changelog/unreleased/bump-reva.md
@@ -2,6 +2,7 @@ Enhancement: Bump Reva
 
 bumps reva version
 
+https://github.com/owncloud/ocis/pull/9041
 https://github.com/owncloud/ocis/pull/9002
 https://github.com/owncloud/ocis/pull/8917
 https://github.com/owncloud/ocis/pull/8795

diff --git a/go.mod b/go.mod
@@ -15,7 +15,7 @@ require (
 	github.com/cenkalti/backoff v2.2.1+incompatible
 	github.com/coreos/go-oidc/v3 v3.10.0
 	github.com/cs3org/go-cs3apis v0.0.0-20231023073225-7748710e0781
-	github.com/cs3org/reva/v2 v2.19.2-0.20240429085656-5faad8dad61e
+	github.com/cs3org/reva/v2 v2.19.2-0.20240502102837-7e48a5145133
 	github.com/dhowden/tag v0.0.0-20230630033851-978a0926ee25
 	github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e
 	github.com/egirna/icap-client v0.1.1
@@ -84,6 +84,7 @@ require (
 	github.com/spf13/afero v1.11.0
 	github.com/spf13/cobra v1.8.0
 	github.com/stretchr/testify v1.9.0
+	github.com/test-go/testify v1.1.4
 	github.com/thejerf/suture/v4 v4.0.5
 	github.com/tidwall/gjson v1.17.1
 	github.com/tus/tusd v1.13.0

diff --git a/go.sum b/go.sum
@@ -1025,8 +1025,8 @@ github.com/crewjam/saml v0.4.14 h1:g9FBNx62osKusnFzs3QTN5L9CVA/Egfgm+stJShzw/c=
 github.com/crewjam/saml v0.4.14/go.mod h1:UVSZCf18jJkk6GpWNVqcyQJMD5HsRugBPf4I1nl2mME=
 github.com/cs3org/go-cs3apis v0.0.0-20231023073225-7748710e0781 h1:BUdwkIlf8IS2FasrrPg8gGPHQPOrQ18MS1Oew2tmGtY=
 github.com/cs3org/go-cs3apis v0.0.0-20231023073225-7748710e0781/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY=
-github.com/cs3org/reva/v2 v2.19.2-0.20240429085656-5faad8dad61e h1:4Z9nfRWM2VnTX0EOBaIv0Rn27Fw7VYqOrcCntXZ3b40=
-github.com/cs3org/reva/v2 v2.19.2-0.20240429085656-5faad8dad61e/go.mod h1:GRUrOp5HbFVwZTgR9bVrMZ/MvVy+Jhxw1PdMmhhKP9E=
+github.com/cs3org/reva/v2 v2.19.2-0.20240502102837-7e48a5145133 h1:zuqfsPUYjJjIiq4wx2xTDxlJH+Sh3H3m8ynSjbQ/e8w=
+github.com/cs3org/reva/v2 v2.19.2-0.20240502102837-7e48a5145133/go.mod h1:GRUrOp5HbFVwZTgR9bVrMZ/MvVy+Jhxw1PdMmhhKP9E=
 github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4=
 github.com/cyphar/filepath-securejoin v0.2.4 h1:Ugdm7cg7i6ZK6x3xDF1oEu1nfkyfH53EtKeQYTC3kyg=
 github.com/cyphar/filepath-securejoin v0.2.4/go.mod h1:aPGpWjXOXUn2NCNjFvBE6aRxGGx79pTxQpKOJNYHHl4=

diff --git a/services/storage-users/README.md b/services/storage-users/README.md
@@ -60,58 +60,90 @@ ocis storage-users uploads <command>
 ```plaintext
 COMMANDS:
    sessions   Print a list of upload sessions
-   clean      Clean up leftovers from expired uploads
+   clean      Clean up leftovers from expired uploads (deprecated)
    list       Print a list of all incomplete uploads (deprecated)
 ```
 
+#### Sessions command
+
+The `sessions` command is the entry point for listing, restarting and cleaning unfinished uploads.
+
+```bash
+ocis storage-users uploads sessions <command>
+```
+
+```
+NAME:
+   ocis storage-users uploads sessions - Print a list of upload sessions
+
+USAGE:
+   ocis storage-users uploads sessions [command options]
+
+OPTIONS:
+   --id value    filter sessions by upload session id (default: unset)
+   --processing  filter sessions by processing status (default: unset)
+   --expired     filter sessions by expired status (default: unset)
+   --has-virus   filter sessions by virus scan result (default: unset)
+   --json        output as json (default: false)
+   --restart     send restart event for all listed sessions (default: false)
+   --clean       remove uploads (default: false)
+   --help, -h    show help
+```
+
+This will always output a list of uploads that match the criteria. See Command Examples section.
+
+Some additional information on returned information:
+  -  `Offset` is the amount of bytes the server has already received. If `Offset` == `Size` the server has reveived all bytes of the upload.
+  -  `Processing` indicates if the uploaded file is currently going through postprocessing.
+  -  `Scan Date` and `Scan Result` indicate the scanning status. If `Scan Date` is set and `Scan Result` is empty the file is not virus infected.
+
 #### Command Examples
 
 Command to list ongoing upload sessions
 
 ```bash
-ocis storage-users sessions --expired=false
+ocis storage-users uploads sessions --expired=false --processing=false
 ```
 
 ```plaintext
 Not expired sessions:
-+--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+
-|                Space                 |              Upload Id               |  Name   | Offset | Size |              Executant               |                Owner                 |          Expires          | Processing |
-+--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+
-| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 5e387954-7313-4223-a904-bf996da6ec0b | foo.txt |      0 | 1234 | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2024-01-26T13:04:31+01:00 | false      |
-| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f066244d-97b2-48e7-a30d-b40fcb60cec6 | bar.txt |      0 | 4321 | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2024-01-26T13:18:47+01:00 | false      |
-+--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+
++--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+---------------------------+-----------------------+
+|                Space                 |              Upload Id               |  Name   | Offset | Size |              Executant               |                Owner                 |          Expires          | Processing |         Scan Date         |      Scan Result      |
++--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+---------------------------+-----------------------+
+| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 5e387954-7313-4223-a904-bf996da6ec0b | foo.txt |      0 | 1234 | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2024-01-26T13:04:31+01:00 | false      | 2024-04-24T11:24:14+02:00 |   infected: virus A   |
+| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f066244d-97b2-48e7-a30d-b40fcb60cec6 | bar.txt |      0 | 4321 | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2024-01-26T13:18:47+01:00 | false      | 2024-04-24T14:38:29+02:00 |                       |
++--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+---------------------------+-----------------------+
 ```
 
 The sessions command can also output json
 
 ```bash
-ocis storage-users sessions --expired=false --json
+ocis storage-users uploads sessions --expired=false --processing=false --json
 ```
 
 ```json
-{"id":"5e387954-7313-4223-a904-bf996da6ec0b","space":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","filename":"foo.txt","offset":0,"size":1234,"executant":{"idp":"https://cloud.ocis.test","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"spaceowner":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"expires":"2024-01-26T13:04:31+01:00","processing":false}
-{"id":"f066244d-97b2-48e7-a30d-b40fcb60cec6","space":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","filename":"bar.txt","offset":0,"size":4321,"executant":{"idp":"https://cloud.ocis.test","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"spaceowner":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"expires":"2024-01-26T13:18:47+01:00","processing":false}
+{"id":"5e387954-7313-4223-a904-bf996da6ec0b","space":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","filename":"foo.txt","offset":0,"size":1234,"executant":{"idp":"https://cloud.ocis.test","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"spaceowner":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"expires":"2024-01-26T13:04:31+01:00","processing":false, "scanDate": "2024-04-24T11:24:14+02:00", "scanResult": "infected: virus A"}
+{"id":"f066244d-97b2-48e7-a30d-b40fcb60cec6","space":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","filename":"bar.txt","offset":0,"size":4321,"executant":{"idp":"https://cloud.ocis.test","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"spaceowner":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"expires":"2024-01-26T13:18:47+01:00","processing":false, "scanDate": "2024-04-24T14:38:29+02:00", "scanResult": ""}
 ```
 
-Command to clear expired uploads
+The sessions command can also clear and restart uploads. The output is the same as if run without `--clean` or `--restart` flag.
+Note: It is recommended to run to command first without the `--clean` (`--processing`) flag to double check which uploads get cleaned (restarted).
 ```bash
-ocis storage-users uploads clean
-```
+# cleans all expired uploads regardless of processing and virus state.
+ocis storage-users uploads sessions --expired=true --clean
 
-```plaintext
-Cleaned uploads:
-- 455bd640-cd08-46e8-a5a0-9304908bd40a (Filename: file_example_PPT_1MB.ppt, Size: 1028608, Expires: 2022-08-17T12:35:34+02:00)
+# restarts all uploads that are processing and are not virus infected
+ocis storage-users uploads sessions --processing=false --has-virus=false --restart
 ```
 
-Deprecated list command to identify unfinished uploads
+IMPOTANT: `list` and `clean` commands are deprecated. Do not use them.
 
 ```bash
+# deprecated
 ocis storage-users uploads list
-```
 
-```plaintext
-Incomplete uploads:
- - 455bd640-cd08-46e8-a5a0-9304908bd40a (file_example_PPT_1MB.ppt, Size: 1028608, Expires: 2022-08-17T12:35:34+02:00)
+# deprecated
+ocis storage-users uploads clean
 ```
 
 ### Manage Trash-Bin Items