Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove GCP dependencies #1440

Merged
merged 15 commits into from
Feb 27, 2024
Merged

Remove GCP dependencies #1440

merged 15 commits into from
Feb 27, 2024

Conversation

jleaniz
Copy link
Collaborator

@jleaniz jleaniz commented Feb 19, 2024

Description of the change

This PR removes deprecated Google PSQ/Pubsub code and associated unit tests.

Applicable issues

Additional information

Documentation should be updated on a separate PR.

Checklist

  • All tests were successful.
  • Unit tests added.
  • Documentation updated.

@jleaniz jleaniz changed the title Remove gcp dependencies Remove GCP dependencies Feb 19, 2024
@jleaniz jleaniz self-assigned this Feb 19, 2024
@jleaniz
Copy link
Collaborator Author

jleaniz commented Feb 19, 2024

New error reporting code seems to work after testing on a GCP instance

@jleaniz jleaniz marked this pull request as ready for review February 19, 2024 20:51
Copy link
Collaborator

@hacktobeer hacktobeer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonderful to see so much code removed!

From the functional side: have a look at my comments and see if this version would still work in cloud where we need to mount GCP disks.

pyproject.toml Show resolved Hide resolved
turbinia/lib/google_cloud.py Show resolved Hide resolved
@hacktobeer hacktobeer self-requested a review February 20, 2024 15:51
Copy link
Collaborator

@hacktobeer hacktobeer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, as per online sync

@aarontp
Copy link
Member

aarontp commented Feb 21, 2024

Wow, that's great, yeah nice to see this getting cleaned out, thanks! Will take a closer look at the PR tomorrow.

@jleaniz
Copy link
Collaborator Author

jleaniz commented Feb 21, 2024

Manual testing in GCP, Github e2e and Jenkins e2e tests pass.

Copy link
Member

@aarontp aarontp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one small comment about potentially using a more narrowly scoped exception.

So good to see all this old code and especially PSQ and the other dependencies go away!

turbinia/evidence.py Show resolved Hide resolved
turbinia/lib/google_cloud.py Outdated Show resolved Hide resolved
@jleaniz jleaniz merged commit d8c7377 into google:master Feb 27, 2024
5 checks passed
jleaniz added a commit to jleaniz/turbinia that referenced this pull request Feb 28, 2024
* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint
jleaniz added a commit to jleaniz/turbinia that referenced this pull request Mar 18, 2024
* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint
jleaniz added a commit to jleaniz/turbinia that referenced this pull request Mar 18, 2024
* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint
jleaniz added a commit to jleaniz/turbinia that referenced this pull request Apr 3, 2024
* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint
jleaniz added a commit to jleaniz/turbinia that referenced this pull request May 30, 2024
* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint
jleaniz added a commit that referenced this pull request Jun 13, 2024
* Added the TurbiniaRequest hashed object in Redis

* Converted TurbiniaTasks to hash objects in Redis

* Made get_request_data more efficient

* Made get_requests_summary more efficient

* Fixed some issues in request_status

* Remove GCP dependencies (#1440)

* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint

* Lint fixes

* Updates

* Updates

* Various updates and fixes

* Updates

* --- (#1483)

updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updates

* Update typo

* Updates to unit tests

* Updates to unit tests and linter fixes

* Update table width UI

* Bump the pip group across 2 directories with 1 update (#1487)

Bumps the pip group with 1 update in the / directory: [requests](https://github.com/psf/requests).
Bumps the pip group with 1 update in the /turbinia/api/cli directory: [requests](https://github.com/psf/requests).


Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix small UI bug

* Minor typos/errors

* Set celery task limits and fix worker timeout (#1486)

* Set celery task limits and fix worker timeout

* Remove solo pool and change concurrency=1

* Add soft/hard limit buffers

* Fix server timeout

* Late import for psutil

* small fixes, tests

* fix run tests

* test format string

* Fix process_result test

* fix execute test

* revert unnecessary config changes

* Handle soft timeout exception in execute()

* update timeout message

* Yaaaaaaaapf

* Review fixes

* Updates and yapf fix

* Minor updates

* Update docstrings

* Updates to evidence_size

* Updates

* Change log level for message

* Lint

* bug fixes

* Address review comments

* Fix docstrings

* Minor UI update

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Igor Rodrigues <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Aaron Peterson <[email protected]>
jleaniz added a commit that referenced this pull request Jun 13, 2024
* Added the TurbiniaRequest hashed object in Redis

* Converted TurbiniaTasks to hash objects in Redis

* Made get_request_data more efficient

* Made get_requests_summary more efficient

* Fixed some issues in request_status

* Remove GCP dependencies (#1440)

* Remove gcp dependencies

* Update dockerfiles

* Update dockerfiles

* Update gcp error reporting

* Updates to formatting

* Add unit test

* Update unit test

* Clean up

* Update unit test

* Update error reporting

* Update file

* Update config template

* Catch exception

* Updates

* fix lint

* Lint fixes

* Updates

* Updates

* Various updates and fixes

* Updates

* --- (#1483)

updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...




* Updates

* Update typo

* Updates to unit tests

* Updates to unit tests and linter fixes

* Update table width UI

* Bump the pip group across 2 directories with 1 update (#1487)

Bumps the pip group with 1 update in the / directory: [requests](https://github.com/psf/requests).
Bumps the pip group with 1 update in the /turbinia/api/cli directory: [requests](https://github.com/psf/requests).


Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

Updates `requests` from 2.32.0 to 2.32.3
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.3)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...




* Fix small UI bug

* Minor typos/errors

* Set celery task limits and fix worker timeout (#1486)

* Set celery task limits and fix worker timeout

* Remove solo pool and change concurrency=1

* Add soft/hard limit buffers

* Fix server timeout

* Late import for psutil

* small fixes, tests

* fix run tests

* test format string

* Fix process_result test

* fix execute test

* revert unnecessary config changes

* Handle soft timeout exception in execute()

* update timeout message

* Yaaaaaaaapf

* Review fixes

* Updates and yapf fix

* Minor updates

* Update docstrings

* Updates to evidence_size

* Updates

* Change log level for message

* Lint

* bug fixes

* Address review comments

* Fix docstrings

* Minor UI update

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Igor Rodrigues <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Aaron Peterson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Deprecate PSQ Increase PSQ Timeout
3 participants