Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write Customization tests in python #2918

Merged
merged 18 commits into from
Mar 26, 2024

Conversation

hubertdeng123
Copy link
Member

@hubertdeng123 hubertdeng123 commented Mar 21, 2024

This PR adds a few things

  1. Codecov reporting for self-hosted
  2. Proper flaky test detection for all of integration tests in self-hosted with pytest-sentry
  3. All integration tests are now running through pytest

Copy link

codecov bot commented Mar 22, 2024

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@hubertdeng123 hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from 11f9bea to 5c2fd16 Compare March 22, 2024 22:19
@hubertdeng123 hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from c93b5e5 to 2ccd1a5 Compare March 22, 2024 22:43
@hubertdeng123 hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from bcdccaa to 82eed1e Compare March 25, 2024 16:30
@hubertdeng123 hubertdeng123 force-pushed the hubertdeng123/customization-tests-python branch from a4f67e4 to 1799f09 Compare March 25, 2024 19:30
@hubertdeng123 hubertdeng123 marked this pull request as ready for review March 25, 2024 21:38
@@ -16,7 +16,7 @@ defaults:
jobs:
e2e-test:
if: github.repository_owner == 'getsentry'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bumped this while at it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Imo we should split this into a separate PR, since I could imagine cases where we want to only roll this bump back.

### pytest-sentry configuration ###
if [ "$GITHUB_REPOSITORY" = "getsentry/self-hosted" ]; then
echo "PYTEST_SENTRY_DSN=${{ env.SELF_HOSTED_TESTING_DSN }}" >> $GITHUB_ENV
echo "PYTEST_SENTRY_DSN=$SELF_HOSTED_TESTING_DSN" >> $GITHUB_ENV
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was previously not being picked up correctly

time.sleep(1)
else:
if response.status_code == 200:
break
else:
raise AssertionError("timeout waiting for self-hosted to come up")

if request.config.getoption("--customizations") == "True":
os.environ['TEST_CUSTOMIZATIONS'] = "True"
script_content = '''\
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines are definitely covered by testing, not sure why Codecov is complaining here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the reason why Test / integration test (True, v2.0.1) and Test / integration test (True, v2.7.0) are taking 8 minutes longer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It's pretty hard for a naive observer to figure out what the "True" in these variants means. Maybe we could have values be Customization Enabled vs Customization Disabled rather than True/False?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently, you can create a custom name for the job. Now it looks like
Test / integration test v2.0.1 - customizations disabled

@@ -16,7 +16,7 @@ defaults:
jobs:
e2e-test:
if: github.repository_owner == 'getsentry'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Imo we should split this into a separate PR, since I could imagine cases where we want to only roll this bump back.

time.sleep(1)
else:
if response.status_code == 200:
break
else:
raise AssertionError("timeout waiting for self-hosted to come up")

if request.config.getoption("--customizations") == "True":
os.environ['TEST_CUSTOMIZATIONS'] = "True"
script_content = '''\
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It's pretty hard for a naive observer to figure out what the "True" in these variants means. Maybe we could have values be Customization Enabled vs Customization Disabled rather than True/False?

for command in commands:
result = subprocess.run(command, check=False)
if os.getenv("TEST_CUSTOMIZATIONS", "False") == "True":
assert result.returncode == 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does test_customization run in both matrix variants?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does. If customizations enabled, it checks to ensure that they are truly enabled by making sure a dependency can be imported that is included in the custom image (ldap). If not, it checks to ensure that an error is thrown when trying to import a dependency that comes from the custom image.

@hubertdeng123 hubertdeng123 merged commit 9b3b9bc into master Mar 26, 2024
12 checks passed
@hubertdeng123 hubertdeng123 deleted the hubertdeng123/customization-tests-python branch March 26, 2024 18:14
@github-actions github-actions bot locked and limited conversation to collaborators Apr 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants