Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terragrunt performance improvements #3307

Merged
merged 23 commits into from
Aug 6, 2024
Merged

Terragrunt performance improvements #3307

merged 23 commits into from
Aug 6, 2024

Conversation

denis256
Copy link
Member

@denis256 denis256 commented Jul 31, 2024

Description

A couple of performance improvements by using in-memory cache of git execution and parsing of dependencies

Example runs on my test repos:
output

Included changes:

  • added caching of HCL parsing struct
  • added caching of git --show-toplevel
  • updated fetching of TerragruntConfig to do full clone of struct
  • replaced global variables cache with data from context
  • added metrics for cache miss/hits

Close: #2344

References: #2980

TODOs

Read the Gruntwork contribution guidelines.

  • Update the docs.
  • Run the relevant tests successfully, including pre-commit checks.
  • Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
  • Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added / Removed / Updated [X].

Migration Guide

@denis256 denis256 linked an issue Aug 5, 2024 that may be closed by this pull request
@denis256 denis256 marked this pull request as ready for review August 5, 2024 20:45
yhakbar
yhakbar previously approved these changes Aug 5, 2024
Copy link
Collaborator

@yhakbar yhakbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great Job!

levkohimins
levkohimins previously approved these changes Aug 6, 2024
Copy link
Contributor

@levkohimins levkohimins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! I left some comments, what to do with them is up to you.

config/cache_test.go Outdated Show resolved Hide resolved
config/context.go Outdated Show resolved Hide resolved
internal/cache/cache.go Outdated Show resolved Hide resolved
util/reflect.go Outdated Show resolved Hide resolved
@denis256 denis256 dismissed stale reviews from levkohimins and yhakbar via dbe6e82 August 6, 2024 16:20
yhakbar
yhakbar previously approved these changes Aug 6, 2024
Copy link

sonarcloud bot commented Aug 6, 2024

@denis256 denis256 merged commit 02ad89f into master Aug 6, 2024
4 checks passed
@denis256 denis256 deleted the 226-perf-improvements branch August 6, 2024 19:16
akrantz01 pushed a commit to akrantz01/homelab that referenced this pull request Aug 8, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
|
[gruntwork-io/terragrunt](https://github.com/gruntwork-io/terragrunt)
| patch | `v0.66.1` -> `v0.66.3` |
|
[gruntwork-io/terragrunt](https://github.com/gruntwork-io/terragrunt)
| patch | `0.66.1` -> `0.66.3` |

---

### Release Notes

<details>
<summary>gruntwork-io/terragrunt (gruntwork-io/terragrunt)</summary>

###
[`v0.66.3`](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.66.3)

[Compare
Source](https://github.com/gruntwork-io/terragrunt/compare/v0.66.2...v0.66.3)

#### Updated CLI args, config attributes and blocks

-   `dependency`
-   `get_repo_root`

#### Description

- Improved efficiency in evaluating dependencies and locating the git
repository root.

#### Related links

-
[gruntwork-io/terragrunt#3307

**Full Changelog**:
gruntwork-io/terragrunt@v0.66.2...v0.66.3

###
[`v0.66.2`](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.66.2)

[Compare
Source](https://github.com/gruntwork-io/terragrunt/compare/v0.66.1...v0.66.2)

#### What's Changed

- fix: Reference OpenTofu Everywhere We Can by
[@&#8203;yhakbar](https://github.com/yhakbar) in
[gruntwork-io/terragrunt#3319
- add env TERRAGRUNT_EXCLUDE_DIR to docs by
[@&#8203;pbrogan12](https://github.com/pbrogan12) in
[gruntwork-io/terragrunt#3320
- build(deps-dev): bump nokogiri from 1.16.3 to 1.16.5 in /docs by
[@&#8203;dependabot](https://github.com/dependabot) in
[gruntwork-io/terragrunt#3135
- build(deps-dev): bump rexml from 3.3.1 to 3.3.3 in /docs by
[@&#8203;dependabot](https://github.com/dependabot) in
[gruntwork-io/terragrunt#3316
- feat: Adding `TG_CTX_TF_PATH` to hook context by
[@&#8203;yhakbar](https://github.com/yhakbar) in
[gruntwork-io/terragrunt#3323

#### New Contributors

- [@&#8203;pbrogan12](https://github.com/pbrogan12) made their first
contribution in
[gruntwork-io/terragrunt#3320

**Full Changelog**:
gruntwork-io/terragrunt@v0.66.1...v0.66.2

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View the
[repository job
log](https://developer.mend.io/github/akrantz01/homelab).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC4xOC4xNyIsInVwZGF0ZWRJblZlciI6IjM4LjE4LjE3IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
aliscott added a commit to infracost/terragrunt that referenced this pull request Aug 23, 2024
Previously when using the PartialConfig cache, the previous entries were modified after they'd been added to the cache. This was causing modules to inherit any dependencies if they were included in other modules, which in turn caused circular depedency issues. This ensures the cache is cloned when it is read using logic similar to gruntwork-io#3307
aliscott added a commit to infracost/terragrunt that referenced this pull request Sep 9, 2024
Previously when using the PartialConfig cache, the previous entries were modified after they'd been added to the cache. This was causing modules to inherit any dependencies if they were included in other modules, which in turn caused circular depedency issues. This ensures the cache is cloned when it is read using logic similar to gruntwork-io#3307
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extremely poor performance in v0.55.5+ git show-toplevel should use caching
3 participants