Slow/exploding build times with new workload version 8078 #20848
Assignees
Milestone
Comments
|
As discussed over Discord, the regression comes from #20687 With deduplication disabled, the size regression is expected. However, we should investigate if we can do something about long build times when this optimization is disabled. /cc: @kotlarmilos |
|
I too see issues with this new workload version. IPAs generated go up 27MB in size and crash when launched. Our prior version was 17.2.8053 and we were running 8.0.100 dotnet. This is for our pipeline builds using dotnet publish. |
|
I'm more concerned that workload versions can change when the .net version does not. This creates pipeline build inconsistencies if workload manifest can be changed on a whim for existing .NET versions |
As a result of the current behaviour, we've gone and updated our current build script, namely the "dotnet workload install " step: The --from-rollback-file parameter is new and should guarantee that the silent workload updates won't be pulled in by our automated builds. |
That's how workloads were designed: to keep you up-to-date with the latest version. As other comments have said, there's a way to install a specific workload version using a rollback file (see https://github.com/dotnet/sdk/blob/main/documentation/general/workloads/workload-rollback.md for more info). |
Please file a new issue for the crash so that we can investigate and try to find out happened. |
|
Interesting! I've just ran into an app start crash this week/end of last week. I thought it was linked to a cert in our apple developer account (one just happened to expire this week but I didn't think it was related). Might have ran into the same issue so will be try a workload rollback tomorrow and see if that 'fixes' the issue. Builds last week started crashing, the week before was fine. We've had huge AOT times (and near 50% app size increase) since porting from Xamarin to .net around March time I think - I might try the deduplication fix on discord too and see if that resolves our large app size + long aot time too |
|
Can't see a seperate issue for crashing so I'm going to raise one now (feel free to delete if there is one and I just couldn't see it). |
14 tasks
This was referenced Jul 25, 2024
This was referenced Jul 25, 2024
dalexsoto
pushed a commit
that referenced
this issue
Jul 25, 2024
## Description As part of the fix for: dotnet/runtime#99248 we disabled dedup optimization in partial/hybrid AOT mode (when both interpreter and AOT compiler are enabled). This change got backported to .NET 8 and with the latest servicing release regressed build times and app sizes significantly as reported in: #20848 However, it turns out that disabling dedup optimization is not required to fix dotnet/runtime#99248 but instead we should correct the Xamarin SDK integration with this optimization which this PR is doing. The following section describes the initial problem in more details. ## Overview of AOT modes and dedup optimization When the repro project from dotnet/runtime#99248 is built with dedup enabled in hybrid AOT+interpreter mode, the app crashes with: ``` 024-07-23 14:32:37.524110+0200 IvansApp[12711:20244208] debug: AOT NOT FOUND: (wrapper other) object:gsharedvt_out_sig (intptr). 2024-07-23 14:32:37.524120+0200 IvansApp[12711:20244208] error: * Assertion at /Users/ivan/repos/runtime-mono-iOS/src/mono/mono/mini/interp/interp.c:2667, condition `is_ok (error)' not met, function:init_jit_call_info, Attempting to JIT compile method '(wrapper other) void object:gsharedvt_out_sig (intptr)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. ``` To track down why these wrappers which are used to transition from interpreter to AOT code, are not generated we need to understand when they are compiled in different AOT modes with and without dedup optimization enabled: - In full AOT setup - all assemblies AOT compiled - `gsharedvt_out_sig` methods are never generated - In hybrid AOT + interpreter setup - all assemblies AOT compiled: `MtouchInterpreter=-all` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT images of every assembly (to enable interpreter calling into each specific assembly - here wrappers with same signatures are duplicated) - Dedup ON: - `gsharedvt_out_sig` methods are generated only in `aot-instances` AOT image - during AOT compilation of individual assemblies generation of `gsharedvt_out_sig` is skipped - during AOT compilation of `aot-instances` assembly we collect all `gsharedvt_out_sig` variants from the full program scope and generate code for them in `aot-instances` AOT image - In hybrid AOT + interpreter setup - all assemblies interpreted except a given assembly: `MtouchInterpreter=all,-MyAssembly` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT image of `MyAssembly` (to enable interpreter calling into it) - Dedup ON: <- $${\color{red} ISSUE }$$ - `gsharedvt_out_sig` methods *should* be generated only in `aot-instances` AOT image, but the `aot-instances` image is missing - explanation: - what happens is that generation of `gsharedvt_out_sig` is skipped during AOT compilation of `MyAssembly` (as expected). - But, the build does not mark `aot-instances` assembly as the one that should be AOT compiled. - The reason for this is that we have a global `_IsDedupEnabled` flag, but when custom linker step analysis `aot-instances.dll` it does not see it as an assembly which should not be interpreted. - To explain that better: we mark *all* assemblies as to be interpreted (via: `MtouchInterpreter=all`), but exclude only `MyAssembly` (via: `MtouchInterpreter=all,-MyAssembly`). - So when custom linker step processes `aot-instaces.dll` it treats it as an assembly to be interpreted, so it does not mark it for AOT compilation. - This further results with `aot-instances` AOT image missing, and all the methods which we skipped during AOT compilation never get generated. ## The fix To fix this and address regressions reported in: #20848 we are reenabling dedup optimization whenever AOT compilation is requested and fixing the issue where the custom linker step for generating AOT parameters always treates the dedup assembly as the one to be AOTed. Once approved this should be backported to .NET 8 as servicing releases are also affected with it. --------- Co-authored-by: GitHub Actions Autoformatter <[email protected]>
dalexsoto
added a commit
that referenced
this issue
Jul 25, 2024
…odes (#20940) ## Description As part of the fix for: dotnet/runtime#99248 we disabled dedup optimization in partial/hybrid AOT mode (when both interpreter and AOT compiler are enabled). This change got backported to .NET 8 and with the latest servicing release regressed build times and app sizes significantly as reported in: #20848 However, it turns out that disabling dedup optimization is not required to fix dotnet/runtime#99248 but instead we should correct the Xamarin SDK integration with this optimization which this PR is doing. The following section describes the initial problem in more details. ## Overview of AOT modes and dedup optimization When the repro project from dotnet/runtime#99248 is built with dedup enabled in hybrid AOT+interpreter mode, the app crashes with: ``` 024-07-23 14:32:37.524110+0200 IvansApp[12711:20244208] debug: AOT NOT FOUND: (wrapper other) object:gsharedvt_out_sig (intptr). 2024-07-23 14:32:37.524120+0200 IvansApp[12711:20244208] error: * Assertion at /Users/ivan/repos/runtime-mono-iOS/src/mono/mono/mini/interp/interp.c:2667, condition `is_ok (error)' not met, function:init_jit_call_info, Attempting to JIT compile method '(wrapper other) void object:gsharedvt_out_sig (intptr)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. ``` To track down why these wrappers which are used to transition from interpreter to AOT code, are not generated we need to understand when they are compiled in different AOT modes with and without dedup optimization enabled: - In full AOT setup - all assemblies AOT compiled - `gsharedvt_out_sig` methods are never generated - In hybrid AOT + interpreter setup - all assemblies AOT compiled: `MtouchInterpreter=-all` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT images of every assembly (to enable interpreter calling into each specific assembly - here wrappers with same signatures are duplicated) - Dedup ON: - `gsharedvt_out_sig` methods are generated only in `aot-instances` AOT image - during AOT compilation of individual assemblies generation of `gsharedvt_out_sig` is skipped - during AOT compilation of `aot-instances` assembly we collect all `gsharedvt_out_sig` variants from the full program scope and generate code for them in `aot-instances` AOT image - In hybrid AOT + interpreter setup - all assemblies interpreted except a given assembly: `MtouchInterpreter=all,-MyAssembly` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT image of `MyAssembly` (to enable interpreter calling into it) - Dedup ON: <- $${\color{red} ISSUE }$$ - `gsharedvt_out_sig` methods *should* be generated only in `aot-instances` AOT image, but the `aot-instances` image is missing - explanation: - what happens is that generation of `gsharedvt_out_sig` is skipped during AOT compilation of `MyAssembly` (as expected). - But, the build does not mark `aot-instances` assembly as the one that should be AOT compiled. - The reason for this is that we have a global `_IsDedupEnabled` flag, but when custom linker step analysis `aot-instances.dll` it does not see it as an assembly which should not be interpreted. - To explain that better: we mark *all* assemblies as to be interpreted (via: `MtouchInterpreter=all`), but exclude only `MyAssembly` (via: `MtouchInterpreter=all,-MyAssembly`). - So when custom linker step processes `aot-instaces.dll` it treats it as an assembly to be interpreted, so it does not mark it for AOT compilation. - This further results with `aot-instances` AOT image missing, and all the methods which we skipped during AOT compilation never get generated. ## The fix To fix this and address regressions reported in: #20848 we are reenabling dedup optimization whenever AOT compilation is requested and fixing the issue where the custom linker step for generating AOT parameters always treates the dedup assembly as the one to be AOTed. Once approved this should be backported to .NET 8 as servicing releases are also affected with it. Backport of #20936 --------- Co-authored-by: Ivan Povazan <[email protected]> Co-authored-by: GitHub Actions Autoformatter <[email protected]>
dalexsoto
added a commit
that referenced
this issue
Jul 26, 2024
Backport of #20936 --- ## Description As part of the fix for: dotnet/runtime#99248 we disabled dedup optimization in partial/hybrid AOT mode (when both interpreter and AOT compiler are enabled). This change got backported to .NET 8 and with the latest servicing release regressed build times and app sizes significantly as reported in: #20848 However, it turns out that disabling dedup optimization is not required to fix dotnet/runtime#99248 but instead we should correct the Xamarin SDK integration with this optimization which this PR is doing. The following section describes the initial problem in more details. ## Overview of AOT modes and dedup optimization When the repro project from dotnet/runtime#99248 is built with dedup enabled in hybrid AOT+interpreter mode, the app crashes with: ``` 024-07-23 14:32:37.524110+0200 IvansApp[12711:20244208] debug: AOT NOT FOUND: (wrapper other) object:gsharedvt_out_sig (intptr). 2024-07-23 14:32:37.524120+0200 IvansApp[12711:20244208] error: * Assertion at /Users/ivan/repos/runtime-mono-iOS/src/mono/mono/mini/interp/interp.c:2667, condition `is_ok (error)' not met, function:init_jit_call_info, Attempting to JIT compile method '(wrapper other) void object:gsharedvt_out_sig (intptr)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. ``` To track down why these wrappers which are used to transition from interpreter to AOT code, are not generated we need to understand when they are compiled in different AOT modes with and without dedup optimization enabled: - In full AOT setup - all assemblies AOT compiled - `gsharedvt_out_sig` methods are never generated - In hybrid AOT + interpreter setup - all assemblies AOT compiled: `MtouchInterpreter=-all` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT images of every assembly (to enable interpreter calling into each specific assembly - here wrappers with same signatures are duplicated) - Dedup ON: - `gsharedvt_out_sig` methods are generated only in `aot-instances` AOT image - during AOT compilation of individual assemblies generation of `gsharedvt_out_sig` is skipped - during AOT compilation of `aot-instances` assembly we collect all `gsharedvt_out_sig` variants from the full program scope and generate code for them in `aot-instances` AOT image - In hybrid AOT + interpreter setup - all assemblies interpreted except a given assembly: `MtouchInterpreter=all,-MyAssembly` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT image of `MyAssembly` (to enable interpreter calling into it) - Dedup ON: <- $${\color{red} ISSUE }$$ - `gsharedvt_out_sig` methods *should* be generated only in `aot-instances` AOT image, but the `aot-instances` image is missing - explanation: - what happens is that generation of `gsharedvt_out_sig` is skipped during AOT compilation of `MyAssembly` (as expected). - But, the build does not mark `aot-instances` assembly as the one that should be AOT compiled. - The reason for this is that we have a global `_IsDedupEnabled` flag, but when custom linker step analysis `aot-instances.dll` it does not see it as an assembly which should not be interpreted. - To explain that better: we mark *all* assemblies as to be interpreted (via: `MtouchInterpreter=all`), but exclude only `MyAssembly` (via: `MtouchInterpreter=all,-MyAssembly`). - So when custom linker step processes `aot-instaces.dll` it treats it as an assembly to be interpreted, so it does not mark it for AOT compilation. - This further results with `aot-instances` AOT image missing, and all the methods which we skipped during AOT compilation never get generated. ## The fix To fix this and address regressions reported in: #20848 we are reenabling dedup optimization whenever AOT compilation is requested and fixing the issue where the custom linker step for generating AOT parameters always treates the dedup assembly as the one to be AOTed. Once approved this should be backported to .NET 8 as servicing releases are also affected with it. --------- Co-authored-by: Ivan Povazan <[email protected]> Co-authored-by: GitHub Actions Autoformatter <[email protected]>
|
This was fixed in #20936, and will be included in the next service release. |
mcumming
pushed a commit
that referenced
this issue
Aug 1, 2024
## Description As part of the fix for: dotnet/runtime#99248 we disabled dedup optimization in partial/hybrid AOT mode (when both interpreter and AOT compiler are enabled). This change got backported to .NET 8 and with the latest servicing release regressed build times and app sizes significantly as reported in: #20848 However, it turns out that disabling dedup optimization is not required to fix dotnet/runtime#99248 but instead we should correct the Xamarin SDK integration with this optimization which this PR is doing. The following section describes the initial problem in more details. ## Overview of AOT modes and dedup optimization When the repro project from dotnet/runtime#99248 is built with dedup enabled in hybrid AOT+interpreter mode, the app crashes with: ``` 024-07-23 14:32:37.524110+0200 IvansApp[12711:20244208] debug: AOT NOT FOUND: (wrapper other) object:gsharedvt_out_sig (intptr). 2024-07-23 14:32:37.524120+0200 IvansApp[12711:20244208] error: * Assertion at /Users/ivan/repos/runtime-mono-iOS/src/mono/mono/mini/interp/interp.c:2667, condition `is_ok (error)' not met, function:init_jit_call_info, Attempting to JIT compile method '(wrapper other) void object:gsharedvt_out_sig (intptr)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. ``` To track down why these wrappers which are used to transition from interpreter to AOT code, are not generated we need to understand when they are compiled in different AOT modes with and without dedup optimization enabled: - In full AOT setup - all assemblies AOT compiled - `gsharedvt_out_sig` methods are never generated - In hybrid AOT + interpreter setup - all assemblies AOT compiled: `MtouchInterpreter=-all` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT images of every assembly (to enable interpreter calling into each specific assembly - here wrappers with same signatures are duplicated) - Dedup ON: - `gsharedvt_out_sig` methods are generated only in `aot-instances` AOT image - during AOT compilation of individual assemblies generation of `gsharedvt_out_sig` is skipped - during AOT compilation of `aot-instances` assembly we collect all `gsharedvt_out_sig` variants from the full program scope and generate code for them in `aot-instances` AOT image - In hybrid AOT + interpreter setup - all assemblies interpreted except a given assembly: `MtouchInterpreter=all,-MyAssembly` - Dedup OFF: - `gsharedvt_out_sig` methods are generated in AOT image of `MyAssembly` (to enable interpreter calling into it) - Dedup ON: <- $${\color{red} ISSUE }$$ - `gsharedvt_out_sig` methods *should* be generated only in `aot-instances` AOT image, but the `aot-instances` image is missing - explanation: - what happens is that generation of `gsharedvt_out_sig` is skipped during AOT compilation of `MyAssembly` (as expected). - But, the build does not mark `aot-instances` assembly as the one that should be AOT compiled. - The reason for this is that we have a global `_IsDedupEnabled` flag, but when custom linker step analysis `aot-instances.dll` it does not see it as an assembly which should not be interpreted. - To explain that better: we mark *all* assemblies as to be interpreted (via: `MtouchInterpreter=all`), but exclude only `MyAssembly` (via: `MtouchInterpreter=all,-MyAssembly`). - So when custom linker step processes `aot-instaces.dll` it treats it as an assembly to be interpreted, so it does not mark it for AOT compilation. - This further results with `aot-instances` AOT image missing, and all the methods which we skipped during AOT compilation never get generated. ## The fix To fix this and address regressions reported in: #20848 we are reenabling dedup optimization whenever AOT compilation is requested and fixing the issue where the custom linker step for generating AOT parameters always treates the dedup assembly as the one to be AOTed. Once approved this should be backported to .NET 8 as servicing releases are also affected with it. --------- Co-authored-by: GitHub Actions Autoformatter <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The build-pipeline for the iOS version of our MAUI app unexplicably exploded in terms of build times and output size this tuesday.
What took around 20 minutes on monday now takes 70-80 minutes and the outputted ipa grew from 60 MB to ca. 90 MB.
There were no changes/checkins to (NuGet)dependencies, *.csproj, *.props, etc. files that could/should affect the build process inbetween these dates. It's virtually the same solution now taking three times longer.
On closer inspection of the build logs, the most prominent changes seemed to be in the automatically installed workloads by the command
dotnet workload install ios maui --source https://api.nuget.org/v3/index.jsonMonday:
Tuesday:
Adding an additional
--from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.61.jsonto the workload install command reverts to Monday's log output and decreases build time & output.I'm sorry if I missed any announcements where new, drastical AOT changes were discussed, but the recent release notes only mention a "dedup optimization" which doesn't sound like something that should cause this situation, right?
Also: The linked PR mentions "FullAOT" mode, which we don't use. the relevant csproj-lines from our release-configuration are:
Steps to Reproduce
Expected Behavior
The build takes a "reasonable" amount of time to finish
Actual Behavior
The build takes 3 times longer for no discernable reason
Environment
Build Logs
msbuild_workload_17.2.8022_redacted.binlog.zip
msbuild_workload_17.2.8078_redacted.binlog.zip
Example Project (If Possible)
Sorry, not possible I'm afraid, but I will reveal as much config/code you need, that I can.
The text was updated successfully, but these errors were encountered: