Building a NET7/NET8 iOS project hangs with LLVM #20210
Comments
|
Forgot to mention it in the OP, but the hang issue seems to happen when |
takla21
changed the title
|
rolfbjarne
added
the
need-info
|
microsoft-github-policy-service
bot
added
need-attention
I think I expressed myself badly: sometimes it can look like a hang, but it's just timing out too early. For example: if the build typically takes 58-59 minutes, and the task times out after 1 hour, then it would seem the build hung if it occasionally hit the 1 hour mark, even though it would have passed if given a few more minutes. So my question is: how long is the task timeout for the build? |
rolfbjarne
added
need-info
So I attempted to build it locally, but it seemed quite fast, so I couldn't reproduce the same behavior.
I've never tried without the 1-hour timeout limit. I'll check if I can find a way to remove temporarily the timeout limit to check if it could be a timeout issue as you mentioned.
If it can help, what I meant here, is that I can see it hang/timeout at around 20minutes with the log provided above. Which would mean that it waits for like 40 minutes until the timeout threshold is triggered. |
microsoft-github-policy-service
bot
added
need-attention
rolfbjarne
added
need-info
|
@rolfbjarne so I removed the 1-hour timeout limit and you are right it's not hanging indefinitely as I thought earlier. Funny enough, it does seem that at times it can complete itself in less than an 1 hour which wasn't happening at the time I entered that issue. Looking at the binlog, it seems that it does waste most of its time in the AOTCompile task. I've enabled systems diagnostics on one of the runs and here are the logs near the end where it timeouts |
microsoft-github-policy-service
bot
added
need-attention
|
@takla21 it's expected that the AOT compiler is significantly slower when LLVM is enabled (2-3x slower is not surprising), so I'm not sure if there's anything we can do on our side? |
rolfbjarne
added
need-info
|
Lastly, I've tried to add
|
microsoft-github-policy-service
bot
added
need-attention
rolfbjarne
added
bug
|
So the log revealed an issue: we're running the AOT compiler in parallel, one process per assembly, but we're not limiting the concurrent number of processes, so the build runs a lot of AOT compilers in parallel. Obviously that's not optimal, so I'm working on a fix. In the meantime I guess the workaround is to just use a longer timeout for your build. |
|
Not sure if its related - but we've ran into similar long AOT builds of 40 minutes+ compared to 10 minutes before after we added a signed nuget package (so our Core project is now a signed project) - we don't know if this is the cause, we're just investigating. We've been trying to narrow down what sent our builds from 12mins to 40-60+, and we saw similar long 'pauses' during the AOT stage in the binlog. @rolfbjarne is what you're mentioning LLVM specific, or general AOT specific? We've also noticed we only get the long build times on Azure Mac13 images, running locally is fine - so we're also looking to just setup a local agent in the meantime to potentially work around the 'issue'. |
The issue I found is not specific to when LLVM is enabled, it's for any AOT compilation. However, the problem is that the machine can get overloaded with too many processes doing too much, and since LLVM is much more CPU intensitive than standard AOT compilation, it's entirely possible the problem won't really manifest in standard AOT compilation. |
|
Gotcha. WE've narrowed it down to a specific nuget we've added. We thought it might have been because we also have to sign our project to use the nuget, but turning off signing didn't change things, we think its just having the nuget in the project. We also don't see slow downs when running builds/publish on local machine. We'll keep an eye out and see if it resolves in the same way for us or not 👍 |
rolfbjarne
added a commit
to rolfbjarne/xamarin-macios
that referenced
this issue
Mar 5, 2024
…t call Parallel.ForEach. Fixes xamarin#20210. This deduplicates some code, and also ensures we specifiy a max limit to the level of parallelism. This fixes an issue in the AOTCompile and ScnTool tasks, where we'd have no limit, launching as many subprocesses as tasks there are to execute. In particular for the AOTCompile task, this can put a rather heavy burden on build machines, slowing everything down to a crawl. Fixes xamarin#20210.
|
Hi @rolfbjarne - would this change appear in |
I'm not sure how this shows up in these images, if the images have to be updated, or if the build will automatically get the newest versions.
The fix will be in .NET 9 for sure. It will probably be in a .NET 8 service release as well, but it's hard to tell at this point, because it depends on other things. In particular it might end up in the supporting release for Xcode 15.3, but I can only confirm when that release has actually been released. And at that point I can give you version numbers you can check on the vm images to see what your build ends up using.
The actions/runner-images repository has this information. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Steps to Reproduce
net8.0-android/net8.0-iosproject with the following csproj:Expected Behavior
The project should not hang while building.
Actual Behavior
When it targets
net7.0-ios, it hangs intermittently. See microsoft/azure-pipelines-agent#4677.When it targets
net8.0-ios, it always hangs with the following message:Environment
It's mostly happening on our Azure DevOps CI, which is using Microsoft hosted agents. I've tried with the following:
macos-12: https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.mdmacos-13: https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.mdThe text was updated successfully, but these errors were encountered: