-
-
Notifications
You must be signed in to change notification settings - Fork 12.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Several bottles resulting in "segmentation fault" on Red Hat Enterprise Linux release 8.6 #116841
Comments
I've seen some reports of this on WSL (for gcc-12 at least: #115895) but have so far not managed to reproduce. I can't reproduce this in a |
Cantos 8 isn't really RHEL, or supported. I'd recommend Alma Linux 8 or Rocky 8 |
Seems fine on AlmaLinux 8 at least:
I do recognise there is likely a problem somewhere. Though there seems to be a step to reproduce that's not obvious and not something I'm hitting on fresh installations. |
I'm not sure if the following helps or not. I thought I might try to zero in on if the segmentation faults were coming from a common dependency; however, the deps of curl include all those of ccat, and the former works just fine. Both curl and ccat were poured from bottles on my system. |
I thought that since I was having problems with the bottles, I'd go ahead and explore building some of these from source. At least two of these, R and util-linux, are failing during the build with statements such as the following.
|
Yeah you might be hitting segfaults when running |
Exactly as you guessed.
Luckily we keep snapshots, and I was able to rollback to a Linuxbrew image where most bottles were working well. This will be temporary, of course. |
Does running a broken binary with |
No debug output.
|
Can you try |
Unfortunately I do not have root on these systems. Root seems to be required by UPDATE: gdb was already on the system.
|
Can you upload that bad binary so I can compare with what I have? |
Hi, I'm trying to run brew on Redhat 8, hitting the exact same problem for a week or so (it was working before). Segfaults in gcc, binutils etc. I do have root access...anything I can do to help? Any updates? |
As I mentioned earlier, I've rolled back to a snapshot version of brew. That is version 3.6.11-5-g06b7573. I'm not getting seg faults with the bottles in this version. |
Curiously the versions of at least two of the applications that lead to seg faults are the same as before the seg faults started happening. more from util-linux 2.38.1 And the only differences between each of the respective latest and former formulas is the addition of the sha256 hash of the ventura versions in the latest. The seg faults seem to be coming from some common dependency; however, |
Another curious thing is that I get a bit different behavior on another machine with the same Homebrew version; although, I have less applications installed to it. The following is the config for what I will call Machine2, Machine1 having the config which I initially posted. The only major difference that I can see is that the linux kernels are different, but only minorly.
The curious behavior:
Machine2:
|
Would you please post your output of both |
I don't know enough about the more binary to know if the following is a valid test. I've copied the broken version from Machine1 to Machine2 (where the Linuxbrew-installed more works). The following was done on Machine2. Machine2 version below is the one in the Linuxbrew path.
Alternatively, if I copy the functional more binary from Machine2 to Machine1 and test there, it works:
Same behavior with ccat.
|
I can't reproduce in the RHEL UBI 8 Docker image, so I'm guessing this is something lower level (e.g. kernel). WSL2 had similar issues but it seems like the latest version fixed them (with newer kernel etc.). I'll try set up a full VM and see if I can hit the issue in that. |
Can you also upload the working binary so I can compare? Might also be worth comparing installation dates ("Poured from bottle" date in |
[kls@rhlibpubwdev01 ~]$ brew config [kls@rhlibpubwdev01 ~]$ brew dr |
@Bo98 , I really appreciate your time on this. I know that the Homebrew maintainers do so on a volunteer basis. This issue is not super critical to me as I state above that I was able to rollback to a functional version of which I'd taken a snapshot; however, I'd like to be able to have a functional up-to-date Homebrew at some point in the near future. Please let me know if there is anything else that I can do to help. I'm not sure how we'll discern anything from when the bottles were poured. Aren't those times kind of arbitrary? Wouldn't the Homebrew versions which I've posted above be more relevant? Anyhow, I'm posting the pour times as you requested. I'm also attaching the zipped functional more from util-linux on Machine2. Pour times: |
The version of patchelf used changed in that timeframe. I'm guessing we're patching something about the ELFs in a way that only breaks on very specific Linux distributions. |
@Bo98 , my situation is more urgent...currently trying to provision 3 new servers using Brew via a script. I developed script, all worked, now I'm trying to use it in anger on 3 'clean' servers and it fails. Is there anything I can do to expedite this issue? Is there some way I can roll brew back to a version that works? Also this may or may not be relevant, but when I was trying to find a workaround and build some packages from source, I would occasionally run into an ELF bug...I suspected this might be in play https://bugzilla.redhat.com/show_bug.cgi?id=1678204 For what its worth, i was able to download gcc, for example, from git repo, and compile / build manually without brew...that worked. But trying to to install with brew --build-from-source did not. |
We can't really help on an urgent basis. One thing to possibly try to is to try with old versions of patchelf.rb (and disable Homebrew from requiring the newest version of such by e.g. setting the versioned directories appropriately appropriately). If this works then we know it's a patchelf/patchelf.rb bug |
I have the following ./Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/patchelf-1.4.0/lib/patchelf.rb I do have an install on a machine from October that works...which files should I replace and what do you mean exactly by 'setting the versioned directories appropriately' ? |
I don't know what files would need to be replaced, since I can't reproduce this issue. You would have to do this experimentation on your own. |
as a sanity check tried to reproduce on a freshly spun up EC2 running Red Hat Enterprise Linux 8 image cat /etc/redhat-release I did sudo yum install git made sure my ec2 user had access to /home/linuxbrew/ installed brew go the exact same error as I've been getting on my other server: ==> Pouring gcc--12.2.0.x86_64_linux.bottle.2.tar.gz and gcc-12 and other gcc binaries just segfault with all efforts to use them tried brew install gcc --build-from-source --debug -v error is gcc: internal compiler error: Segmentation fault signal terminated program as |
@jwhite007, maybe try |
I don't know how long it'll take until someone who has this issue contacts the patchelf developers, works with them to find and resolve the bug, and for the patchelf developers to release that fix. |
I'm looking into it as soon as I can. Hardware issues mean I don't have access to my usual setup, but I'm looking at WSL1 to see what the issues are there which might be liked to this one. I don't know if a fix will be released by Christmas but I'm hoping a possible fix will be identified by then. The issues seem to affect very specific setups, particularly given the issues on the WSL side were fixed recently by Microsoft. Note that the older patchelf does have issues that are more widespread, like qt installs being broken. It'll still be a better situation for your particular case though. |
Thanks. However, you probably meant to address @dbroadfoot in your comment above. I had rolled back to a functional Linuxbrew from a snapshot I had taken. |
As I stated, I was able to get a functional Linuxbrew from a snapshot that I had, and it seems that @dbroadfoot was able to revert to a commit with a functional Linuxbrew, so I think we're both probably good for the next several weeks. Please enjoy the holidays and don't feel as if you guys need to fix anything before Christmas or the New Year even. I'm most appreciative that Linuxbrew is even a thing. Again, if there is anything else with which I can assist in the matter, please let me know. |
I'm not sure if the setup we use will work with your particular framework; however, our production-based stuff is only dependent on one particular tool installed with Linuxbrew, and that is pyenv. We install pyenv via Linuxbrew and then miniconda via pyenv. We then have all of our production pipelines working within locked-down, stable Conda environments. Back in Linuxbrew, I extract all Formulas for pyenv and all of its dependencies. Then when I go to upgrade via Linuxbrew or install something to Linuxbrew that might upgrade one of pyenv's dependencies, I first upgrade anything pyenv related and check for pyenv functionality before proceeding. In this way, even though a lot of applications in Linuxbrew were seg faulting in this latest update, I had ensured that pyenv was still solid, thus ensuring that all of our production code was still functional. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Commenting to keep issue active. |
There's no need to keep it active for the bot if nobody is doing anything with it. |
@SMillerDev, I thought that this was still being worked on. The bot marked it as stale and then subsequently closed as "not planned." Shall we assume that Linuxbrew will not support RHEL going forward? Thanks. |
Since it doesn't affect all RHEL systems, that seems a bit premature. But seeing how it does seem to affect your configuration that assumption probably puts you on the safer side of this problem. |
So from here on out, determining whether Linuxbrew will support a particular Linux will just be hit or miss? I mean we really don't know why my configuration and that of @dbroadfoot are not working. Right? It's also curious that Linuxbrew had been working just fine for almost a year on my configuration. I guess I'd really like to figure out exactly what it is about our configurations that are screwing things up. Any idea on how I would go about determining that? Is it the version of RHEL? Is it the version of the Linux kernel? |
#116841 (comment) suggests it's a bug in patchelf, but without anyone pursuing that I guess we'll never know. |
As an update: I've identified a possible issue, but haven't actually managed to reproduce the issue yet to test though. I'll update back here again next week. |
@Bo98, let me know if I can help with reproduction or testing on a different architecture (ARM Linux) that also went pear-shaped after the patchelf 1.4.0 upgrade. I suddenly saw the following error when using previously-built relocatable ARM Linux bottles:
so I had to rebuild all my bottles with |
Just chiming in that I'm seeing a lot of the same issues:
The revert from #116841 (comment) didn't work for me -- I probably didn't resolve the conflict correctly; I instead did |
It appears that there has been some work in fixing a related bug in patchelf, NixOS/patchelf#447. According to that pull request, 0.17.2 has been released to address the related bug. Does anyone know if this latest version of patchelf is now being used for Linuxbrew bottles? |
Nope. Homebrew uses a different pure-Ruby |
For me this is a post-4.0.0 priority: so next week. I've got this, WSL1 and upx noted to investigate. |
brew gist-logs <formula>
link ORbrew config
ANDbrew doctor
outputThe text was updated successfully, but these errors were encountered: