TROOT::EndOfProcessCleanups fails when using TCMalloc on different destructors
#13429
Comments
hahnjo
added
the
experiment
|
Hi @aandvalenzuela, unfortunately I am not allowed to run the reproducer: Any chance there can be "public" reproducer? |
|
Hi @hahnjo, Let me know if that works for you. Sorry for the inconvenience, |
|
That command gives me errors: outside of Singularity and inside: Note that I'm not a CMS member, so maybe I'm missing some permissions? |
|
@smuzaffar is there an equivalent of doing |
SITECONFIG_PATH=/cvmfs/cms-ib.cern.ch/SITECONF/local cmsRun ...should do the job |
|
Unfortunately it still doesn't work for me, neither setting |
|
@hahnjo , what do you get when run the following cmmands on lxplus8 normally it should read the input file from |
|
@smuzaffar the output of the |
|
@hahnjo What happens if you open the |
|
I think as @hahnjo is running as non-cms user so he might not have rights to access this file. I am looking how I can make /eos/cms/store/user/cmsbuild readable by all CERN users |
|
@hahnjo , I have copied the required root file on public afs. Please try this |
|
Thanks @smuzaffar, that worked. At least from a first look, ROOT doesn't seem to be doing anything wrong here: The |
|
Ok, but there is a double delete, and with a bit of
namespace StopReasonName {
static const std::string StopReasonName[] = {
"UNINITIALIZED", // 0
"MAX_HITS", // 1
"MAX_LOST_HITS", // 2
"MAX_CONSECUTIVE_LOST_HITS", // 3
"LOST_HIT_FRACTION", // 4
"MIN_PT", // 5
"CHARGE_SIGNIFICANCE", // 6
"LOOPER", // 7
"MAX_CCC_LOST_HITS", // 8
"NO_SEGMENTS_FOR_VALID_LAYERS", // 9
"SEED_EXTENSION", // 10
"NOT_STOPPED" // 11 (be careful, NOT_STOPPED needs to be the last,
// its index differs from the enumeration value)
};
};That header + This now jumps from one eventually freeing |
This works around root-project/root#13429 by not having std::string arrays in headers that are double-freed.
|
Even easier way to reproduce: With that, confirmed that this affects |
|
Results of today's investigations: The reason is that We cannot disable Conclusion: What I think we should really do here is patch |
Otherwise they are emitted as interal and we get double-construction and -destruction on the same memory address due to the way we promote interal declarations in KeepLocalGVPass. Fixes root-project#13429
Otherwise they are emitted as interal and we get double-construction and -destruction on the same memory address due to the way we promote interal declarations in KeepLocalGVPass. Fixes root-project#13429
Otherwise they are emitted as internal and we get double-construction and -destruction on the same memory address due to the way we promote interal declarations in KeepLocalGVPass. According to upstream tests, the de-duplication doesn't work on Windows (yet), but I think this problem is severe enough to fix it on the other platforms sooner rather than later. Fixes root-project#13429
Otherwise they are emitted as internal and we get double-construction and -destruction on the same memory address due to the way we promote internal declarations in KeepLocalGVPass. According to upstream tests, the de-duplication doesn't work on Windows (yet), but I think this problem is severe enough to fix it on the other platforms sooner rather than later. Fixes #13429
Otherwise they are emitted as internal and we get double-construction and -destruction on the same memory address due to the way we promote internal declarations in KeepLocalGVPass. According to upstream tests, the de-duplication doesn't work on Windows (yet), but I think this problem is severe enough to fix it on the other platforms sooner rather than later. Fixes root-project#13429 (cherry picked from commit 2dc5786)
Otherwise they are emitted as internal and we get double-construction and -destruction on the same memory address due to the way we promote internal declarations in KeepLocalGVPass. According to upstream tests, the de-duplication doesn't work on Windows (yet), but I think this problem is severe enough to fix it on the other platforms sooner rather than later. Fixes #13429 (cherry picked from commit 2dc5786)
Otherwise they are emitted as internal and we get double-construction and -destruction on the same memory address due to the way we promote internal declarations in KeepLocalGVPass. According to upstream tests, the de-duplication doesn't work on Windows (yet), but I think this problem is severe enough to fix it on the other platforms sooner rather than later. Fixes root-project#13429
Check duplicate issues.
Description
Hello,
Since we moved to TCMalloc (instead of JeMalloc) for
cmsRunwe have seen some failures inTROOT::EndOfProcessCleanupsupon destruction ofTString(and some other destructors) affecting CMSSW ROOT6 (version6.29.01) and ROOT628 (version6.28.05) IBs (el8_amd64_gcc11).We reported the issue in the cmssw repo at #42468.
Reproducer
The errors can be reproduced on lxplus as follows:
And for gdb:
A sample stacktrace upon destruction of the
TString:See cms-sw/cmssw#42468 (comment) for other destructors failing on cleanup.
It runs fine when using
cmsRunJEinstead (for JeMalloc allocation type):cmsRunJE /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/src/AlignmentunitTest=True trackCollection=ALCARECOTkAlCosmicsCTF0T.ROOT version
CMSSW ROOT6 (version
6.29.01) and ROOT628 (version6.28.05) IBs.Installation method
from source
Operating system
el8
Additional context
FYI, @smuzaffar @makortel
Reported to ROOT as discussed in Core Software meeting
The text was updated successfully, but these errors were encountered: