-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vmstorage GC overhead grows on long running instance #5379
Comments
@waldoweng , thanks for filing the detailed bugreport! It is unexpected that the high number of items in the
I see the following solutions if this is the real issue (probably, it is triggered by periodic deletion of items from the map):
|
@valyala, thanks for the quick response! |
…r to reduce map fragmentation Hopefully, this may help reducing memory usage and GC pressure for long-running processes, which use fastcache. See VictoriaMetrics/VictoriaMetrics#5379
…1.12.2 This should help reducing GC overhead growth at #5379
…1.12.2 This should help reducing GC overhead growth at #5379
@waldoweng , I implemented the first solution mentioned above in VictoriaMetrics/fastcache@959e53c . This change has been added into VictoriaMetrics repository in the commit a7800cd . Could you try building VictoriaMetrics from this commit according to these instructions and verify whether it helps reducing GC overhead growth for long-running VictoriaMetrics instances under your workload? If you use cluster version of VictoriaMetrics, then try building and running |
@valyala, just want to let you know that by talking to people at golang nut, it seems to confirm this issue. But your implementation is coming way sooooooooooo fast! I should try to verify this fix, will report back when I got some results, thanks! |
…1.12.2 This should help reducing GC overhead growth at VictoriaMetrics#5379
Describe the bug
vmstorage rely on
lib/workingsetcache
module to cache a lots of useful data:TSIDcache
,MetricNamecache
, etc.lib/workingsetcache
operate mainly on two modes:split
andwhole
mode.lib/workingsetcache
usefastcache
internally, when inwhole
mode, internalfastcache
never get reset, but do cleanup by callingfastcache.bucket.cleanLocked()
function which callsdelete()
regularly.But this won't be enough since map of golang never shrink:
memory occupied and pointer reference by
fastcache.bucket.m
will never go away, result in a very smooth memory leak and increase of GC overhead.To Reproduce
TSIDcache
switch fromsplit
mode towhole
modeVersion
all cluster version
Logs
No response
Screenshots
Used command-line flags
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: