-
Notifications
You must be signed in to change notification settings - Fork 125
Open
Description
Bug Description
There is a critical off-heap memory leak in ByteBufferProxy.java. The AbstractByteBufferProxy class utilizes a ThreadLocal to cache ByteBuffer instances in an ArrayDeque. However, this queue is entirely unbounded and caches Direct ByteBuffers, leading to massive native memory accumulation and physical memory fragmentation.
Root Cause
In ByteBufferProxy.java, the cache is defined as:
private static final ThreadLocal<ArrayDeque<ByteBuffer>> BUFFERS =
withInitial(() -> new ArrayDeque<>(16));
When allocate() is called and the queue is empty, it allocates a direct buffer via ByteBuffer.allocateDirect(0).
When deallocate(ByteBuffer buff) is invoked, it blindly returns the buffer to the queue:
queue.offer(buff);
There is no maximum capacity limit on the ArrayDeque, and BUFFERS.remove() is never invoked.
Impact
In environments utilizing thread pools (e.g., web servers, async task workers), threads are long-lived. Because the ArrayDeque is unbounded and never explicitly cleared, each thread retains all the Direct ByteBuffers it has ever processed at peak load.
This causes:
1. Unbounded Native Memory Growth: Direct buffers bypass the JVM heap. Their accumulation quickly exhausts the OS physical memory.
2. Memory Fragmentation & OOM Killer: The uncontrolled accumulation of direct byte blocks leads to severe physical memory fragmentation, eventually triggering the OS-level OOM Killer and crashing the JVM.
Proposed Fix
1. Bound the Queue: Enforce a maximum capacity (e.g., MAX_CACHED_BUFFERS) in the deallocate method. If the ArrayDeque reaches this limit, allow the buffer to be naturally garbage-collected instead of re-queueing it.
2. Lifecycle Management: Provide a mechanism to call BUFFERS.remove() to clean up the ThreadLocalMap when a thread's lifecycle ends, or replace the ThreadLocal approach with a centralized, globally bounded object pool (e.g., ConcurrentLinkedQueue with a strict size limit).Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels