preface

If you do not return pooled resources, they may be leaked, such as database connections. For Netty memory pools, memory leaks can occur if a memory block is used and not returned. Netty provides the leak detection service to help users troubleshoot memory leaks. Learn from several angles in this chapter:

How to use Netty leak detection
How can I properly configure the Netty leak detection level
How to implement Netty leak detection

I. Use of leak detection

public class MyLeakTest {
    public static final int _1MB = 1024 * 1024;
    public static final int _17MB = 17 * 1024 * 1024;
    @Before
    public void init(a) {
        ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
        scheduledExecutorService.scheduleAtFixedRate(() -> {
            long memory = PlatformDependent.usedDirectMemory()/ _1MB;
            System.out.println(Use direct memory: + memory + "MB");
        }, 0.1, TimeUnit.SECONDS);
    }

    @Test
    public void testLeak01(a) throws InterruptedException {
        ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);
        for (int i = 0; i < 3; i++) {
            allocate17();
            System.gc();
            Thread.sleep(1000); }}private void allocate17(a) {
        System.out.println("Start allocating");
        PooledByteBufAllocator allocator = (PooledByteBufAllocator) ByteBufAllocator.DEFAULT;
        allocator.newDirectBuffer(_17MB, Integer.MAX_VALUE)/*.release()*/;
        System.out.println("Successful distribution"); }}Copy the code

The init method starts a thread responsible for printing the immediate memory currently in use. TestLeak01 Method simulates a memory leak.

TestLeak01 method first by ResourceLeakDetector. SetLevel method sets the level at the Netty leak detection PARANOID, This step can also use * * ty – Dio.net. LeakDetection. * * = the paranoid level Settings. Then allocate 17MB in a loop without calling ByteBuf’s release method and calling System.gc notification for garbage collection.

Console output:

Start using direct memory distribution success: 17 MB distribution 14:12:51. 719 ERROR [main] io.net ty. Util. ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:372) io.netty.buffer.MyLeakTest.allocate17(MyLeakTest.java:65) io.netty.buffer.MyLeakTest.testLeak01(MyLeakTest.java:40) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 62) successful distribution using direct memory: 34 MB distribution success using direct memory: 51 MBCopy the code

If we use the SIMPLE or ADVANCED levels, it will take several memory allocations before this exception log is printed.

@Test
public void testLeak03(a) throws InterruptedException {
    ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.ADVANCED);
    for (int i = 0; i < 300; i++) {
        allocate1KB(); / / assign 1 KB
        System.gc();
        Thread.sleep(100); }}private void allocate1KB(int count) {
    System.out.println("Start distribution, number one." + count + "Time");
    PooledByteBufAllocator allocator = (PooledByteBufAllocator) ByteBufAllocator.DEFAULT;
    allocator.newDirectBuffer(1024, Integer.MAX_VALUE)/*.release()*/;
    System.out.println("Successful allocation, no." + count + "Time");
}
Copy the code

Console output:

Direct memory 243rd use :16MB start allocate, 244th use successful allocate, 244th use... Start distribution, the 251th successful distribution, distribution of 251th start, 252th 14:21:33. 063 ERROR [main] io.net ty. Util. ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:372) io.netty.buffer.MyLeakTest.allocate1KB(MyLeakTest.java:75) io.netty.buffer.MyLeakTest.testLeak03(MyLeakTest.java:66) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 62) distribution of success, for the 252th time using direct memory: 16 MB distribution, for the 253th timeCopy the code

Why is the direct memory here always 16MB? Recall the process of Netty memory allocation. Netty applies for 16MB memory at a time from the system. The subsequent memory allocation is based on Chunk allocation or Subpage allocation, and does not apply for memory from the system.

Two, leak detection level

Leak detection Level corresponding enumeration types ResourceLeakDetector. Level.

public enum Level {
    /** * Disables resource leak detection. */
    DISABLED,
    /** * Enables simplistic sampling resource leak detection which reports there is a leak or not, * at the cost of small overhead (default). */
    SIMPLE,
    /** * Enables advanced sampling resource leak detection which reports where the leaked object was accessed * recently at  the cost of high overhead. */
    ADVANCED,
    /** * Enables paranoid resource leak detection which reports where the leaked object was accessed recently, * at the cost of the highest possible overhead (for testing purposes only). */
    PARANOID;
}
Copy the code

Different leak detection levels correspond to different leak detection behaviors.

level	Open detection probability	Corresponds to the ByteBuf implementation class
DISABLED	0%	–
SIMPLE	The default 1/128	SimpleLeakAwareByteBuf
ADVANCED	The default 1/128	AdvancedLeakAwareByteBuf
PARANOID	100%	AdvancedLeakAwareByteBuf

By default, the leak detection level is SIMPLE, the sample rate for SIMPLE and ADVANCED is 1/128, and the maximum Record for a single leak place is 4.

private static final Level DEFAULT_LEVEL = Level.SIMPLE;
private static final int DEFAULT_TARGET_RECORDS = 4;
private static final int DEFAULT_SAMPLING_INTERVAL = 128;
private static Level level;
static {
    final boolean disabled;
    if (SystemPropertyUtil.get("io.netty.noResourceLeakDetection") != null) {
        disabled = SystemPropertyUtil.getBoolean("io.netty.noResourceLeakDetection".false);
    } else {
        disabled = false;
    }
    / / io.net. Ty noResourceLeakDetection default to false, the default level is SIMPLE
    Level defaultLevel = disabled? Level.DISABLED : DEFAULT_LEVEL;

    // First read old property name
    String levelStr = SystemPropertyUtil.get(PROP_LEVEL_OLD, defaultLevel.name());

    // If new property name is present, use it
    levelStr = SystemPropertyUtil.get(PROP_LEVEL, levelStr);
    Level level = Level.parseLevel(levelStr);
    / / for a place to leak, RECORD a few RECORD, most of the default io.net. Ty leakDetection. TargetRecords = 4
    TARGET_RECORDS = SystemPropertyUtil.getInt(PROP_TARGET_RECORDS, DEFAULT_TARGET_RECORDS);
    / / leak detection sampling rate ty. The default io.net leakDetection. SamplingInterval = 128
    SAMPLING_INTERVAL = SystemPropertyUtil.getInt(PROP_SAMPLING_INTERVAL, DEFAULT_SAMPLING_INTERVAL);
    ResourceLeakDetector.level = level;
}
Copy the code

The leak detection implementation classes are all subclasses of WrappedByteBuf, WrappedByteBuf wraps a ByteBuf, and all ByteBuf interfaces delegate to the ByteBuf implementation. For SIMPLE levels, select SimpleLeakAwareByteBuf package leak detection. For levels higher than SIMPLE, select AdvancedLeakAwareByteBuf package leak detection.

For details, see PooledByteBufAllocator#newDirectBuffer, PooledByteBufAllocator#newHeapBuffer, and UnpooledByteBufAllocator#newDirectBuffe Entrance r. PooledByteBufAllocator#newDirectBuffer is used as an example.

@Override
protected ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity) {
    // 1. Get the PoolArena for the current thread cache and thread cache
    PoolThreadCache cache = threadCache.get();
    PoolArena<ByteBuffer> directArena = cache.directArena;
    final ByteBuf buf;
    if(directArena ! =null) {
        // 2. Select pool
        buf = directArena.allocate(cache, initialCapacity, maxCapacity);
    } else {
        // Select unpooled
        buf = PlatformDependent.hasUnsafe() ?
            UnsafeByteBufUtil.newUnsafeDirectByteBuf(this, initialCapacity, maxCapacity) :
        new UnpooledDirectByteBuf(this, initialCapacity, maxCapacity);
    }
    // 3. If memory leak detection is configured, wrap ByteBuf
    return toLeakAwareBuffer(buf);
}
Copy the code

In the last step, newDirectBuffer wraps PooledByteBuf as LeakAware ByteBuf. Based on the Level field of the ResourceLeakDetector class, determine which LeakAwareByteBuf is instantiated and a ResourceLeakTracker instance is passed in during construction.

protected static ByteBuf toLeakAwareBuffer(ByteBuf buf) {
    ResourceLeakTracker<ByteBuf> leak;
    switch (ResourceLeakDetector.getLevel()) {
        case SIMPLE:
            leak = AbstractByteBuf.leakDetector.track(buf);
            if(leak ! =null) {
                buf = new SimpleLeakAwareByteBuf(buf, leak);
            }
            break;
        case ADVANCED:
        case PARANOID:
            leak = AbstractByteBuf.leakDetector.track(buf);
            if(leak ! =null) {
                buf = new AdvancedLeakAwareByteBuf(buf, leak);
            }
            break;
        default:
            break;
    }
    return buf;
}
Copy the code

Note that the ResourceLeakTracker instance is created by the AbstractByteBuf class variable leakDetector’s Track method.

public abstract class AbstractByteBuf extends ByteBuf {
    static final ResourceLeakDetector<ByteBuf> leakDetector = ResourceLeakDetectorFactory.instance().newResourceLeakDetector(ByteBuf.class);
}
Copy the code

SimpleLeakAwareByteBuf Simple leak detection that does not record additional information about memory leaks. The feature is that leak detection is turned off using the Close method of ResourceLeakTracker only when the Release method is called.

class SimpleLeakAwareByteBuf extends WrappedByteBuf {
	// ByteBuf participating in memory leak detection
    private final ByteBuf trackedByteBuf;
    final ResourceLeakTracker<ByteBuf> leak;

    SimpleLeakAwareByteBuf(ByteBuf wrapped, ByteBuf trackedByteBuf, ResourceLeakTracker<ByteBuf> leak) {
        super(wrapped);
        this.trackedByteBuf = ObjectUtil.checkNotNull(trackedByteBuf, "trackedByteBuf");
        this.leak = ObjectUtil.checkNotNull(leak, "leak");
    }
    @Override
    public boolean release(a) {
        if (super.release()) {
            closeLeak();
            return true;
        }
        return false;
    }

    @Override
    public boolean release(int decrement) {
        if (super.release(decrement)) {
            closeLeak();
            return true;
        }
        return false;
    }
    // When release is called correctly, leak detection is turned off
    private void closeLeak(a) {
        boolean closed = leak.close(trackedByteBuf);
        assertclosed; }}Copy the code

AdvancedLeakAwareByteBuf inherits SimpleLeakAwareByteBuf, and records Record information when ByteBuf’s API is executed, facilitating subsequent troubleshooting. ResourceLeakTracker’s record method comes later.

final class AdvancedLeakAwareByteBuf extends SimpleLeakAwareByteBuf {

    private static final String PROP_ACQUIRE_AND_RELEASE_ONLY = "io.netty.leakDetection.acquireAndReleaseOnly";
    // Whether to record only when the record is acquired and released. Default is false
    private static final boolean ACQUIRE_AND_RELEASE_ONLY;

    static {
        ACQUIRE_AND_RELEASE_ONLY = SystemPropertyUtil.getBoolean(PROP_ACQUIRE_AND_RELEASE_ONLY, false);
    }

    AdvancedLeakAwareByteBuf(ByteBuf buf, ResourceLeakTracker<ByteBuf> leak) {
        super(buf, leak);
    }

    AdvancedLeakAwareByteBuf(ByteBuf wrapped, ByteBuf trackedByteBuf, ResourceLeakTracker<ByteBuf> leak) {
        super(wrapped, trackedByteBuf, leak);
    }
	// ACQUIRE_AND_RELEASE_ONLY Is false when the Record is leaked
    static void recordLeakNonRefCountingOperation(ResourceLeakTracker<ByteBuf> leak) {
        if (!ACQUIRE_AND_RELEASE_ONLY) {
            leak.record();
        }
    }
    // Read & write if ACQUIRE_AND_RELEASE_ONLY is false, then leak.record is executed
    @Override
    public byte getByte(int index) {
        recordLeakNonRefCountingOperation(leak);
        return super.getByte(index);
    }
    
    @Override
    public ByteBuf writeByte(int value) {
        recordLeakNonRefCountingOperation(leak);
        return super.writeByte(value);
    }

    // retain (increase reference count) /release (decrease reference count) /touch(specifically for a record)
    // Execute leak.record directly
    @Override
    public ByteBuf retain(a) {
        leak.record();
        return super.retain();
    }

    @Override
    public boolean release(a) {
        leak.record();
        return super.release();
    }

    @Override
    public ByteBuf touch(a) {
        leak.record();
        return this; }}Copy the code

Third, Record

Record represents a Record, linked list data structure, and is an internal class of Resource Detector.

private static final class Record extends Throwable {
    // The last node of the Record list
    private static final Record BOTTOM = new Record();
    // External incoming hint. ToString to record additional information about memory leaks
    private final String hintString;
    // Point to the next Record
    private final Record next;
    // The current Record is in the list position 0-n, only the BOTTOM node is -1
    private final int pos;

    Record(Record next, Object hint) {
        hintString = hint.toString();
        this.next = next;
        this.pos = next.pos + 1;
    }

    Record(Record next) {
        hintString = null;
        this.next = next;
        this.pos = next.pos + 1;
    }

    private Record(a) {
        hintString = null;
        next = null;
        pos = -1; }}Copy the code

Record inherits Throwable, omitting some of the code, and you can see that the Throwable is inherited to get the call stack (getStackTrace).

@Override
public String toString(a) {
    StringBuilder buf = new StringBuilder(2048);
    if(hintString ! =null) {
        buf.append("\tHint: ").append(hintString).append(NEWLINE);
    }
    // From Throwable, you can print StackTrace
    StackTraceElement[] array = getStackTrace();
    Skip the first three stacks
    for (int i = 3; i < array.length; i++) {
        StackTraceElement element = array[i];
        buf.append('\t');
        buf.append(element.toString());
        buf.append(NEWLINE);
    }
    return buf.toString();
}
Copy the code

Take a unit test to see what Record does.

@Test
public void testRecord(a) throws InterruptedException {
    ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);
    PooledByteBufAllocator allocator = (PooledByteBufAllocator) ByteBufAllocator.DEFAULT;
    // Created at
    ByteBuf byteBuf = allocator.newDirectBuffer(1024, Integer.MAX_VALUE); 
    / / # 2
    byteBuf.touch();
    / / # 1
    ByteBuf slice = byteBuf.slice();
    // Release strong references, notify GC
    slice = null;
    byteBuf = null;
    System.gc();
    Thread.sleep(1000);
    // To trigger leak detection log printing
    byteBuf = allocator.newDirectBuffer(1024, Integer.MAX_VALUE);
}
Copy the code

Console output:

Direct memory :0MB direct memory :16MB14:18:34.888 [main] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: io.netty.buffer.AdvancedLeakAwareByteBuf.slice(AdvancedLeakAwareByteBuf.java:76) io.netty.buffer.MyLeakTest.testRecord(MyLeakTest.java:80) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) #2: io.netty.buffer.MyLeakTest.testRecord(MyLeakTest.java:79) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:372) io.netty.buffer.MyLeakTest.testRecord(MyLeakTest.java:78) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)Copy the code

You can see that the output records contains three sections

#1: Stack information for the slice method call
#2: Stack information for the touch method call
Created at: Bytebuf Created stack information

Four, DefaultResourceLeak

The ResourceLeakTracker interface is exposed for use by clients, providing the ability to Record records on the one hand and to turn off leak detection for an object on the other.

public interface ResourceLeakTracker<T>  {
    // Record a Record
    void record(a);
    // Record a Record with some hint information to help troubleshoot
    void record(Object hint);
    // Turn off memory leak detection for trackedObject, which is usually a ByteBuf instance
    boolean close(T trackedObject);
}
Copy the code

ResourceLeakTracker the default implementation is DefaultResourceLeak.

private static final class DefaultResourceLeak<T>
            extends WeakReference<Object> implements ResourceLeakTracker<T>, ResourceLeak {
		// Atomic updater for head pointer
        private static finalAtomicReferenceFieldUpdater<DefaultResourceLeak<? >, Record> headUpdater = (AtomicReferenceFieldUpdater) AtomicReferenceFieldUpdater.newUpdater(DefaultResourceLeak.class, Record.class,"head");
        // Discard atomic updater for droppedRecords number of records
        private static finalAtomicIntegerFieldUpdater<DefaultResourceLeak<? >> droppedRecordsUpdater = (AtomicIntegerFieldUpdater) AtomicIntegerFieldUpdater.newUpdater(DefaultResourceLeak.class,"droppedRecords");

		// Record list for T leak detection
        private volatile Record head;
        // Discard Record number
        private volatile int droppedRecords;
		// Contains all instances of DefaultResourceLeak, passed in externally
        private finalSet<DefaultResourceLeak<? >> allLeaks;// Hash value of T instance
        private final inttrackedHash; DefaultResourceLeak(Object referent, ReferenceQueue<Object> refQueue, Set<DefaultResourceLeak<? >> allLeaks) {// Call WeakReference constructor, pass in reference object and reference queue
            super(referent, refQueue);
            // Evaluates the hash value of the reference object
            trackedHash = System.identityHashCode(referent);
            // Add the current instance to allLeaks
            allLeaks.add(this);
            // Set the head node of the Record list to record.bottom
            headUpdater.set(this.new Record(Record.BOTTOM));
            this.allLeaks = allLeaks; }}Copy the code

The ResourceLeak interface is obsolete and can be ignored. The key point is that DefaultResourceLeak inherits WeakReference, passing in external reference object and reference queue through WeakReference structure. This reference object is the ByteBuf instance monitored for Netty memory leak detection.

1, record0

The record0 method is responsible for recording a Record. Abstract methods for Record () and Record (T) that implement the ResourceLeakTracker interface.

private void record0(Object hint) {
    if (TARGET_RECORDS > 0) {
        Record oldHead;
        Record prevHead;
        Record newHead;
        boolean dropped;
        do {
            // If the head node is empty, the close method has been called, the resource has been freed, and no need to continue
            if ((prevHead = oldHead = headUpdater.get(this)) = =null) {
                return;
            }
            // Number of current records
            final int numElements = oldHead.pos + 1;
            / / if the current Record number greater than io.net. Ty leakDetection. TargetRecords (4) by default, may list a Record of the head
            if (numElements >= TARGET_RECORDS) {
                // Discard probability = 1/2 ^ (numElements - targetRecords)
                final int backOffFactor = Math.min(numElements - TARGET_RECORDS, 30);
                if (dropped = PlatformDependent.threadLocalRandom().nextInt(1<< backOffFactor) ! =0) { prevHead = oldHead.next; }}else {
                dropped = false;
            }
            // cas sets the list head node to the newly created RecordnewHead = hint ! =null ? new Record(prevHead, hint) : new Record(prevHead);
        } while(! headUpdater.compareAndSet(this, oldHead, newHead));
        if (dropped) {
            droppedRecordsUpdater.incrementAndGet(this); }}}Copy the code

The record0 method basically constructs a new Record and inserts the CAS header into the Record list. It is worth noting that after four records are inserted, each new Record may cause the linked list head node to be discarded.

2, close

The close method is responsible for turning off leak detection for the current reference object. Implement the close(T) abstract method of the Resource Tracker interface.

 private static finalAtomicReferenceFieldUpdater<DefaultResourceLeak<? >, Record> headUpdater = (AtomicReferenceFieldUpdater) AtomicReferenceFieldUpdater.newUpdater(DefaultResourceLeak.class, Record.class,"head");
private finalSet<DefaultResourceLeak<? >> allLeaks;private final int trackedHash;

@Override
public boolean close(a) {
    // Remove the current DefaultResourceLeak instance from Set
    if (allLeaks.remove(this)) {
        // Call weakReference. clear to remove the referent
        clear();
        // Set the head of the Record list to null
        headUpdater.set(this.null);
        return true;
    }
    return false;
}

@Override
public boolean close(T trackedObject) {
    // Ensure that the incoming instance and the monitored instance are the same instance
    assert trackedHash == System.identityHashCode(trackedObject);
    try {
        return close();
    } finally{ reachabilityFence0(trackedObject); }}Copy the code

The close method does three things:

Remove the current instance from allLeaks
Call the clear method of parent class WeakReference to set the reference object as empty
Set the head node of Record to null

Take a look at the reachabilityFence0 method, which prevents the current DefaultResourceLeak instance from being recycled early. May lead to determine in advance before because JDK9 objects inaccessible, lead to early recovery, here by the finally + synchronized that objects will not be early recovery, JDK9 provides the Reference. ReachabilityFence method to solve this problem.

/**
* Recent versions of the JDK have a nasty habit of prematurely deciding objects are unreachable.
* see: https://stackoverflow.com/questions/26642153/finalize-called-on-strongly-reachable-object-in-java-8
* The Java 9 method Reference.reachabilityFence offers a solution to this problem.
*/
private static void reachabilityFence0(Object ref) {
    if(ref ! =null) {
        synchronized (ref) {
            // Empty synchronized is ok: https://stackoverflow.com/a/31933260/1151521}}}Copy the code

3, the dispose

The dipose method is a non-public method, which is used by Netty to check whether the close method of DefaultResourceLeak is called by the client. If the close method is called, it means that the monitored resources are collected normally (ByteBuf’s release method is called normally). The current DefaultResourceLeak instance should not exist in the allLeaks collection.

/** * @return true Close not executed, memory leak * false Close executed, no memory leak */ Boolean dispose() {clear(); return allLeaks.remove(this); }Copy the code

The reason why we use a Set to determine if DefaultResourceLeak instances are called close normally, rather than a Boolean, is to ensure that DefaultResourceLeak is strongly referenced and not GC.

Fifth, ResourceLeakDetector

ResourceLeakDetector manages leak detection for a class of instances (generic T).

public class ResourceLeakDetector<T> {
    // A collection of all leak detection tasks currently managed by Detector
    private finalSet<DefaultResourceLeak<? >> allLeaks = Collections.newSetFromMap(newConcurrentHashMap<DefaultResourceLeak<? >, Boolean>());// Reference queue, where DefaultResourceLeak instances can be received when the detection object of the DefaultResourceLeak package is reclaimed
    private final ReferenceQueue<Object> refQueue = new ReferenceQueue<Object>();
    // A collection of report strings that have done memory leak report output
    private final Set<String> reportedLeaks =
            Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>());

    / / io.net ty. LeakDetection. SamplingInterval 128 by default
    private final int samplingInterval;
}
Copy the code

ResourceLeakDetector mainly does the following three things:

Establish the relationship between the detection object and ResourceLeakTracker to enable leak detection
Detect leaks
Let the cat out of the report

All three things are done in the track method, and for Netty memory leak detection, all three things are done when ByteBuf is created.

This is why the last line of the unit test on Record in section 3 allocates memory once, because only when ByteBuf is allocated, the toLeakAwareBuffer method will be called, and the toLeakAwareBuffer method will call track method, which can do a leak detection and print a leak report.

private DefaultResourceLeak track0(T obj) {
    Level level = ResourceLeakDetector.level;
    if (level == Level.DISABLED) {
        return null;
    }
    // If level is less than PARANOID, DefaultResourceLeak is not necessarily returned
    / / depends on random PlatformDependent. ThreadLocalRandom () nextInt (samplingInterval)
    if (level.ordinal() < Level.PARANOID.ordinal()) {
        // a 1 in 128 chance of doing a leak check and returning DefaultResourceLeak
        if ((PlatformDependent.threadLocalRandom().nextInt(samplingInterval)) == 0) {
            reportLeak();
            return new DefaultResourceLeak(obj, refQueue, allLeaks);
        }
        return null;
    }
    // Level equals PARANOID Do a leak check immediately and return DefaultResourceLeak
    reportLeak();
    return new DefaultResourceLeak(obj, refQueue, allLeaks);
}
Copy the code

Track method has slightly different behaviors according to different detection levels. The difference is that if the PARANOID level is lower than the PARANOID level, a leak detection + leak report + OBJ leak detection may not be performed simultaneously.

In the track method, reportLeak = leak detection + leak report; New DefaultResourceLeak = Enable leak detection for OBJ. The constructor for DefaultResourceLeak was looked at in Section 4, where we focus on the reportLeak method.

private void reportLeak(a) {
    // Whether to enable leak detection (currently based on whether the log level allows error control)
    if(! needReport()) {/ / empty refQueue
        clearRefQueue();
        return;
    }

    for (;;) {
        // Get the DefaultResourceLeak from ReferenceQueue
        // DefaultResourceLeak the obJ (ByteBuf instance) wrapped as a virtual reference has been reclaimed at this point
        DefaultResourceLeak ref = (DefaultResourceLeak) refQueue.poll();
        if (ref == null) {
            break;
        }
        // Determine if a leak has occurred
        if(! ref.dispose()) {continue;
        }
        // Prints leak logs
        String records = ref.toString();
        // If records has already been output, it will not be output again
        if (reportedLeaks.add(records)) {
            if (records.isEmpty()) {
                reportUntracedLeak(resourceType);
            } else{ reportTracedLeak(resourceType, records); }}}}Copy the code

First, the reportLeak method is controlled by the log level. If the log level is less than Error, leak detection will not be done.

protected boolean needReport(a) {
    return logger.isErrorEnabled();
}
Copy the code

Next, the DefaultResourceLeak virtual reference instance is pulled through the reference queue.

If you obtain DefaultResourceLeak, the referenced instances monitored by DefaultResourceLeak instances have been garbage collected. Use the Dispose method to determine whether close is invoked to close the instance.

If a leak occurs, check whether the output of the toString method of DefaultResourceLeak is in the reportedLeaks set. If there is, it indicates that the leak has been output and the leak report will not be output again. Otherwise, the leak report will be output.

Six, doubt

Why use an allLeaks Set to use the remove method instead of a Boolean to determine if the resource is properly released?

private finalSet<DefaultResourceLeak<? >> allLeaks;private final inttrackedHash; DefaultResourceLeak( Object referent, ReferenceQueue<Object> refQueue, Set<DefaultResourceLeak<? >> allLeaks) {super(referent, refQueue);
    trackedHash = System.identityHashCode(referent);
    // Note that this allLeaks is external
    // Is the allLeaks collection of ResourceLeakDetector
    allLeaks.add(this);
    this.allLeaks = allLeaks;
}
/** * Netty calls this method to determine whether the close method was properly called on the object instance. *@returnTrue If close is not executed, there is a memory leak. * False If close is executed, there is no memory leak. */
boolean dispose(a) {
    clear();
    return allLeaks.remove(this);
}

/** * is exposed to the client call, which is called when the monitor object is released normally */
@Override
public boolean close(a) {
    // Remove the current DefaultResourceLeak instance from Set
    if (allLeaks.remove(this)) {
        // Call clear so the reference is not even enqueued.
        // Call weakReference. clear to remove the referent
        clear();
        // Set the head of the Record list to null
        headUpdater.set(this.null);
        return true;
    }
    return false;
}
Copy the code

If DefaultResourceLeak instances are not strongly referenced by Set, DefaultResourceLeak is directly GC if ByteBuf does not have strong references.

For example, if SimpleLeakAwareByteBuf holds DefaultResourceLeak instances, if SimpleLeakAwareByteBuf is not strongly referenced, DefaultResourceLeak instances are also collected and memory leaks cannot be detected.

class SimpleLeakAwareByteBuf extends WrappedByteBuf {
	// The detected ByteBuf
    private final ByteBuf trackedByteBuf;
    / / DefaultResourceLeak instance
    final ResourceLeakTracker<ByteBuf> leak;

    SimpleLeakAwareByteBuf(ByteBuf wrapped, ByteBuf trackedByteBuf, ResourceLeakTracker<ByteBuf> leak) {
        super(wrapped);
        this.trackedByteBuf = ObjectUtil.checkNotNull(trackedByteBuf, "trackedByteBuf");
        this.leak = ObjectUtil.checkNotNull(leak, "leak"); }}Copy the code

conclusion

How to use Netty leak detection?
- Code: ResourceLeakDetector. SetLevel (ResourceLeakDetector. Level. The PARANOID);
- Configuration: – Dio.net ty. LeakDetection. Level = the paranoid

How can I properly configure the Netty leak detection level?

Different leak detection levels correspond to different leak detection behaviors.

level	Open detection probability	Corresponds to the ByteBuf implementation class
DISABLED	0%	–
SIMPLE	The default 1/128	SimpleLeakAwareByteBuf
ADVANCED	The default 1/128	AdvancedLeakAwareByteBuf
PARANOID	100%	AdvancedLeakAwareByteBuf

By default, the leak detection level is SIMPLE, the sample rate for SIMPLE and ADVANCED is 1/128, and the maximum Record for a single leak place is 4.

How to implement Netty leak detection?
- When ByteBuf is created, leak detection is enabled. When the ByteBuf reference count is 0, leak detection is turned off when the release method is executed.
- Only when ByteBuf is created, leak detection is triggered and a leak report is printed.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Netty source code (seven) memory leak detection

preface

I. Use of leak detection

Two, leak detection level

Third, Record

Four, DefaultResourceLeak

1, record0

2, close

3, the dispose

Fifth, ResourceLeakDetector

Six, doubt

conclusion

Netty source code (seven) memory leak detection

preface

I. Use of leak detection

Two, leak detection level

Third, Record

Four, DefaultResourceLeak

1, record0

2, close

3, the dispose

Fifth, ResourceLeakDetector

Six, doubt

conclusion

Related Posts

Multi-threaded join(

Small six usually development tips two (Nacos in the service configuration center clever use)

Deploy k3S clusters across cloud vendors