1, the phenomenon of

On February 11th, 20+ access errors occurred on the gateway ina short period of time. Check kibina as follows:

According to the trace, the specific stack exception is found to be load balancing and the same POD all display connection exception:

Go to Grafana to check the running status of this POD, and it can be found that the memory suddenly decreases, and the POD has reached the memory limit (8G). In the case that resources cannot be applied, the POD may restart:

The AD service logs verify that the restart occurred:

At this point, we can basically locate the service restart due to memory problems, so the gateway can not load balance to the service instance, leading to the gateway error. Next, we need to clarify the reason why the POD memory consumption is so high.

2. Locate memory problems

  • According to grafana’s diagram, we can see that heap memory is normal, and non-heap memory is abnormal (utilization is over 100%) :

Non-heap is what the JVM keeps for itself, with the method area, the memory needed for processing or optimization within the JVM (such as jIT-compiled code caches), the code for each class structure (such as runtime pool, field, and method data), and the method and constructor code.

  • Open the file with JProfiler (you can also use jVisualVM), find the Biggest Objects, and find a large number of instances of the AdDto class in the live Objects, which take up more than 1 gb:

    Three, code inspection

Go back to the code and examine how AdDto was generated.

Each time modelMapper calls addMapping, it creates a structure of the class (via bytecode and then loaded by the class loader). See the documentation on the official website. If the conversion type is determined, ModelMapper should be set as a singleton (modelmapper.org/user-manual…

Fourth, offline reproduction

Using this code to verify the conjecture on a normal interface, it is clear that non-heap memory is going up and the number of classes loaded is going up, basically confirming the problem:

Repair and verify

Change the code to the following form and re-pressure:

    private static ModelMapper modelMapper;

    static {
        modelMapper = new ModelMapper();
        TypeMap<Person, PersonDTO> typeMap = modelMapper.createTypeMap(Person.class, PersonDTO.class);
        typeMap.addMappings(mapper -> {
            mapper.skip(PersonDTO::setAge);
        });
    }

    @GetMapping("api/benchmark/qa")
    public PersonDTO qa(a) {
        Person p = new Person();
        p.setName("wangao");
        p.setAge(30);
        return modelMapper.map(p, PersonDTO.class);
    }
Copy the code

Find that non-heap memory is stable and the number of class loads is stable: