Advanced Techniques for Identifying Memory Issues with Analyzer Tool
Memory issues in production can be extremely disruptive, resulting in degraded performance, system crashes and excessive cloud provider costs.
They may be caused by several things. These include memory leaks, sub-optimal configuration of the JVM, and inefficient coding practices. To fix the problem, and prevent it recurring, we need to use memory analyzer tools to find and rectify the root cause.
In this article, we’ll look at tools and techniques we can use for fast and effective memory troubleshooting.
Memory Analyzer Tools
First let’s make sure our toolbox is effectively stocked for the task.
The tools we should have on hand are:
A heap dump analyzer, for example HeapHero or Eclipse MAT. We can use this to explore and analyze a heap dump in detail.
A garbage collection (GC) Log Analyzer such as GCeasy or GCViewer. Ensuring the GC is running efficiently is an important part of memory management, and GC logs contain valuable clues that help us diagnose root causes.
A lightweight Java profiler such as VisualVM. This allows us to look inside a running JVM, and get a quick overview of what is and isn’t working well.
Native memory tracking (NMT). This technique shows us how memory outside the heap is being used. Hotspot Java JDKs include tools for extracting NMT information. To save time, we can quickly visualize this information with the GC log analyzer GCeasy.
Understanding JVM Memory
Next let’s make sure we understand how the JVM arranges and manages its memory.
We can visualize runtime memory like this:
Fig: JVM Runtime Memory
At the highest level, memory is split into heap memory and native memory. Heap memory is managed by the JVM, whereas native memory is managed by the operating system.
Heap memory is a central storage area, where all classes that make up an application store their objects. It’s usually further subdivided into generations. Objects are placed in a generational area depending on their age, which makes garbage collection more efficient.
Native memory includes several areas, for example:
The metaspace, where class information is stored;
The direct buffer area, which is used in fast I/O operations.
For a full explanation of JVM memory, you may like to watch JVM Explained in 10 Minutes.
Memory problems can occur in any of these areas, although most commonly they relate to the heap. If we get an OutOfMemoryError, our first troubleshooting step is to identify the type of error, and which area it relates to. This article describes the different errors in detail: Types of OutOfMemoryErrors.
Problem-Solving Techniques
Briefly, a good strategy is to use a Java profiler to take a quick look and get an idea of the general health of the application. This helps pinpoint the area we need to focus on. Next, we can take a look at GC activity using a GC log analyzer. The objective here is to identify patterns that help us decide whether the problem relates to heap or non-heap memory.
We’d then zoom in on the affected area, using either heap dump analysis or native memory tracking.
Once we’ve isolated the problem, we can work out what needs to be done: either change the configuration or fix the code.
1. Java Profiling
We’ll use VisualVM, which is a convenient, easy-to-use tool, although there are several more sophisticated profilers on the market.
This tool can very quickly answer questions such as:
Is CPU usage high?
Is heap usage close to the maximum?
Is non-heap usage excessive?
Are too many threads blocked?
Is GC effectively reducing heap usage when it runs?
For more information, see VisualVM Features
2. GC Log Analysis
The GC is very resource-hungry, and if it runs too often due to memory problems, it has a huge impact on performance. The logs provide all sorts of useful information, and it’s always worth enabling them in production. They add very little overhead. To find out how to enable and manage the logs, you may like to read this article: GC Best Practices.
Perhaps the most useful troubleshooting technique is to study GC patterns, which can answer questions such as:
Do we have a memory leak?
Is the heap size under-configured?
Is the problem in heap memory or native memory?
As an example, compare the two graphs produced by GCeasy as shown below. The graphs show heap usage by time, with GC events marked by small red triangles.
Fig: Healthy GC Pattern vs Memory Leak Pattern
In the healthy pattern, the GC is able to bring memory usage down to a consistent level. If we have a leak, the GC still brings usage down, but never back to the same level. The bottom line keeps increasing over time.
For more information about recognizing GC patterns, see Interesting Garbage Collection Patterns.
3. Heap Dump Analysis
By now, we should have established whether the memory problem relates to heap memory or native memory. We should also have established whether the problem is an actual memory leak, or just excess memory usage.
If the problem relates to the heap, the next step is to take a heap dump, and submit it to a heap dump analyzer, such as HeapHero or Eclipse MAT. We’ll use HeapHero for the purpose of this article.
We’d use the dump to answer questions such as:
What’s taking up the most space in memory? The cause of memory issues is almost always found among the three or four largest objects in memory. Alternatively, in a few cases, the cause may be a proliferation of objects of the same class. HeapHero provides an interactive Largest Object report, where we can explore the largest objects. It also provides a class histogram, so we can see if we have large numbers of objects of the same class.
What do these large objects consist of? To find this out, we can interactively explore the object’s children, and optionally see their contents.
Why are these objects not being garbage collected? There could be two answers to this.
The most common is that another object is holding a reference that is not being released when it should. To check this, use the Largest Object report to browse incoming references to find the parents.
More rarely, GC may be held up by the finalizer queue. This can happen if any object’s finalize() method is slow to execute, perhaps because it’s waiting for a resource. HeapHero includes an ‘Objects Waiting for Finalization’ report.
What GC roots are in memory? The GC works from these roots, which include static objects and object pointers on the stack, to find all objects that are still being referenced. It then marks all other objects for GC. HeapHero allows us to browse these roots, seeing how much memory is retained by their children, and working up the tree to view all dependent objects. This is useful, because one of the major causes of memory issues is static objects that occupy a large retained heap.
Is memory being wasted by poor coding practices? HeapHero includes a memory wastage report, which we can expand and explore to find offending objects such as duplicate strings or poorly-sized collections. We are often surprised by how much memory is actually being wasted.
Fig: HeapHero’s Interactive Largest Object Chart
To see a demonstration of using HeapHero to explore a heap dump, you may like to watch How to Explore a Heap Dump Fast.
4. Native Memory Tracking
If the first two steps have revealed that the problem is in native memory rather than in the heap, we must resort to Native Memory Tracking (NMT).
To do this in any Hotspot Java implementation, we must enable NMT when the JVM is invoked, using the command line arguments shown in the example below:
java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail MyProg
At any time while the program is running, we can then take a dump of NMT information to a file using the following CLI command:
jcmd <pid> VM.native_memory detail > MyProg_nmt.txt
<pid> represents the process ID of the running program, which we can find by running the jps command.
Since this is a text file, we can interpret it manually. However, this may be time-consuming, since the file may be very long. It may contain information like this:
2740:
Native Memory Tracking:
(Omitting categories weighting less than 1KB)
Total: reserved=3523889KB, committed=201461KB
- Java Heap (reserved=2066432KB, committed=131072KB)
(mmap: reserved=2066432KB, committed=131072KB)
- Class (reserved=1048668KB, committed=220KB)
(classes #652)
( instance classes #550, array classes #102)
(malloc=92KB #948)
(mmap: reserved=1048576KB, committed=128KB)
( Metadata: )
( reserved=8192KB, committed=576KB)
( used=515KB)
( waste=61KB =10.62%)
( Class space:)
( reserved=1048576KB, committed=128KB)
( used=19KB)
( waste=109KB =84.95%)
- Thread (reserved=19501KB, committed=757KB)
(thread #19)
(stack: reserved=19456KB, committed=712KB)
(malloc=25KB #120)
(arena=20KB #36
A quick way to visualize this information is to upload it to GCeasy. This produces a summary of native memory in total, and a breakdown of how much memory is occupied by each native memory area.
It’s often useful to extract and analyze the text files at intervals, and compare them to see if any of these areas are growing. In the two screenshots below, we can see that the ‘Other’ area is growing larger over time, indicating a possible memory leak.
Fig: Initially, the ‘Other’ Area is Quite Small
Fig: As Time Goes On, it’s Growing
To find out how NMT tracking classifies the memory areas, see the Oracle NMT documentation.
Conclusion
Memory-related issues in Java are notorious performance-killers, and may result in system crashes.
Fortunately, several excellent memory analyzer tools are available. In this article, we’ve learned how to use a simple profiler to get an overview of where problems may exist in an application. We’ve also seen how to use GCeasy to analyze GC behavior and establish patterns, and also to visualize native memory usage.
Most importantly, we’ve looked at how to use heap dump analyzers such as HeapHero to explore heap memory and identify problem areas.
With these tools, we can keep our production systems free from the pain of memory glitches.
###
