How to save 24G of memory per application instance

Apache Pinot is a column-oritented distributed data store written in Java. It is open-source code used in applications where fast analytics is needed on immutable data. For example, at LinkedIn it was initially used to power the feature that allows members to see who has viewed their profile in real-time. Thanks to JXRay analysis, it has been recently discovered that in some installations, Pinot instances running with big (over 60G) heaps wasted a huge amount …

Continue Reading

JXRay 2.8u3 released

In this version: Detection of several new memory leak kinds added. In particular, objects held in memory only via anonymous lambda class instances, only via Builder class instances, etc. New anomaly detection: strings that contain exception stack traces and occupy a significant percentage of the heap. This signals that the app is in bad state, throwing exceptions all the time. Improved information about thread status in the “Thread stacks” report section UI improvements and bug …

Continue Reading

Fixing Azkaban’s excessive memory usage

Project Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track user’s workflows. Some Azkaban servers may grow huge, using tens of gigabytes of memory. Heap dumps that we analyzed were generated by the JVM running with 45G max heap. It routinely utilized more than 35G, had long GC pauses and was occasionally …

Continue Reading