How to save 24G of memory per application instance

Apache Pinot is a column-oritented distributed data store written in Java. It is open-source code used in applications where fast analytics is needed on immutable data. For example, at LinkedIn it was initially used to power the feature that allows members to see who has viewed their profile in real-time.

Thanks to JXRay analysis, it has been recently discovered that in some installations, Pinot instances running with big (over 60G) heaps wasted a huge amount of memory (nearly a half of the heap!) due to duplicate Strings and byte[] arrays. After a successful application of the fast and economical interning technique (FALFInterner) that we recommend, memory footprint was reduced by more than 24G. As always, after such a memory optimization, GC performance has improved a lot as well.

Leave a Reply

Your email address will not be published. Required fields are marked *