Profiling the JVM on Linux: A Hybrid Approach

Friday, July 7, 2017

I hope you're outraged that your performance tools are lying to you. For quite a while, many Java sampling profilers have been known to blatantly misrepresent reality. In a nutshell, stack sampling using the documented JVMTI GetStackTrace method produces results that are biased towards safepoints, and not representative of the real CPU processing performed by your program. Over the years, alternative profilers popped up, trying to fix this problem by using AsyncGetCallTrace, a less-documented API that doesn't wait for a safepoint, and can produce more accurate results. Simply calling AGCT from a timer signal handler gives you a fairly reliable way...
no comments

Tracing .NET Core on Linux with USDT and BCC

Sunday, April 2, 2017

In my last post, I lamented the lack of call stack support for LTTng events in .NET Core. Fortunately, being open source, this is somewhat correctable -- so I set out to produce a quick-and-dirty patch that adds USDT support for CoreCLR's tracing events. This post explores some of the things that then become possible, and will hopefully become available in one form or another in CoreCLR in the future. Very Brief USDT Primer USDT (User Statically Defined Tracing) is a lightweight approach for embedding static trace markers into user-space libraries and applications. I've taken a closer look a year ago when...

Tracing Runtime Events in .NET Core on Linux

Thursday, March 30, 2017

After exploring the basic profiling story, let's turn to ETW events. On Windows, the CLR is instrumented with a myriad of ETW events, which can be used to tackle very hard problems at runtime. Here are some examples of these events: Garbage collections Assembly load/unload Thread start/stop (including thread pool threads) Object allocations Exceptions thrown, caught, filtered Methods compiled (JIT) By collecting all of, or a subset of, these events, you can get a very nice picture of what your .NET application is doing. By combining these with Windows kernel events for CPU sampling, file accesses, process creations and more -- you have a fairly...

Profiling a .NET Core Application on Linux

Monday, February 27, 2017

In the same vein of my previous post on analyzing core dumps of .NET Core applications on Linux, let's take a look at what it takes to do some basic performance profiling. When starting out, here are a few things I wrote down that would be nice to do: CPU profiling (sampling) to see where the CPU bottlenecks are Grabbing stacks for interesting system events (file accesses, network, forks, etc.) Tracing memory management activity such as GCs and object allocations Identifying blocked time and the block and wake-up reasons With this task list in mind, let's get started! Collecting Call Stacks of .NET Core Processes Generally speaking, a...
one comment

USDT/BPF Tracing Tools: Java, Python, Ruby, Node, MySQL, PostgreSQL

Friday, December 23, 2016

A lot of high-level languages, runtimes, and libraries people use on Linux have USDT probes embedded in them. In some cases, you have to compile with a specific flag to get the probes embedded (e.g. in Node.js), and in other cases they are part of the default package on most major distributions (e.g. in OpenJDK). Some examples of information these probes can provide include: Garbage collection events and latencies in Java and Node Method invocations and latencies in Python and Ruby Object allocations in Ruby and Java Thread start and stop events in Java Class load events in Java and Ruby This...
no comments

Two New eBPF Tools: memleak and argdist

Sunday, February 14, 2016

Warning: This post requires a bit of background. I strongly recommend Brendan Gregg's introduction to eBPF and bcc. With that said, the post below describes two new bcc-based tools, which you can use directly without perusing the implementation details. A few weeks ago, I started experimenting with eBPF. In a nutshell, eBPF (introduced in Linux kernel 3.19 and further improved in 4.x kernels) allows you to attach verifiably-safe programs to arbitrary functions in the kernel or a user process. These little programs, which execute in kernel mode, can collect performance information, trace diagnostic data, and aggregate statistics that are then...
2 comments

Shared Memory Queue, Adaptive pthread_mutex, and Dynamic Tracing

Friday, January 22, 2016

This blog post is also on GitHub in its entirety. If you prefer to read it there along with the code, I won't mind. Go ahead. In one of my recent training classes, I was asked to demonstrate some practical uses of shared memory. My knee-jerk reply was that shared memory can be used for inter-process communication and message-passing. In fact, most IPC mechanisms are based on shared memory in their implementation. The question was whether it's worth the effort to build a message-passing interface on top of shared memory queues, or whether sockets or pipes could produce a better result...
4 comments

Wrapping Up DotNext 2015

Saturday, January 2, 2016

A few weeks ago, I had the honor of being invited to speak at DotNext 2015, Russia's only .NET conference and one of the leading developer conferences in the country. As some of my readers probably know already, I was born in the USSR, so I speak Russian with a heavy Israeli accent but can understand both written and spoken Russian very well. The fact it was my wife's birthday and we could elope for a weekend of wintery weather and hardcore CLR internals only added to my resolve. I proposed two talks, and the organizers had such difficulty picking...
no comments

Live360! and BuildStuff Talks: SIMD, Visual Studio Diagnostic Hub, and Swift

Wednesday, December 9, 2015

I'm writing this on the plane back home from a week-long trip to Orlando, Vilnius, and Kiev, where I had the chance to speak at Live360! and BuildStuff; I've just counted and it's my tenth flight in three weeks, which is quite insane. But this is my second-to-last trip for 2015 -- the last one is going to be in December to DotNext Moscow. SIMD In this talk, we discussed vector registers and instructions that you could use from other languages like FORTRAN and C++ for more than 15 years. Starting from the MMX instruction set extensions in the 1997...
2 comments

Materials from TechDays Netherlands 2015

Monday, August 31, 2015

Oops! This was sitting in my queue for several months now, and I just noticed it needs to be published. But better late than never I guess. Here goes: I've been lucky enough to be invited to speak at TechDays Netherlands again this year. This time I was asked to do four talks on some of my favorite subjects -- performance optimization, debugging, and diagnostics. Same as last year, the conference was impeccably organized. I'm really looking forward to next year's TechDays :-) In the meantime, here are the materials from my talks. Making .NET Applications Faster My usual favorite on improving...
7 comments