A native executable contains a minimalist JVM called SubstratVM, this one has some limitations:<\/p>\n
- Partial support for reflection<\/li>\n\n
- Partial support for dynamic proxy<\/li>\n\n
- Partial support for dynamic class loading<\/li>\n\n
- No JNI<\/li>\n\n
- No JVMTI<\/li>\n<\/ul>\n
No support for JMVTI<\/strong> means no support for Java agents, JMX, Java profilers, Java debuggers, Java Flight Recorder and Java Mission Control, as well as all the tools delivered with the JDK (jps, jstack, jmap).<\/p>\n
For all the needs covered by these tools, you must therefore use a solution integrated into the application (for example, replacing the JMX metrics by Prometheus metrics), or standard tools provided by your operating system.<\/p>\n
To profile the execution of an application, the Linux OS has a very powerful tool: perf<\/strong>.<\/p>\n
The perf tool has many features, it can access all OS and CPU metrics (performance counters, hence its name: perf) and profile the application in many different ways.<\/p>\n
Using perf to profile the CPU<\/h3>\n
The perf tool will use the symbols integrated in the binary of your application to make the link between a memory pointer and the corresponding Java method (or the system call).<\/p>\n
By default, these symbols are not integrated into the native executables, so you must ask the native-image tool to leave them there via the options
H:-DeleteLocalSymbols -H:+PreserveFramePointer<\/code>.<\/p>\nIf you want to test these steps, you can use the Quarkus getting-started<\/a> application. Quarkus has easy native-image support, just add to your application’s application.properties the property
quarkus.native.additional-build-args=-H:-DeleteLocalSymbols,-H:+PreserveFramePointer<\/code> and it will automatically add these options to the command line of the native-image tool.<\/p>\nAfter having generated your native executable, you can launch it, then recover its PID; we will use this one in the command line of the perf tool.<\/p>\n
Once your application is launched, and ideally under load (you can use a tool such as wrk to generate load), you can profile it via the following perf command:
perf record -F 99 -p PID --call-graph dwarf sleep 10<\/code>.<\/p>\n- record<\/strong> : asks perf to start profiling the application.<\/li>\n\n
- -F 99<\/strong> : profiles at 99 Hertz, which means 99 samples per second.<\/li>\n\n
- -p PID<\/strong> : asks perf to profile this particular PID (the one of your application).<\/li>\n\n
- –call-graph dwarf<\/strong> : tells perf to use the symbols built into your application (ELF symbol).<\/li>\n\n
- sleep 10<\/strong> : as perf profiles a PID and not a command, it must be given a command to execute. When this command is complete, perf will stop profiling your application. By using
sleep 10<\/code> as a command, we will therefore profile the application for 10 seconds.<\/li>\n<\/ul>\nWhen the command is finished, perf will have generated a data file containing the profile of your application (CPU profile here, because it has not been told which event it should profile):
perf.data<\/code>.<\/p>\nYou can use the following command to view this profile in the console:
perf report --stdio<\/code>, you will then have a result close to this one:<\/p>\n\n# Children Self Command Shared Object Symbol >\n# ........ ........ ............... ................................... ..............................................................................................................................>\n#\n 13.47% 0.00% tloop-thread-19 libpthread-2.31.so [.] start_thread\n |\n ---start_thread\n IsolateEnterStub_PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df_06195ea7c1ac11d884862c6f069b026336aa4f8c\n JavaThreads_threadStartRoutine_241bd8ce6d5858d439c83fac40308278d1b55d23\n Thread_run_857ee078f8137062fcf27275732adf5c4870652a\n FastThreadLocalRunnable_run_0329ad2c5210a091812879bcecd155c58e561e60\n ThreadExecutorMap$2_run_66c8943ee6536a10df07f979fb6cd278adcf96bc\n SingleThreadEventExecutor$4_run_1b47df7867e302a2fb7f28d7657a73e92f89d91f\n | \n |--12.64%--NioEventLoop_run_be89580b4d16514bef6e948913d2ed21c5e4f679\n | | \n | |--5.14%--NioEventLoop_processSelectedKeys_9a76c58d657b781ee037bbb65f41f01d2eb54e7c\n | | NioEventLoop_processSelectedKeysOptimized_c36ca161e53573665bc03cb5392e91c123bcd359\n | | NioEventLoop_processSelectedKey_3a0d92ce472db6c251df4485227a85acb9d3a1ca\n | | AbstractNioByteChannel$NioByteUnsafe_read_45358e803c643a6380776021e488e79d981b159d\n<\/pre>\n
And this over thousands of lines … not easy to analyze eh?<\/p>\n
To easily analyze a profile generated by perf, you can use the FlameGraph tool, accessible here: https:\/\/github.com\/brendangregg\/FlameGraph<\/a><\/a><\/p>\n
A FlameGraph is a way of visualizing the profile of an application allowing to instantly detect the most frequent code path. It will display on the x-axis the population (generally the method) whose size is proportional to the number of samples in the profile, and on the y-axis the depth in the stack. More information on FlameGraphs here<\/a>.<\/p>\n
We can note a small problem in the profile data, the column Command<\/strong> instead of containing the given command, contains the name of the thread (truncated moreover). This is a bug in the native-image tool, to work around it we will use sed<\/strong> to modify the profile data before using it in the FlameGraph tool. The value of the Command<\/strong> column is found at the base of the FlameGraph, it must normally be unique for the aggregation of stacks to be done.<\/p>\n
The first step is to use
perf script<\/code> to extract the profile data into a textual format, then usesed<\/code> to correct the command name so that you can then generate a FlameGraph.<\/p>\n\nperf script > out.perf\nsed -i -E \"s\/cutor-thread-[0-9]*\/executor-thread\/\" out.perf\nsed -i -E \"s\/ntloop-thread-[0-9]*\/eventloop-thread\/\" out.perf\nsed -i -E \"s\/tloop-thread-[0-9]*\/eventloop-thread\/\" out.perf\n~\/FlameGraph\/stackcollapse-perf.pl out.perf | ~FlameGraph\/flamegraph.pl > perf.svg\n<\/pre>\n
Here is an example of a generated FlameGraph:<\/p>\n
![\"\"](\"https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/flamegraph-1.png?resize=640%2C502&ssl=1\")
\nWhat I like most about FlameGraphs is that you can zoom in on them by clicking on a frame:<\/p>\n
![\"\"](\"https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/flamegraph-2.png?resize=640%2C615&ssl=1\")
\nUsing perf to profile memory<\/h3>\n
To profile memory, we’ll use the same technique with a slightly modified command.<\/p>\n
There are several ways to profile memory with perf, you can ask perf to record memory related OS events, profile one of the system methods that allocate memory, or use
pef mem<\/code>. We will use the last solution.<\/p>\nFor this, you must start your application using the perf tool:
perf mem record --call-graph dwarf -F 99 .\/getting-started-1.0-SNAPSHOT-runner<\/code>.<\/p>\nWhen the application stops, perf will save the profile data on disk which can then be used in the same way as the CPU profile data (via perf report, perf script and the FlameGraph tool).<\/p>\n
Pour aller plus loin<\/h3>\n
- -F 99<\/strong> : profiles at 99 Hertz, which means 99 samples per second.<\/li>\n\n
- record<\/strong> : asks perf to start profiling the application.<\/li>\n\n