I’ve been using Kafka a lot lately, and in Kafka a lot of things are byte arrays, even headers!<\/p>\n
As I have many components that exchange messages, I added headers to help with message tracking, including a So I had to transform a Looking a bit further, Guava has a solution based on bitwise calculation via the To compare the three, nothing better than JMH – The Java Microbenchmark Harness<\/a>. This tool allows to write relevant micro-benchmarks taking into account the internal characteristics of the JVM. It also offers integrated tools to analyze the performance of our tests (profiling, disassembling, …). If you don’t know JMH, you can refer to this article: INTRODUCTION TO JMH – JAVA MICROBENCHMARK HARNESS<\/a>.<\/p>\n First of all, I configured the benchmark with a Thread State so that the setup is played for each Thread. Among other things, I created a Then, I implement a benchmark method for each way of converting a The full benchmark is accessible here<\/a>.<\/p>\n All the tests were run on my laptop: Intel(R) Core(TM) i7-8750H 6 cores (12 with hyperthreading) – Ubuntu 19.10.<\/p>\n The Java version used was: First observation: my intuition was good, instantiating a Surprisingly Kafka, which is known for its performance, has a slower implementation than Guava or that via a The results obtained via Let’s put aside the implementation via a For this I will use the profiling capabilities of JMH via the If we profile the memory allocations via We can clearly see the advantage of reusing the On the other hand, the memory allocations are close for the three other implementations, so that does not give us much more information.<\/p>\n Now let’s try to profile the CPU with The results are not easily understandable (to see them this is here<\/a>), some observations :<\/p>\n Finally, out of curiosity, I looked at the code for Be careful what you measure, the implementation via a Reusing a Follow the force, read the code;)<\/p>\n Although JMH is a great tool, it needs technical skills and a lot of time to fully analyze its results. Even if the observed differences are not fully explained; I’m still happy with my little experimentation;)<\/p>","protected":false},"excerpt":{"rendered":" I’ve been using Kafka a lot lately, and in Kafka a lot of things are byte arrays, even headers! As I have many components that exchange messages, I added headers to help with message tracking, including a timestamp header which has the value System.currentTimeMillis(). So I had to transform a long into a byte array; in a very naive way, I coded this: String.valueOf(System.currentTimeMillis()).getBytes(). But instantiating a String each time a header is created does not seem very optimal to…timestamp<\/code> header which has the value System.currentTimeMillis()<\/code>.<\/p>\nlong<\/code> into a byte array; in a very naive way, I coded this: String.valueOf(System.currentTimeMillis()).getBytes()<\/code>. But instantiating a String<\/code> each time a header is created does not seem very optimal to me!<\/p>\nLongs<\/code> class, as well as Kafka via its LongSerializer<\/code>. You can also use a ByteBuffer<\/code> to perform the conversion.<\/p>\nThe benchmark<\/h2>\n
ByteBuffer<\/code> per thread to compare implementations with and without re-use of the buffer.<\/p>\n\n@State(Scope.Thread)\n\/\/ other JMH annotations ...\npublic class LongToByteArray {\n private static final LongSerializer LONG_SERIALIZER = new LongSerializer();\n\n long timestamp;\n ByteBuffer perThreadBuffer;\n\n @Setup\n public void setup() {\n timestamp = System.currentTimeMillis();\n perThreadBuffer = ByteBuffer.allocate(Long.BYTES);\n }\n\n \/\/ benchmark methods\n}\n<\/pre>\nlong<\/code> to byte[]<\/code>. I implemented two different algorithms for ByteBuffer<\/code>: one with an instantiation of a buffer on each conversion, and the other with a recycling of an existing buffer using the ByteBuffer<\/code> instantiated in the benchmark setup phase.<\/p>\n \n@Benchmark\n public byte[] testStringValueOf() {\n return String.valueOf(timestamp).getBytes();\n }\n\n @Benchmark\n public byte[] testGuava() {\n return Longs.toByteArray(timestamp);\n }\n\n @Benchmark\n public byte[] testKafkaSerde() {\n return LONG_SERIALIZER.serialize(null, timestamp);\n }\n\n @Benchmark\n public byte[] testByteBuffer() {\n ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);\n buffer.putLong(timestamp);\n return buffer.array();\n }\n\n @Benchmark\n public byte[] testByteBuffer_reuse() {\n perThreadBuffer.putLong(timestamp);\n byte[] result = perThreadBuffer.array();\n perThreadBuffer.clear();\n return result;\n }\n<\/pre>\nThe results<\/h2>\n
openjdk version "11.0.7" 2020-04-14<\/code>.<\/p>\n\nBenchmark Mode Cnt Score Error Units\nLongToByteArray.testByteBuffer avgt 5 4,429 \u00b1 0,204 ns\/op\nLongToByteArray.testByteBuffer_reuse avgt 5 5,655 \u00b1 0,793 ns\/op\nLongToByteArray.testGuava avgt 5 6,422 \u00b1 0,428 ns\/op\nLongToByteArray.testKafkaSerde avgt 5 9,103 \u00b1 1,515 ns\/op\nLongToByteArray.testStringValueOf avgt 5 39,660 \u00b1 4,372 ns\/op\n<\/pre>\n
String<\/code> for each conversion is very bad, 4 to 10 times slower than all the other implementations. When we look at the result of the conversion, we understand why. By using a String<\/code> we no longer convert a 64bit number but a character string where each character (each digit of the number) is coded on a byte. So we do not compare exactly the same thing since the result of the conversion via a String<\/code> will give you an array of 13 bytes, while a Long<\/code> can be encoded in 8 bytes, as gives us the conversion via Guava, Kafka or a ByteBuffer.<\/p>\nByteBuffer<\/code>.<\/p>\nByteBuffer<\/code> are surprising, the instantiation of a ByteBuffer<\/code> for each conversion is more efficient than the reuse of an existing one (which requires a clean of the buffer) .<\/p>\nA little more detailed analysis<\/h2>\n
String<\/code> and try to better understand the differences between the other implementations.<\/p>\n-prof<\/code> option.<\/p>\n-prof gc<\/code> we have the following results:<\/p>\n\nLongToByteArray.testByteBuffer \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 avgt 5 4,492 \u00b1 0,708 ns\/op\nLongToByteArray.testByteBuffer:\u00b7gc.alloc.rate\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 avgt 5 4635,903 \u00b1 712,889 MB\/sec\nLongToByteArray.testByteBuffer_reuse \u00a0 \u00a0 \u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 avgt 5 5,798 \u00b1 1,139 ns\/op\nLongToByteArray.testByteBuffer_reuse:\u00b7gc.alloc.rate avgt 5 \u2248 10\u207b\u2074 MB\/sec\nLongToByteArray.testGuava\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 avgt 5 6,939 \u00b1 0,899 ns\/op\nLongToByteArray.testGuava:\u00b7gc.alloc.rate\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 avgt 5 3000,818 \u00b1 376,613 MB\/sec\nLongToByteArray.testKafkaSerde \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0 avgt 5 9,317 \u00b1 0,842 ns\/op\nLongToByteArray.testKafkaSerde:\u00b7gc.alloc.rate \u00a0 \u00a0 \u00a0 avgt 5 4467,791 \u00b1 405,897 MB\/sec\n<\/pre>\n
ByteBuffer<\/code>: there is no memory allocation, while by creating a new buffer for each conversion, we have 4 GB\/s of memory allocation !<\/p>\n-prof perf<\/code> which will use the perf tool to profile the application.<\/p>\nByteBuffer<\/code> seems to involve a lot more CPU branches, maybe this is the cause of the performance difference.<\/li>\n\nHeapByteBuffer.putLong()<\/code>, this is the implementation used via ByteBuffer<\/code> because I don’t do any direct allocation. This uses a Unsafe.putLongUnaligned()<\/code> method. Unsafe<\/code> is known for its high performance implementations (but should not be used by everyone), here this method is annotated with @HotSpotIntrinsicCandidate<\/code>code>@HotSpotIntrinsicCandidate<\/code<\/a> which means that an intrinsic<\/strong> may exists for it and could explain its difference in performance with other implementations. An intrinsic<\/strong> can be seen as a piece of native code, optimized for your OS \/ CPU architecture, which the JVM will substitute for the Java implementation of the method under certain conditions.<\/p>\nConclusion<\/h2>\n
String<\/code> does not generate the same array of bytes as the others, and is therefore much less efficient.<\/p>\nByteBuffer<\/code> is not always the best solution, as the cost of recycling canbe significant. Allocations are not very expensive within the JVM, and sometimes it is better to allocate than execute more instructions.<\/p>\n