{"id":1459,"date":"2022-05-04T12:01:19","date_gmt":"2022-05-04T10:01:19","guid":{"rendered":"https:\/\/www.loicmathieu.fr\/wordpress\/?p=1459"},"modified":"2022-05-04T12:01:19","modified_gmt":"2022-05-04T10:01:19","slug":"profiler-un-pod-dans-kubernetes-avec-kubectl-flame","status":"publish","type":"post","link":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/profiler-un-pod-dans-kubernetes-avec-kubectl-flame\/","title":{"rendered":"Profiling a pod in Kubernetes with kubectl flame"},"content":{"rendered":"<p>Kubectl flame is a plugin for kubectl that allows you to profile applications in production with low overhead by generating FlameGraphs.<\/p>\n<p>It is a Yahoo project.<\/p>\n<p>It is installed via <a href=\"https:\/\/krew.sigs.k8s.io\/\" rel=\"noopener\" target=\"_blank\">krew<\/a>, the plugin manager for kubectl, and allows you to generate FlameGraphs for applications in Go, Java (all JVM languages), Python, Ruby, and NodeJS.<\/p>\n<p>After installing krew, you can install flame via: <code>kubectl krew install flame<\/code>.<\/p>\n<p>For each supported language, it will use a different profiler to generate a FlameGraph. These profilers are of <strong>sampling<\/strong> type, and based on this sampling information (how many times a method has been called), it will generate a FlameGraph.<\/p>\n<h2>FlameGraph<\/h2>\n<p>A FlameGraph is a way to visualize the profile of an application allowing to instantly (or very quickly) detect the most frequent code path.<\/p>\n<p>It will display, on the x-axis, the population (usually the method) whose size is proportional to the number of samples in the profile,\nand on the y-axis, the depth in the call stack.<\/p>\n<p>More information on Brendan Gregg&#8217;s website: <a href=\"https:\/\/www.brendangregg.com\/flamegraphs.html\" rel=\"noopener\" target=\"_blank\">FlameGraphs<\/a>.<\/p>\n<p>Here is an example of a FlameGraph from Brendan Gregg&#8217;s site.<\/p>\n<img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/cpu-mysql-updated.png?resize=640%2C401&#038;ssl=1\" alt=\"\" width=\"640\" height=\"401\" class=\"alignnone size-full wp-image-1465\" srcset=\"https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/cpu-mysql-updated.png?w=1000&amp;ssl=1 1000w, https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/cpu-mysql-updated.png?resize=300%2C188&amp;ssl=1 300w, https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/cpu-mysql-updated.png?resize=768%2C481&amp;ssl=1 768w, https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/cpu-mysql-updated.png?resize=431%2C270&amp;ssl=1 431w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/>\n<h2>kubetcl without and with the flame plugin<\/h2>\n<p>To generate a FlameGraph of a Java application deployed in a <strong>pod-my-app<\/strong> pod using async-profiler, and assuming that the Java process has a pid of 1, we would need to do something close to this:<\/p>\n<pre>\nwget https:\/\/github.com\/jvm-profiling-tools\/async-profiler\/releases\/download\/v2.7\/async-profiler-2.7-linux-x64.tar.gz\nkubectl cp async-profiler-2.7-linux-x64.tar.gz pod-my-app:\/tmp\nkubectl exec -ti pod-my-app -- bash\n$ tar xvf \/tmp\/async-profiler-2.7-linux-x64.tar.gz\n$ \/tmp\/async-profiler-2.7-linux-x64\/profiler.sh -d 60 -e cpu -f \/tmp\/profile-cpu.html 1\n$ exit\nkubectl cp pod-my-app:\/tmp\/profile-cpu.html .\/profile-cpu.html\n<\/pre>\n<p>Lots of steps, some requiring elevated privileges on the Kubernetes cluster.<\/p>\n<p>With kubectl&#8217;s flame plugin, a single command is enough!<\/p>\n<pre>\nkubectl flame -t  30s --lang java -e cpu -f profile-cpu.svg pod-my-app\n<\/pre>\n<p>We can note here a small difference: the Java profiler used by kubectl flame is async-profiler in version 1 which generates FlameGraphs in SVG format, while version 2 generates them in HTML format. Kubectl flame being an open source project, I opened a PR to upgrade to async-profiler 2 but haven&#8217;t had time to finish it yet.<\/p>\n<h2>kubetcl flame in details<\/h2>\n<p>Given the following command:<\/p>\n<pre>\nkubectl flame -t  30s --lang java -e cpu -f profile-cpu.svg pod-my-app\n<\/pre>\n<ul><li><strong>-t 30s<\/strong> : will profile the application for 30s.<\/li>\n\n<li><strong>&#8211;lang java<\/strong> : for Java \/ JVM applications.<\/li>\n\n<li><strong>-e cpu<\/strong> : profile the CPU event, all events supported by async-profiler are supported.<\/li>\n\n<li><strong>-f profile-cpu.svg<\/strong> : will save the profile in the local file profile-cpu.svg.<\/li>\n<\/ul>\n<p>The most interesting thing about kubectl flame is that we will use the same command line with the same options for the different languages it supports.<\/p>\n<p>Example, for an application in Go started in the <strong>hello-go<\/strong> pod:<\/p>\n<pre>\nkubectl flame -t 30s --lang go -f profile-cpu.svg hello-go\n<\/pre>\n<p>When no event is passed to the command line, the default event (usually the CPU) will be profiled.<\/p>\n<p>The way kubectl flame work is as follows:<\/p>\n<ul><li>A Kubernetes Job is launched, this allows us to ensure that the profiling is running smoothly and to restart it if necessary.<\/li>\n\n<li>The Job will launch a Pod.<\/li>\n\n<li>The Pod will use a profiler image corresponding to the language, then locate the application process to be profiled (the <code>--pgrep<\/code> option allows you to specify the name of the command to search for).<\/li>\n\n<li>The resulting profile file will be copied locally.<\/li>\n<\/ul>\n<p>For more information about kubectl flame you can check out its GitHub repo: <a href=\"https:\/\/github.com\/yahoo\/kubectl-flame\" rel=\"noopener\" target=\"_blank\"><a href=\"https:\/\/github.com\/yahoo\/kubectl-flame\">https:\/\/github.com\/yahoo\/kubectl-flame<\/a><\/a>.\n<\/p>","protected":false},"excerpt":{"rendered":"<p>Kubectl flame is a plugin for kubectl that allows you to profile applications in production with low overhead by generating FlameGraphs. It is a Yahoo project. It is installed via krew, the plugin manager for kubectl, and allows you to generate FlameGraphs for applications in Go, Java (all JVM languages), Python, Ruby, and NodeJS. After installing krew, you can install flame via: kubectl krew install flame. For each supported language, it will use a different profiler to generate a FlameGraph&#8230;.<p class=\"read-more\"><a class=\"btn btn-default\" href=\"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/profiler-un-pod-dans-kubernetes-avec-kubectl-flame\/\"> Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[9],"tags":[],"class_list":["post-1459","post","type-post","status-publish","format-standard","hentry","category-informatique"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":1258,"url":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/profiler-une-application-java-dans-un-conteneur-deploye-dans-kubernetes-avec-jfr-java-flight-recorder\/","url_meta":{"origin":1459,"position":0},"title":"(Fran\u00e7ais) Profiler une application Java dans un conteneur d\u00e9ploy\u00e9 dans kubernetes avec JFR &#8211; Java Flight Recorder","author":"admin","date":"Monday April 12th, 2021","format":false,"excerpt":"Sorry, this entry is only available in Fran\u00e7ais.","rel":"","context":"In &quot;informatique&quot;","block_context":{"text":"informatique","link":"https:\/\/www.loicmathieu.fr\/wordpress\/category\/informatique\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":19,"url":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/ruby-on-rails\/","url_meta":{"origin":1459,"position":1},"title":"Ruby on Rails","author":"admin","date":"Thursday February 15th, 2007","format":false,"excerpt":"Bonjour, d'habitude au ton bucolique des vacances ou revendicatif des coups de gueules, aujourd'hui le ton de ce post va \u00eatre technophile. En effet, je bosse dans l'informatique qui est donc un de mes centre d'int\u00e9r\u00eat, et je vous livre ici mon premier message sur les nouvelles technologies. J'ai tester\u2026","rel":"","context":"In &quot;informatique&quot;","block_context":{"text":"informatique","link":"https:\/\/www.loicmathieu.fr\/wordpress\/category\/informatique\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1153,"url":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/profiler-une-image-native-graalvm-avec-perf\/","url_meta":{"origin":1459,"position":2},"title":"Profiling a GraalVM native image with perf","author":"admin","date":"Monday November 16th, 2020","format":false,"excerpt":"The GraalVM native-image tool allows you to generate a native executable (or native image) from your Java application. This native executable will start very quickly and have a much smaller memory footprint than a traditional Java application; at the cost of reduced peak performance and a relatively high packaging build\u2026","rel":"","context":"In &quot;informatique&quot;","block_context":{"text":"informatique","link":"https:\/\/www.loicmathieu.fr\/wordpress\/category\/informatique\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/flamegraph-2.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/flamegraph-2.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/flamegraph-2.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/loicmathieu.fr\/wordpress\/wp-content\/uploads\/flamegraph-2.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":1041,"url":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/introduction-a-jmh-java-microbenchmark-harness\/","url_meta":{"origin":1459,"position":3},"title":"(Fran\u00e7ais) Introduction \u00e0 JMH &#8211; Java Microbenchmark Harness","author":"admin","date":"Wednesday April 29th, 2020","format":false,"excerpt":"Sorry, this entry is only available in Fran\u00e7ais.","rel":"","context":"In &quot;informatique&quot;","block_context":{"text":"informatique","link":"https:\/\/www.loicmathieu.fr\/wordpress\/category\/informatique\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":68,"url":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/friendly-url\/","url_meta":{"origin":1459,"position":4},"title":"Friendly URL","author":"admin","date":"Monday February 11th, 2008","format":false,"excerpt":"Dans cet article, je vais vous parler des Friendly URL et vous donner un exemple d'impl\u00e9mentation en Java\/J2EE. D'autres technologie permettent facilement de mettre en place ce type de fonctionnalit\u00e9. On peut parler de user-friendly ou de search-engine-friendly URL. Pour ma part, une bonne Friendly URL est pour les deux!\u2026","rel":"","context":"In &quot;informatique&quot;","block_context":{"text":"informatique","link":"https:\/\/www.loicmathieu.fr\/wordpress\/category\/informatique\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":827,"url":"https:\/\/www.loicmathieu.fr\/wordpress\/informatique\/devfest-lille-2018\/","url_meta":{"origin":1459,"position":5},"title":"Devfest Lille 2018","author":"admin","date":"Friday July  6th, 2018","format":false,"excerpt":"gRPC ----- - Rappel r\u00e9seau, internet - Probl\u00e9matique du format de donn\u00e9e dans les systeme distribu\u00e9 => modification difficile - Historique des protocoles RCP : CORBA, RMI, EJB, SOAP, REST -> 2008 Protocol Buffers : s\u00e9rialization (pr\u00e9mice du RCP chez Google : open sourcing partiel) -> 2009 Thrift (Fb) ->\u2026","rel":"","context":"In &quot;informatique&quot;","block_context":{"text":"informatique","link":"https:\/\/www.loicmathieu.fr\/wordpress\/category\/informatique\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/posts\/1459","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/comments?post=1459"}],"version-history":[{"count":8,"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/posts\/1459\/revisions"}],"predecessor-version":[{"id":1468,"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/posts\/1459\/revisions\/1468"}],"wp:attachment":[{"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/media?parent=1459"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/categories?post=1459"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.loicmathieu.fr\/wordpress\/wp-json\/wp\/v2\/tags?post=1459"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}