Kestra supports Postgres and MySQL, among other databases; our internal benchmarks showed that Postgres performed much better than MySQL when running Kestra. We therefore generally recommend running Kestra in production with Postgres.

During our benchmarks, Postgres sustains 4,000 executions/min with a latency of 370 ms for our typical flows, and a latency of around 130 ms for a single launch. MySQL, for its part, cannot even sustain 1,000 executions/min, and has a latency of around 160 ms for a single launch.

But even so, some of our users were using MySQL without any issues, which prompted me to look into the issue.

Kestra uses its database for three things:

1. Storing workflow-related resources (flows, metadata, users, etc.)
2. Storing workflow execution data
3. An internal queue for asynchronous message passing between components

The internal queue is the most critical component; it lies at the heart of Kestra, and its Executor, which is responsible for the orchestration logic, relies heavily on it. The Executor processes an execution in a loop; at each step, it advances the workflow, then sends tasks to be executed to the Worker and an execution update event to trigger itself again. So it reads a message from the queue, processes the message, and sends another one. The JDBC queue operates via a SELECT ... FOR UPDATE statement; once the message has been processed, it is deleted within the same transaction. To optimize performance, the queue reads in batches (pollSize, 100 by default), and the Executor processes them in parallel using a number of threads configured by default to the number of available CPU cores.

Each execution can generate dozens or even hundreds of messages and thus DB transactions, which I suspect is the cause of the performance differences.

After verifying that the indexes were identical between the two databases and were being used correctly, I tested several things:

1. Configured the datasource, including enabling the prepared statement cache, which is disabled by default in MySQL but enabled in Postgres (see here for reference). No change.
2. Increased the pollSize to create larger batches (250). No change.
3. Decreased the pollSize to create smaller commits (10). Worse!.

I then asked Claude what it thought, and it replied:

MySQL’s default setting of sync_binlog=1 enforces a synchronous fsync of the binlog on every commit, and Kestra’s JDBC queue design performs extremely frequent commits (approximately 52 commits per execution, or about 2,600 commits per second during a 6-minute performance test with 18,000 executions), because message processing occurs within the same transaction as each query cycle of the queue. By default, Postgres performs one fsync on the WAL per commit and does not have a second log; MySQL performs two (redo log + bin log).

In summary, MySQL is configured by default to perform two fsync operations (system calls to synchronize file writes to disk) per commit, whereas Postgres performs only one. The second point relates to the binlog used for point-in-time recovery and replication between replicas: whereas the redo log is specific to MySQL’s storage engine (InnoDB in this case), the binlog is identical regardless of the storage engine.

When you disable the binlog (--skip-log-bin), performance is on par between MySQL and Postgres! :tada:

However, in a production environment, completely disabling the binlog is not a good idea; instead, you can choose what should be stored in the binlog (using binlog_row_image) or how often it is written (using sync_binlog).

Setting binlog_row_image to MINIMAL will create smaller events in the binlog, thereby reducing the number of bytes written per second. The tests were inconclusive; it is actually the number of writes that is the problem, not their size, which is consistent with the worste performance when pollSize is reduced.

Setting sync_binlog to 25 will write the binlog to disk every 25 commits instead of after every commit. The InnoDB redo log remains synchronized with every commit: a MySQL crash will not result in any data loss. In the event of an OS or server crash, up to 24 commits may be missing from the binlog (but not from the database), which only affects replication and point-in-time recovery.

Performance is then close to that of Postgres. This is a trade-off between performance and binlog durability: it’s up to you to choose the value based on your tolerance, but binlog configuration is a classic part of MySQL tuning.

To wrap it up, I had a negative preconception about MySQL’s performance, but knowing that some heavy users of Kestra (thousands of flows, millions of executions per month) were using it, I thoight I needed to look into it. And since I’m not an expert, Claude’s little help pointed me in the right direction and allowed me to learn a little more about how MySQL works under the hood.