Java’s SecurityManager was an essential component of the Java security model. It allowed developers to enforce security rules by implementing check*() methods, such as checkExec(String cmd) for executing a command on the host machine or checkRead(String file) for reading a file. For more details, see the JavaDoc.

It was deprecated in Java 17 (JEP 411) and removed in Java 24 (JEP 486).

Within Kestra, the universal orchestration platform, we used it to:

• Restrict access to the host machine’s file system (allow/deny list).
• Prevent a command from being executed on the host machine.
• Prevent a thread from starting.
• Prevent for exiting the JVM.

Kestra has an extensible plugin system; since each plugin can execute untrusted code, providing advanced security is essential in critical runtime environments.

Following the removal of the SecurityManager, we replaced it with a Byte Buddy agent: the same rules, implemented differently.

Here, I’ll use file access restrictions as the sole example, but the same technique has been used for other security rules.

Byte Buddy is an open-source library that allows you to create or modify classes at runtime. It offers a high-level, declarative API that does not require knowledge of Java bytecode. It is used by many popular frameworks, such as Hibernate, Mockito, OpenTelemetry, …

Here is a by-the-book example of a Byte Buddy agent for intercepting file access via a RandomAccessFile.

static void main() {
    Instrumentation instrumentation = ByteBuddyAgent.install(); // <1>

    new AgentBuilder.Default() // <2>
        .with(new AgentBuilder.Listener.WithErrorsOnly(AgentBuilder.Listener.StreamWriting.toSystemError())) // <3>
        .type(ElementMatchers.is(RandomAccessFile.class)) // <4>
        .transform((builder, _, _, _, _) ->
                builder.method(
                    ElementMatchers.isMethod().and(ElementMatchers.named("open")) // <5>
                        .and(ElementMatchers.takesArguments(String.class, int.class))
                        .and(ElementMatchers.returns(void.class))
                        .and(ElementMatchers.isPrivate())
                    )
                    .intercept(MethodDelegation.to(FileInterceptor.class)) // <6>
            )
            .installOn(instrumentation); // <7>
}

public static class FileInterceptor { // <8>
    public static void open(String path, int mode) {
        throw new RuntimeException("You shall not pass");
    }
}

1. Install the Byte Buddy agent , it’s a kind of meta agent (my term) that makes it easy to install your own agent. It returns an instance of Instrumentation.
2. To create our agent, we’ll use Byte Buddy’s AgentBuilder, which offers a fluid API that allows you to chain transformations.
3. Log errors to stderr.
4. Target the RandomAccessFile class. ElementMatchers is one of Byte Buddy’s core classes, allowing you to target elements of your application code such as a class, a method, a field, etc.
5. Target the method named open, which takes a String and an int as arguments, returns nothing, and is private. This is the method in the RandomAccessFile class that is called every time a file is opened.
6. Delegates the implementation of this method to the FileInterceptor class: this is one form of transformation, where the transformed method is replaced by another.
7. Installs the agent in the application.
8. A static class containing a static method whose signature matches the intercepted method: this method will be called instead of the corresponding method in the RandomAccessFile class.

Unfortunately, this by-the-book example does not work for internal JDK classes loaded when the application starts, that is, via the Bootstrap class loader. For these classes, you must specifically configure the agent and use an Advice rather than method delegation.

Byte Buddy Advice, although not mentioned in the documentation, is more powerful and flexible than method delegation; it incorporates the principles of aspect-oriented programming (AOP) by allowing you to define a method that is called before or after the target method is called.

static void main() {
    Instrumentation instrumentation = ByteBuddyAgent.install();

    new AgentBuilder.Default()
        .ignore(ElementMatchers.none()) // <1>
        .with(AgentBuilder.RedefinitionStrategy.REDEFINITION) // <2>
        .with(new AgentBuilder.Listener.WithErrorsOnly(AgentBuilder.Listener.StreamWriting.toSystemError()))
        .type(ElementMatchers.is(RandomAccessFile.class))
        .transform((builder, _, _, _, _) ->
                builder.visit( // <3>
                    Advice.to(FileInterceptor.class).on(
                    ElementMatchers.isMethod().and(ElementMatchers.named("open"))
                            .and(ElementMatchers.takesArguments(String.class, int.class))
                            .and(ElementMatchers.returns(void.class))
                            .and(ElementMatchers.isPrivate())
                    ))
            )
            .installOn(instrumentation);
}

public static class FileInterceptor { // <4>
    @Advice.OnMethodEnter
    public static void openEnter(String path, int mode) {
        IO.println("Try to access file: " + path);
    }

     @Advice.OnMethodExit
    public static void openExit(String path, int mode) {
        throw new RuntimeException("You shall not pass");
    }
}

1. By default, JDK classes are ignored; you must configure the agent to ignore no classes.
2. By default, classes that have already been loaded are not re-transformed; you must configure the agent to re-transform already-loaded classes via redefinition.
3. Instead of transforming a method, the method is visited with an advice.
4. An advice is defined using methods annotated with @Advice.OnMethodEnter, which is executed before entering the target method, and/or @Advice.OnMethodExit, which is executed after exiting the method.

There you have it, now you know all the secrets of transforming the JVM’s internal classes!