Introduction to Mixins Understanding Mixin Architecture

Before you start writing mixins, it is important to develop an understanding of the basic concepts that allow them to work. This section gives a brief introduction to these concepts. Even though you may be familiar with all of the information presented here I recommend at least skim reading the first 3 sections as they introduce the example case I will be using to demonstrate how mixins are applied, and some peculiar corners of Java and the JVM which are leveraged heavily in mixins.

This is not a tutorial! This introduction is not intended as a tutorial, for more details about mixin implementation consult the mixin example code in the examples in the Sponge repository.

NOTE

If you already have a comprehensive understanding of bytecode, name binding and frankly if you already know your INVOKESPECIAL from your INVOKEVIRTUAL, then feel free to skip to section 4 where mixins themselves are introduced.

1. Thinking with portals mixins - the example case

In order to be able to think about how mixins work, I will present a canned example. Note that this example is purely made up for the purposes of demonstration and isn't anything like the namesakes in the real code base!

In our canned example, we will be looking at a class EntityPlayer, whose immediate (and only) superclass is Entity. We can represent this in a UML-esque manner like this:

Figure 1 - a simple (imaginary) class hierarchy

In mixin jargon EntityPlayer is the target class, it is the class which the mixin will be applied to.

To flesh out the example, let's add some imaginary fields and methods to our imaginary example classes:

Figure 2 - adding some imaginary fields and methods to our example

This representation is chosen deliberately to represent which members represent our class's public surface area, with pubic methods and fields jutting outside of the class body since they are visible to other objects. This "view" that the outside world has of our class is an important concept to keep in mind when working with mixins.

Notice that the inherited public methods from Entity also represent part of our class's publicly visible surface area, and the "ghost" methods getHealth and setHealth which are inherited from the parent class represent this presence in the class's overall external appearance.

Before working with mixins, it is vital to have a deep understanding of the two Java keywords this and super. That may seem like an odd statement since anyone who has worked with Java for more than five minutes will recognise these keywords and their usage, yet appreciating the subtle implications of both is important if you don't want to go insane when writing mixins.

First let's look at some of the possible invocations and accesses in our imaginary class:

Figure 3 - some possible field and method accesses

There's nothing particularly controversial about this scenario, this.level, this.update() and this.food all seem pretty standard. On the other hand the calls to super.health and this.health, and the call to super.onUpdate() from update are designed to highlight an aspect of the JVM which isn't obvious when writing Java code.

Ask yourself the following questions:

• What are the practical implications of qualifying the call to onUpdate with super rather than qualifying it with this?

• What are the practical implications of qualifying the access of health with super rather than qualifying it with this?

After all, both qualifications will work exactly the same in practice, right?

The answer to the two questions is:

• super.onUpdate() will always call the method in Entity even if a subclass overrides it, whereas this.onUpdate() will, in subclasses which override the method, call the overridden method where appropriate.

• There is no difference, the field in Entity will always be accessed from the method takeDamage, even if a subclass "hides" the field by declaring it again.

The underlying reason for this behaviour is that invocations qualified with super, and all field accesses are statically bound at compile time, this means they always reference the member. Conversely, accesses qualified with this are dynamically bound, this means they don't resolve their target until they are actually called, allowing subclasses to override methods and have them called when appropriate.

NOTES

As well as invocations qualified with super, access to private and static methods are always statically bound as well.

In bytecode, statically-bound invocations are represented by the INVOKESPECIAL and INVOKESTATIC opcodes, dynamic calls are represented by the INVOKEVIRTUAL opcode.

Realising the precise nature of these keywords is useful when developing mixins and is the reason for some of the restrictions imposed on mixin classes, more on this later.

2. Through the looking glass

I avoided using the word Interface above to describe the publicly visible members in order to avoid confusion with actual Java interfaces, since interfaces themselves play a key role in how mixins can be employed.

To see how interfaces affect our interaction with a class, let's look at what happens if we create an interface which contains a few of the methods in our example, and access those methods via the interface.

Side note: yes this goes completely off the UML rails but UML is not really useful for representing the concepts here, the bottom of this block diagram is the 'visible surface' of a class from the point of view of any other object, the interface is - in effect - sitting "in front of" the public façade of the class and presenting a subset of it.

Figure 4 - a diagram to annoy UML enthusiasts

There are some useful things to make a note of here:

• Firstly, it's crucial to note that the getHealth and setHealth in the Entity class are actually implementing the interface methods, even though the class Entity has no knowledge of the interface LivingThing, the implication being that there is nothing special about interface methods in a class: as long as the method signatures¹ match those in the interface, then the class method is deemed to implement the interface method. It should be clear from this that interface method calls are dynamically bound.

• We also made no changes to either class except for declaring that it implements LivingThing. In fact, if Java didn't require us to include the implements clause then this program structure would be allowed with no changes to the program at all. What this tells us is that if we can somehow sneakily insert the implements clause onto a target class, then provided the methods in our interface exist we will be able to invoke them on the target class.

¹ A method's signature is it's set of parameters and its return type. For example for the method:

public ThingType getThingAtLocation(double scale, int x, int y, int z, boolean squash) {

the signature would be:

(double,int,int,int,boolean)com.mypackage.ThingType

note that we put the parameters in parentheses and the return type on the end. In practice to save space, a more compact syntax is used and in bytecode the above signature would look like this:

(DIIIZ)Lcom/mypackage/ThingType;

You will need to become familiar with bytecode descriptors if you plan to work with Injectors.

3. Quack, quack

The final piece to the puzzle which we are slowly assembling is a useful Java language feature relating to interfaces, namely the fact that you can cast any object reference to any interface and the compiler will happily compile it.

For example, let's say we invent a new interface for objects which can level up, and call it Leveller, like this:

Figure 5 - what a beautiful day, hey, hey

The following code will happily compile:

public void method() {
    // Make a new EntityPlayer
    EntityPlayer player = new EntityPlayer();

    // This will compile, even though EntityPlayer doesn't
    // actually implement the interface, but it will throw
    // a ClassCastException at runtime
    Leveller lev = (Leveller)player;

    // We will never reach this code, but again it will
    // compile just fine.
    int level = lev.getLevel();
} 

We know from the last section that the method getLevel() in EntityPlayer can happily implement the interface with no changes, but the fact that the implements clause doesn't explicitly declare the interface causes the cast to fail at runtime. If we can somehow apply the implements cause at runtime, then we finally have a viable way of implementing duck typing in Java using interfaces.

"implementing what?"
                     - you, probably

Duck typing is a method of implicit typing used in dynamically typed languages which allows object members to be accessed or invoked based simply on whether they exist or not. It takes its name from the "duck test" expressed as

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.

In other words, if all we care about is that an object has methods quack() and walk() then as far as we are concerned it's a Duck and don't actually care if it's just a very smart Pigeon, as long as it has the methods then it's a Duck to us.

If it's still not obvious what's happening here then I suggest reading the wikipaedia article as it covers the concept in detail which is beyond the scope of this introduction.

So what do we know so far?

• We know that the relationship between classes and interfaces is quite flimsy, and with a little spit and sellotape can be manipulated in a number of ways which are beneficial to us.

• We know that we can leverage dynamic binding in Java to write code which compiles (although it won't run, yet) and that somehow patching the implements clause onto the target object is the key to making this work.

• We know that superclass invocations using the super keyword are statically bound at compile time, and this means we have to give extra thought to exactly what we're referring to when we specify super.

One final thing to consider is what happens when a class doesn't implement an interface. Let's add another method to our Leveller example interface called setLevel():

Figure 6 - adding setLevel()

Adding the second method to the interface adds scope for another - different - runtime error, in this case an AbstractMethodError

public void method() {
    EntityPlayer player = new EntityPlayer();

    // Assume that we can runtime-patch the interface
    // declaration onto the EntityPlayer class, allowing
    // this assignment to succeed
    Leveller lev = (Leveller)player;

    // This statement will throw an AbstractMethodError at
    // runtime because setLevel(I) is not defined in the
    // EntityPlayerClass or any of its superclasses.
    lev.setLevel(10);
} 

Understanding these aspects of Java and the JVM, let's take a look at mixins themselves.

4. Only you mixins can save mankind

So now we know the basic tasks that mixins must achieve in order to allow us to get other objects to quack:

1. Let us apply an interface of our choosing to the target class at runtime
2. Let us insert a method implementation for any methods which are declared in the interface but are not present in the target class

First let's look at how we declare a mixin class with EntityPlayer as its target class:

@Mixin(EntityPlayer.class)
public abstract class MixinEntityPlayer
    extends Entity {
}

Yes it really is that simple. Using the @Mixin annotation defines this class as a mixin and specifies the target class we want to apply it to. Also note that:

• The mixin class is marked with the abstract modifier. Whilst this is not a requirement, it helps when using mixins within an IDE because it means the end user cannot write code which tries to instantiate a mixin class, which would lead to an error at runtime. It also removes the requirement (imposed by the Java compiler) to implement every method within any declared interfaces, which is one of the main goals of mixins.

• The mixin class extends Entity, which is the same superclass as our target class. This is important in order to preserve the semantics of any static bindings which are compiled into our mixin class. More details on this later.

If we were to include this mixin in our runtime right now and run the game, the mixin would be applied and absolutely nothing would be changed, this is because we haven't actually declared anything in our mixin. Let's take a look at how we can achieve objective 1 above, and use our mixin to monkey-patch a new interface onto the target class:

@Mixin(EntityPlayer.class)
public abstract class MixinEntityPlayer
    extends Entity
    implements LivingThing {
}

That's it! Any interfaces declared on the mixin are applied to the target class when the mixin is processed. Let's take a look at the current class hierarchy:

Figure 7 - mixin hierarchy (before application)

While this diagram represents the actual hierarchy of classes we will create, it is actually more useful (and in some more complex cases, vital) to realise that a mixin is not really a class. At runtime, the mixin will be applied to the target class and so it is much more conducive to good thought processes to thin think of mixins as existing within the target class instead.

After the mixin is applied, the new class hierarchy looks like this:

Figure 8 - class hierarchy (after application)

As we can see, the target class now implements the LivingThing interface, which now allows our duck typing to work as we wanted:

public void method() {
    EntityPlayer player = new EntityPlayer();

    // With the mixin applied, this cast succeeds
    LivingThing living = (LivingThing)player;

    // And because the cast suceeded, we can pass our object
    // to other things that require a reference to LivingThing
    // such as the method below
    if (this.isAlive(living)) {
        // hooray
    }
}

public boolean isAlive(LivingThing living) {
    // We can call getHealth() just fine, because the method
    // exists and is accessible via the LivingThing interface
    int health = living.getHealth();
    return health > 0;
} 

Since we've taken care of the first objective and can now successfully apply new interfaces to the target class, let's take a look at the second objective:

• Let us insert a method implementation for any methods which are declared in the interface but are not present in the target class

We'll begin by having our mixin class implement the Leveller interface, which declares a method not currently implemented in our target class or any of its superclasses:

@Mixin(EntityPlayer.class)
public abstract class MixinEntityPlayer
    extends Entity
    implements LivingThing, Leveller {
}

producing the following class hierarchy:

Figure 9 - class hierarchy (after application)

Because our mixin class is abstract, this code will hapily compile, however any runtime call to the setLevel() method will result in an AbstractMethodError as described above. We can fix this by defining the setLevel() method in the mixin itself:

@Mixin(EntityPlayer.class)
public abstract class MixinEntityPlayer
    extends Entity
    implements LivingThing, Leveller {

    @Override
    public void setLevel(int newLevel) {
        // TODO implement this method
    }
}

Figure 10 - adding a method to the mixin

Now when the mixin is applied the new method will also be patched into the target class:

Figure 11 - adding a method to the mixin

Our patched target class now fully implements all of the declared interfaces and we can see how easy it can be to add a new method to our target class. At the moment our new method doesn't actually do anything, we'll see how we can remedy this in the next section.

5. To light a candle is to cast a Shadow

So we now have a way to inject new methods into our target class, but we will fairly quickly encounter a problem with implementing the body of our freshly-injected method: In an ideal world we'd like our new setLevel() implementation to be able to access the level variable in EntityPlayer, but there's a problem... it can't.

Figure 12 - impossible access

We can't access a member of the target class because until the mixin is actually applied, the field doesn't exist! Because the superclass of the mixin is Entity, it doesn't even help if the field is protected: as far as the Java compiler is concerned, the field is nowhere to be seen.

However we know that when the mixin is applied that the field will be there, what we need is some way of telling Java "hey, this field will exist, let me access it". Fortunately mixins provide a mechanism for doing exactly this, via the @Shadow annotation:

@Mixin(EntityPlayer.class)
public abstract class MixinEntityPlayer
    extends Entity
    implements LivingThing, Leveller {
    
    @Shadow
    private int level;
    
    @Override
    public void setLevel(int newLevel) {
        // Refers to the shadow field above, but will refer
        // to the REAL field when the mixin is applied
        this.level = newLevel;
    }
}

The @Shadow annotation creates a "virtual field" in the mixin which mirrors its target class counterpart:

Figure 13 - me and my shadow

It is also possible to apply @Shadow to methods as well, in order to invoke methods which are only defined in the target class, for example say we wanted to call the update() method immediately after setting the level, we can easily shadow the method and then invoke it from our new setLevel() method body:

@Mixin(EntityPlayer.class)
public abstract class MixinEntityPlayer
    extends Entity
    implements LivingThing, Leveller {
    
    @Shadow
    private int level;
    
    @Shadow
    private void update() {}
    
    @Override
    public void setLevel(int newLevel) {
        // Set the level value
        this.level = newLevel;
        
        // Invoke the shadowed method to update the object state
        this.update();
    }
}

We would normally declare the shadow method as abstract simply to avoid having to write a method body, however because it is not possible to declare something as both private and abstract for obvious reasons, we simply declare the the shadow method with an empty body.

Figure 14 - shadow all the things

6. Is it a bird? Is it a plane? No it's superclass!

The final stop on our tour of the basic features of mixins is a brief look at how superclass accesses are handled within mixins. To begin with, we need to first understand why a mixin class is declared with the same superclass as the target class.

First let's take a quick look at our current class hierarchy:

Figure 15 - state of play

Remember from section 1 that invocations qualified with the super keyword are statically bound. In the context of our mixin class, if we call super.onUpdate() as shown in figure 15 then the generated bytecode will reference the onUpdate method in the Entity class specifically.

When the mixin has the same parent class as the target class, this is exactly what we want. However it is actually possible to have a mixin inherit from any class in the target class's hierarchy, up to and including Object.

Let's assume for a minute that EntityPlayer does not inherit directly from Entity, but instead an intermediate class EntityMoving, the mixin class will still extend directly from Entity:

Figure 16 - expanded hierarchy - note: this diagram is deliberately wrong!

Looking at this new hierarchy, it's now obvious why super.onUpdate() will appear to be calling the method in Entity from within the mixin class, but this is where it's important that you ignore what your IDE (and common sense probably) is telling you, and remember that a mixin's point of view is ALWAYS that of the target class!

The problem here is that the intermediate class EntityMoving has overridden the onUpdate moving and the functional contract of the class is such that calling onUpdate in the superclass will actually lead to inconsistent behaviour. When we invoke super.onUpdate() from the mixin, it must have the same semantics as if the same Java statement were invoked from the target class, and this is indeed the case.

• In order to preserve the semantic consistency of Java code you type into a mixin, the mixin transformer updates all static bindings in the mixin class as it is applied. This means that in the above example, the call to super.onUpdate() correctly invokes the method in EntityMoving

• This doesn't affect the semantics of the this keyword. Which for protected and public methods will always use dynamic binding and thus will always invoke the appropriate subclass method.

To get technical, the transformer will process all INVOKESPECIAL opcodes in the mixin and analyse the superclass hierarchy of the target class to find the most specialised version of that method. This process is expensive and is only carried out on "detached" mixins (those mixins whose superclass differs from the target class's superclass). To avoid this processing step, it is recommended that mixin classes have the same superclass as their target wherever possible.

Figure 17 - final hierarchy (mixin applied)

As you can see, after the mixin is applied to the target class, the semantics of the super.onUpdate() call are updated to be consistent with the target class and all is once again well.

7. Wrapping up

While this introduction covers the basics of mixins, there are many more aspects of mixin functionality to explore, especially when working in an environment where the target classes will be obfuscated before being used in a production environment.

More mixin topics (coming soon)

• Introduction to Mixins - The Mixin Environment
• Introduction to Mixins - Overwriting Methods
• Resolving Method Signature Conflicts
• Obfuscation and Mixins
• Advanced Mixin Usage - Soft Implementation
• Advanced Mixin Usage - Using Injection