Why is MessagePack slower than JSON?

[john-boxcar]

I wrote this crude benchmark to compare the performance of JSON and MessagePack. Consistently, JSON is faster during serialization, faster on deserialization, and MessagePack produces smaller payloads. Based on what I've read, I was expecting MessagePack to be faster during both serialization and deserialization in addition to producing smaller payloads. Are my expecations unrealistic or is there room for improvement here? On my machine, I'm getting this output:

==================================== OUTPUT ====================================
JSON serialized 1,000,000 times in 893ms
JSON size 360
MessagePak serialized 1,000,000 times in 1497ms
MessagePak size 304
JSON deserialized 1,000,000 times in 1496ms
MessagePak deserialized 1,000,000 times in 1777ms

=================================== SUMMARY ====================================
JSON serialization is %40.35 faster than MessagePack serialization
JSON deserialization is %15.81 faster than MessagePack deserialization
MessagePack is %15.56 smaller than JSON

Here are my dependency versions:

+- org.msgpack:jackson-dataformat-msgpack:jar:0.9.8:compile
|  |  +- com.google.http-client:google-http-client-jackson2:jar:1.44.1:compile
+- com.fasterxml.jackson.core:jackson-databind:jar:2.17.1:compile
|  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.17.1:compile
|  \- com.fasterxml.jackson.core:jackson-core:jar:2.17.1:compile
+- com.fasterxml.jackson.datatype:jackson-datatype-guava:jar:2.17.1:compile
+- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.17.1:compile
|  +- org.msgpack:jackson-dataformat-msgpack:jar:0.9.8:compile
|  |  |  +- com.google.http-client:google-http-client-jackson2:jar:1.44.1:compile
|  +- com.fasterxml.jackson.core:jackson-databind:jar:2.17.1:compile
|  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.17.1:compile
|  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.17.1:compile
|  +- com.fasterxml.jackson.datatype:jackson-datatype-guava:jar:2.17.1:compile
|  +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.17.1:compile

Here is the code:

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.msgpack.jackson.dataformat.MessagePackMapper;

import java.io.IOException;
import java.text.NumberFormat;
import java.util.List;
import java.util.Locale;

public class MessagePakMain {
    private static final String SAMPLE_JSON = """
        {
            "glossary": {
                "title": "example glossary",
                "GlossDiv": {
                    "title": "S",
                    "GlossList": {
                        "GlossEntry": {
                            "ID": "SGML",
                            "SortAs": "SGML",
                            "GlossTerm": "Standard Generalized Markup Language",
                            "Acronym": "SGML",
                            "Abbrev": "ISO 8879:1986",
                            "GlossDef": {
                                "para": "A meta-markup language, used to create markup languages such as DocBook.",
                                "GlossSeeAlso": ["GML", "XML"]
                            },
                            "GlossSee": "markup"
                        }
                    }
                }
            }
        }
    """;

    public static void main(String[] args) throws IOException {
        ObjectMapper jsonMapper = new ObjectMapper();
        ObjectMapper messagePakMapper = new MessagePackMapper();
        int iterations = 1_000_000;

        Item item = new ObjectMapper().readValue(SAMPLE_JSON, Item.class);

        System.out.println("==================================== OUTPUT ====================================");

        long startTime = System.currentTimeMillis();
        byte[] jsonOutput = null;
        byte[] messagePakOutput = null;

        for (int i =0; i < iterations; i++) {
            byte[] bytes = jsonMapper.writeValueAsBytes(item);
            if (jsonOutput == null) {
                jsonOutput = bytes;
            }
        }

        long endTime = System.currentTimeMillis();
        long jSerializeElapsedTime = endTime - startTime;
        System.out.println("JSON serialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + jSerializeElapsedTime + "ms");
        System.out.println("JSON size " + jsonOutput.length);

        startTime = System.currentTimeMillis();

        for (int i =0; i < iterations; i++) {
            byte[] bytes = messagePakMapper.writeValueAsBytes(item);
            if (messagePakOutput == null) {
                messagePakOutput = bytes;
            }
        }

        endTime = System.currentTimeMillis();
        long mSerializeElapsedTime = endTime - startTime;
        System.out.println("MessagePak serialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + mSerializeElapsedTime + "ms");
        System.out.println("MessagePak size " + messagePakOutput.length);

        // MessagePack is supposed to be faster than JSON, from what I've read. However, the results of this test show
        // that while the serialized data is approximately 30% smaller, the serialization time is also 30% higher. This
        // is very disappointing


        String newJson = new String(jsonOutput);
        startTime = System.currentTimeMillis();

        for (int i =0; i < iterations; i++) {
            jsonMapper.readValue(newJson, Item.class);
        }

        endTime = System.currentTimeMillis();
        long jDeserializeElapsedTime = endTime - startTime;
        System.out.println("JSON deserialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + jDeserializeElapsedTime + "ms");

        startTime = System.currentTimeMillis();

        for (int i =0; i < iterations; i++) {
            messagePakMapper.readValue(messagePakOutput, Item.class);
        }

        endTime = System.currentTimeMillis();
        long mDeserializeElapsedTime = endTime - startTime;
        System.out.println("MessagePak deserialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + mDeserializeElapsedTime + "ms");

        System.out.println("\n=================================== SUMMARY ====================================");

        if (jSerializeElapsedTime > mSerializeElapsedTime) {
            System.out.printf("MessagePack serialization is %%%.02f faster than JSON serialization\n", ((1.0 - (float) mSerializeElapsedTime / (float) jSerializeElapsedTime))*100.0);
        } else {
            System.out.printf("JSON serialization is %%%.02f faster than MessagePack serialization\n", ((1.0 - (float) jSerializeElapsedTime / (float) mSerializeElapsedTime))*100.0);
        }

        if (jDeserializeElapsedTime > mDeserializeElapsedTime) {
            System.out.printf("MessagePack deserialization is %%%.02f faster than JSON deserialization\n", ((1.0 - (float) mDeserializeElapsedTime / (float) jDeserializeElapsedTime))*100.0);
        } else {
            System.out.printf("JSON deserialization is %%%.02f faster than MessagePack deserialization\n", ((1.0 - (float) jDeserializeElapsedTime / (float) mDeserializeElapsedTime))*100.0);
        }

        if (jsonOutput.length > messagePakOutput.length) {
            System.out.printf("MessagePack is %%%.02f smaller than JSON\n", ((1.0 - (float) messagePakOutput.length / (float)jsonOutput.length))*100.0);
        } else {
            System.out.printf("JSON is %%%.02f smaller than MessagePack\n", ((1.0 - (float) jsonOutput.length / (float)messagePakOutput.length))*100.0);
        }
    }

    private static class Item {
        private Glossary glossary;

        public Glossary getGlossary() {
            return glossary;
        }

        public void setGlossary(Glossary glossary) {
            this.glossary = glossary;
        }
    }
    
    private static class GlossaryDef {
        private String para;
        
        @JsonProperty("GlossSeeAlso")
        private List<String> seeAlso;

        public String getPara() {
            return para;
        }

        public void setPara(String para) {
            this.para = para;
        }

        public List<String> getSeeAlso() {
            return seeAlso;
        }

        public void setSeeAlso(List<String> seeAlso) {
            this.seeAlso = seeAlso;
        }
    }
    
    private static class GlossaryEntry {
        @JsonProperty("ID")
        private String id;
        
        @JsonProperty("SortAs")
        private String sortAs;
        
        @JsonProperty("GlossTerm")
        private String glossTerm;
        
        @JsonProperty("Acronym")
        private String acronym;
        
        @JsonProperty("Abbrev")
        private String abbrev;
        
        @JsonProperty("GlossDef")
        private GlossaryDef glossaryDef;
        
        @JsonProperty("GlossSee")
        private String glossSee;

        public String getId() {
            return id;
        }

        public void setId(String id) {
            this.id = id;
        }

        public String getSortAs() {
            return sortAs;
        }

        public void setSortAs(String sortAs) {
            this.sortAs = sortAs;
        }

        public String getGlossTerm() {
            return glossTerm;
        }

        public void setGlossTerm(String glossTerm) {
            this.glossTerm = glossTerm;
        }

        public String getAcronym() {
            return acronym;
        }

        public void setAcronym(String acronym) {
            this.acronym = acronym;
        }

        public String getAbbrev() {
            return abbrev;
        }

        public void setAbbrev(String abbrev) {
            this.abbrev = abbrev;
        }

        public GlossaryDef getGlossaryDef() {
            return glossaryDef;
        }

        public void setGlossaryDef(GlossaryDef glossaryDef) {
            this.glossaryDef = glossaryDef;
        }

        public String getGlossSee() {
            return glossSee;
        }

        public void setGlossSee(String glossSee) {
            this.glossSee = glossSee;
        }
    }
    
    private static class GlossaryList {
        @JsonProperty("GlossEntry")
        private GlossaryEntry glossaryEntry;

        public GlossaryEntry getGlossaryEntry() {
            return glossaryEntry;
        }

        public void setGlossaryEntry(GlossaryEntry glossaryEntry) {
            this.glossaryEntry = glossaryEntry;
        }
    }
    
    private static class GlossaryDiv {
        private String title;
        
        @JsonProperty("GlossList")
        private GlossaryList glossaryList;

        public String getTitle() {
            return title;
        }

        public void setTitle(String title) {
            this.title = title;
        }

        public GlossaryList getGlossaryList() {
            return glossaryList;
        }

        public void setGlossaryList(GlossaryList glossaryList) {
            this.glossaryList = glossaryList;
        }
    }
    
    private static class Glossary {
        private String title;
        
        @JsonProperty("GlossDiv")
        private GlossaryDiv glossaryDiv;

        public String getTitle() {
            return title;
        }

        public void setTitle(String title) {
            this.title = title;
        }

        public GlossaryDiv getGlossaryDiv() {
            return glossaryDiv;
        }

        public void setGlossaryDiv(GlossaryDiv glossaryDiv) {
            this.glossaryDiv = glossaryDiv;
        }
    }
}

[pjfanning]

Jackson is at heart a JSON library that has bindings for other formats. In practice, most effort of the small dev community goes into the JSON use cases. If you would like to volunteer to provide patches to improve the Jackson messagepack support, they will be appreciated.

NB msgpack-jackson is not a Jackson maintained library. It is maintained by the https://github.com/msgpack team so maybe you should redirect this discussion to them.

If you raw performance is what you need, would you not consider using a low level library like msgpack-core? A specialised lib will usually beat the performance of a generalised lib like Jackson.

[cowtowncoder]

@john-boxcar There is room for improvement there: generally MessagePack performance ought to be similar to CBOR codec. This is similar to how Ion support has quite a bit of overhead; whereas Smile, CBOR and Avro (deser) codecs have much less overhead (are faster than JSON).

Please note that jackson-dataformat-msgpack is not directly maintained by core Jackson team but is a third-party extension. So you would want to discuss this within context of that repo, instead of here.
(I am not 100% sure who the author is; could be they are active with core Jackson components too but not necessarily).

[john-boxcar]

My apologies to you both. I was under the mistaken belief that this was the correct repository to post in. I will re-post this in the appropriate repository. Thank you for your time.

[john-boxcar]

Re-posted here: msgpack/msgpack-java#841

[cowtowncoder]

FWTW, this project:

https://github.com/FasterXML/jackson-benchmarks

reproduces your findings, f.ex:

https://github.com/FasterXML/jackson-benchmarks/blob/2.18/results-pojo-2.17-home.txt

has read perf:

c.f.j.p.json.JsonStdReadVanilla.readPojoMediaItem        thrpt   45  542318.950 ± 7441.830  ops/s
....
c.f.j.p.msgpack.MsgpackStdReadVanilla.readPojoMediaItem  thrpt   45  334293.956 ± 9258.433  ops/s

so JSON deserialization is about 60% faster; and serialization:

c.f.j.p.json.JsonStdWriteVanilla.writePojoMediaItem        thrpt   45   910080.427 ± 3971.428  ops/s
...
c.f.j.p.msgpack.MsgpackStdWriteVanilla.writePojoMediaItem  thrpt   45   359612.929 ± 2466.673  ops/s

almost 150% faster.

These are small documents (300 bytes JSON if I remember), difference for bigger content probably less pronunced.

But other binary formats (Smile, CBOR in particular, and for reading, Avro) are faster than JSON.