Why did we create YapDatabase

Databases: Can't live with 'em. Can't live without 'em.

Core Data seems like the natural apple approved thing to use. And I've spent years working with it. I've spent more time than most profiling Core Data performance, and getting acquainted with NSManagedObject. However, it's always a nightmare to use. Here's why:

• Core data requires you use "core data objects". That is, all those classes you've created in your app that you want to stick in the database... they need to be rewritten to be proper NSManagedObject's. And then you have to go into the core data model and set it up properly, with a database column for each and every ivar, properly typed. For a fast moving startup, this is incredibly maddening. Every other week we're adding features and functionality. And this means adding ivars. Which means updating the model. And if it's different than a previously shipped model, then we have to properly version the model. This is because core data sits atop sqlite, and sets up a database table for your class, with a column for each ivar. So if that changes, core data has to update the table by modifying columns. (Except sqlite only supports adding columns. To remove a column one actually has to create a new table, and copy over the old data.) This is an annoying overhead to incur, especially since it hits the user immediately after updating to the "latest and greatest" version.

• In addition to making changes to our existing classes, we often created new classes. (As companies tend to do when they're adding features and functionality). And this meant we had to jump through the same hoops to set everything up. After awhile we started to just "get used to it". Although the 30 some versions of the core data model we had racked up was becoming alarming... But this wasn't the worst part.

• Core Data is NOT thread-safe. Well it can be, but first you need to become a core data expert. I petitioned our development team to read the book 3 times, and then meditate on core data at least 45 minutes per-day, but they just weren't interested for some reason. You see, when you fetch an object (a NSManagedObject) from core data, it's tied to the thread-unsafe NSManagedObjectContext you used to fetch it. And as such, it is tied to the thread/queue you're using the NSManagedObjectContext on/in. So we constantly ran into these situations where an object would get passed to another thread, either on purpose or on accident, and then BAM. Crash!

• The problem is, the entire team is forced to deal with Core Data. With other parts of the application (whether it be some networking protocol, or some custom UI component, or whatever) we could assign it to a single team member. He/she would then deep dive and solve the problem, and present the rest of the team with essentially a black box. A simple API we could use without having to understand all the details. But Core Data escapes the black box. It becomes pervasive throughout the application, and everyone is forced to know it.

We longed to return to simpler times, when our objects were just objects. And so we began wondering what our other options were. One day one of the server developers overheard our musings. He was intrigued, considering most of our data was downloaded directly from the database(s) he managed. And he told us that they stored most of the data we used in mongodb, and then he recommended we switch to a simple key/value database (since mongo is key/value). I was skeptical at first. Surely we're too advanced for such a primitive database. We have all kinds of advanced ... er ... needs and stuff. But then when I got over myself and seriously thought about it, I realized that was exactly what we needed 95% of the time. And the other 5% wasn't even that difficult to solve with the simplest of key/value stores.

So we boldly moved forward with the key/value store idea. But what to use? There were a multitude of options available. Some of them were expensive. Others had questionable license agreements. Eventually we settled on the idea of using sqlite (one way or another). It's free, it's fast, and it's builtin to iOS. It's used under the hood by Core Data. And it's the most widely deployed and used database on the planet.

So we went looking for a simple key/value store built atop sqlite. We knew about LOLdb, which is "laughably" simple. It worked well for us. But it lacked a few useful features. And so we started thinking up what we wanted from our key/value database:

• Concurrency. We wanted to be able to read objects on the main thread while simultaneously writing data on a background thread. We never wanted to block the main thread.

• Built-in Cache. Sure there's caching in sqlite. But that's a cache of raw bytes. We still have to deserialize those bytes into an object. So our database layer should be able to cache our objects.

• Metadata. We often want to store more than just the object. A typical use case for us is the download timestamp. This tells us when the object should be refreshed or considered expired. And it just doesn't seem right stuffing this extra data into the object.

• Collections. Sometimes, but not always, we wanted more than just a single key. We wanted a collection & key. Having this feature simplifies many situations, and allows us to use a single database for a group of related, but separate, data.

And this is how we ended up with YapDatabase. It contains all the features above. And then we added an advanced plugin system to take it to the next level.