WAL-E's development began Heroku in early 2011, commissioned by the Department of Data with the intent of ensuring that appreciable data loss on Heroku would have to be accompanied by spectacularly obscure mechanisms: it was not acceptable to lose a lot of data in common cases of media failure or operator error. The subsequent replication-related "fork" and "follow" features did not exist. The "Heroku Postgres" product was not yet a concept.
Within one month after the first commit of WAL-E, the very earliest versions were deployed to Heroku production, v0.2.0. At this point, it was somewhat unclear that WAL-E even worked, which was to be determined at leisure following deployment. However, within two weeks, the large April 30th, 2011 AWS crash took place, and this was the first of several large-scale disaster recoveries that took place under similar circumstances at Heroku. This fortuitous timing of events — including WAL-E happening to work on the first try — exhibits a rare circumstance where cutting the corners of engineering practice to deploy slightly earlier prevented enormous heartache for Heroku staff and its customers.
At the time, WAL-E was little more than a shell-script-esque Python
program wrapping around s3cmd. It had terrible error messages,
blunt error handling — sometimes none, with hazardous results, and
lacked parallelism. However, by this time, the critical four
operators and one critical concept of WAL-E had gelled and been shown
to work well:
- wal-push
- wal-fetch
- backup-push
- backup-fetch
- WALE_S3_PREFIX
Also in April of 2011, the v0.5 series was released. Notably, this was the first version possessing a stable version of the storage format in S3, version '005'.
By May of 2011, it became clear that a lot more control over error
handling and exceptions would be necessary to make WAL-E an ergonomic
tool not hated by all its operators. It was also desirable to use
less aggregate virtual memory, as tended to occur when fork()-ing
subprocesses. The was the cause of major re-positioning to use
boto and gevent in WAL-E. Although started in May of 2011,
the bulk of development took place in September and October of 2011,
culminating in v0.5.8. This version is the first that is looks
recognizably like WAL-E would for quite some time to come.
In January of 2013, WAL-E inducted its first committers beyond the principal author at Heroku, and its project development gained a distinct identity from Heroku.