Cannot create/use repository in S3 with S3 Object Lock enabled

[eriksw]

Output of restic version

restic 0.9.4 compiled with go1.11.4 on linux/amd64

How did you run restic exactly?

RESTIC_REPOSITORY=s3:s3.amazonaws.com/[myBucketName]/[myPrefixInBucket] \
RESTIC_PASSWORD_FILE=/home/me/.resticpassword \
./restic init --json

What backend/server/service did you use to store the repository?

S3, with S3 Object Lock enabled and configured to apply to all objects.

Expected behavior

Restic should have initialized the repository.

Actual behavior

Fatal: create key in repository at s3:s3.amazonaws.com/[myBucketName]/[myPrefixInBucket] failed: client.PutObject: Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters

Steps to reproduce the behavior

Create a bucket with s3 object lock enabled and configured to apply to all objects. Here's some (partial) terraform code:

resource "aws_s3_bucket" "bucket" {
  bucket = "${var.name}"

  versioning {
    enabled = true
  }

  lifecycle_rule {
    enabled = true
    prefix  = ""

    abort_incomplete_multipart_upload_days = "7"

    noncurrent_version_expiration {
      days = "${var.noncurrent_version_expiration_days}"
    }
  }

  object_lock_configuration {
    object_lock_enabled = "Enabled"

    rule {
      default_retention {
        mode = "COMPLIANCE"
        days = "${var.object_lock_days}"
      }
    }
  }
}

variable "name" {
  type    = "string"
  default = "myBucketName"
}

variable "noncurrent_version_expiration_days" {
  type        = "string"
  description = "In reality, this would be large, like 30-90."
  default     = "1"
}

variable "object_lock_days" {
  type        = "string"
  description = "In reality this would be large, like 180. Don't make this big when testing!"
  default     = "3"
}

Do you have any idea what may have caused this?

Restic is attempting to PUT to S3 without computing Content-MD5 and including that in the request.

Do you have an idea how to solve the issue?

Compute Content-MD5 and set the header when making requests to s3.

Did restic help you or made you happy in any way?

😐

[eriksw]

Attempting to use the rclone backend doesn't solve this either:

$ cat ~/.config/rclone/rclone.conf
[s3]
type = s3
env_auth = true
region = us-east-1
storage_class = STANDARD_IA
disable_checksum = false

$ RESTIC_REPOSITORY=rclone:s3:[myBucjetName]/[myPrefixInBucket] \
RESTIC_PASSWORD_FILE=/home/[me]/.resticpassword \
restic init --json
rclone: 2019/03/08 20:01:07 ERROR : keys/[...]: Post request put error: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
rclone: <Error><Code>InvalidRequest</Code><Message>Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters</Message><RequestId>[...]</RequestId><HostId>[...]</HostId></Error>
rclone: 2019/03/08 20:01:07 ERROR : keys/[...]: Post request rcat error: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
rclone: <Error><Code>InvalidRequest</Code><Message>Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters</Message><RequestId>[...]</RequestId><HostId>[...]</HostId></Error>
Fatal: create key in repository at rclone:s3:[myBucketName]/[myPrefixInBucket] failed: server response unexpected: 500 Internal Server Error (500)

[ibib]

Restic use the minio library for the S3 backend. As for now it seems that the Object Lock feature ist not implemented.
minio/minio#7002

[eriksw]

@ibib There is no need for Restic to perform any PutObjectLegalHold/PutObjectRetention api calls, so minio/minio#7002 isn't actually the blocker for fixing this.

What is necessary is for the Content-MD5 to be calculated and set on PutObject requests made to S3. It looks like the minio could be lacking in this area (I only skimmed the docs for the minio s3 client), but I do see bits in the minio code that look like it supports it on the (minio) server side.

[ibib]

Oh. I'm sorry. I got the server and the client repository mixed up.

[eriksw]

Ah, I hadn't even realized there was a different repository for the client. :)

[chasebolt]

would like to see this feature as we can't currently use restic with s3 and object locking enabled.

[instantlinux]

Is this on the roadmap? Ransomware has been all over the news for months. The only reliable mechanism supplied by AWS and other S3-compatible cloud providers for defending against an attacker who has obtained API credentials to the targeted account's data is Object Lock. This is a critical feature.

[rawtaz]

Not in particular. There's a lot of features and other things requiring attention. This is just one of them. There are other ways to solve the problem by using snapshots and what not, S3 isn't the only supported storage that you can use. But you probably know that there are alternatives.

[instantlinux]

My point is that the "other ways to solve the problem" do not work in the face of today's ransomware attackers. A leaked AWS API credential can be used by an attacker to defeat any AWS-based alternative. Object Lock is the gold standard for protection against ransomware: please consider prioritizing this above "a lot of features". As for non-AWS solutions, I don't believe there is better protection against ransomware on any other cloud storage service; you'd have to air-gap your backups using Iron Mountain's pickup/drop-off vault services (or the equivalent using your own vehicle).

[rawtaz]

My point is that the "other ways to solve the problem" do not work in the face of today's ransomware attackers.

Why would no other solution work to prevent ransomware attacks? Are you suggesting that the only viable solution to ransomware attacks are object locks?

Regarding AWS, what if the attacker manages to gain access to Amazon's systems and kills the object lock? To completely avoid that, you'd need to air-cap those systems too, just like anything else.

I get you, you want this feature. I'm not disputing that, just pointing out that there are other ways to reach similar protection (all of them have their pros and cons of course).

BTW, there's a similar discussion (and some related references) in #3195.

[martint17r]

@instantlinux There are at least three ways to get working append only backups with restic, even with S3 - it is just not working out of the box as easily as it could. I am using different methods successfully for over a year now.

If you need help, send me a DM and I will try to help.

[rawtaz]

@eriksw What's your take on this? I take it you don't agree, but what have I said that you think is not correct/factual?

[eriksw]

Can we please get some attention on a PR that would enable fixing this?: #3246

[eriksw]

@eriksw What's your take on this? I take it you don't agree, but what have I said that you think is not correct/factual?

The remark about "Regarding AWS, what if the attacker manages to gain access to Amazon's systems and kills the object lock? To completely avoid that, you'd need to air-cap those systems too, just like anything else." left me confused whether it was a facetious joke or an unfortunate but genuine lack of consideration of the relative risks.

The likelihood of a system that you're backing up with Restic (or anything else) being compromised by something that can use credentials to destroy the backups of that system is, for all practical purposes, infinitely more likely than an authorization-bypassing data-destroying compromise of S3.

[instantlinux]

If you're using AWS for any purpose (including backups), a ransomware attacker will go after your AWS API secret key(s). Perhaps I should describe Object Lock more clearly here: when an S3 bucket is created, it can be set for Object Lock--only at creation time--and that setting cannot be revoked, ever, except by non-payment or closure of the AWS account.

All other append-only or immutable data stored in AWS can be compromised if the IAM policies associated with an API key aren't sufficiently tight or if a root key is compromised. I think it's safe to assume that most restic users aren't experts in IAM policy management, hence my strong recommendation to prioritize Object Lock. And that's why I disagree that there's an "infinitely more likely" successful attack on a system such as an EC2 instance vs a compromise of S3: for most users, the likelihood of compromising both is exactly the same unless they manage their IAM policies and API keys in a sophisticated way.

[eriksw]

Yes, I agree, without Object Lock in play, compromising IAM credentials and using those to destroy the backups is a very likely, very real/practical risk.

I should have been more clear: by "authorization-bypassing data-destroying compromise of S3" I meant one that bypasses the implementation of Object Lock itself, affecting the contents of a bucket with it enabled. (Or abuses the fact that it apparently is possible for locked files to be removed via highly-manual processes that involves convincing AWS' lawyers that it's a legal liability to retain the data.)

[instantlinux]

I meant one that bypasses the implementation of Object Lock itself, affecting the contents of a bucket with it enabled.

Think of it as a giant write-once CDROM: indeed, I think it's safe to say that a remote attack is exceedingly unlikely to succeed against back-end AWS infrastructure, and an attacker targeting AWS is probably not going to target you as an individual user, so your alternative backups would probably survive such an event. Bottom line is, we agree that supporting Object Lock would address the existing known ransomware attack vectors.

[chasebolt]

I agree that Object Lock should be a top priority as Veeam is really the only player in this space that supports it well. We are unable to use restic on the majority of our clients only because it lacks this feature.

PS. I love restic and use it in a hobbyist/small business environment.

[rawtaz]

@eriksw It was neither of those two, and what @instantlinux wrote didn't seem right, hence my questions. However, reading your second paragraph makes it clear to me that we're talking about two different things. No one ever suggested that what you wrote there isn't true. Of course compromised clients shouldn't be able to delete or modify their backups, that's the entire point of this whole discussion, and obviously it's harder to break into AWS than to compromise a client.

[rawtaz]

Bottom line is, we agree that supporting Object Lock would address the existing known ransomware attack vectors.

Sure, it's a great solution to that attack vector. But it's not the only one, which you seemed to imply earlier. There's a world outside of S3 too, really.

[instantlinux]

@rawtaz where is my analysis off? What hacker-proof solution should I and others who land on this issue via an online search consider as an alternative to S3? Right now I know that my own backups are vulnerable, and though I'm well-versed in security for AWS and Kubernetes, there isn't a cloud-based mechanism I can think of to reduce the vulnerability--so I have to rely on rarely-completed manual offline backups.

[martint17r]

Here are three ways:

1. use an rclone server in append only mode
2. use a restic rest-server in append only mode
3. use a script to change the rights to readonly mode for all existing files - restic only writes new files and does never change existing ones under data/

These methods work regardless of backend, though the second one is somewhat limited to locally mounted storage volumes.

Edited for links

[eriksw]

The use case for Object Lock is to protect against destruction by an attack that gains as much access to the storage infrastructure as the authorized user has.

If Object Lock is supported, it is extremely practical to have a backup solution that is robust under that threat model.

It is challenging to achieve anything that is actually comparable to Object Lock—nearly impossible if you assume compromise of (your) administrator credentials.

• Running a server that stores files locally isn't useful if you assume compromise of credentials that can terminate the instance.
• Using lesser ACL protections in cloud storage isn't useful if you assume compromise of credentials that are empowered to change the ACLs to allow deletion of the objects.

Something that would work, but necessitates additional cost/points of failure:

• Point Restic at a same-host local repo or a server that is then in turn synchronized using a capable client to an Object Lock-enabled bucket.

The cost and fragility of that workaround is upsetting when consideration is given to what the roadblock actually is for Restic to directly use an S3 bucket that has Object Lock enabled: Restic needs to assemble cloud storage objects fully before issuing PutObject and include the MD5 when making the request. That's it. Everything else is non-blocking UX improvements that apply to all WORM storage.

---

Aside: The requirement to build objects fully can actually be cheated a bit! Unlike Google Cloud Storage's equivalent, S3's Object Lock doesn't preclude doing multi-part uploads. You can do the assemble-and-compute-MD5 at the granularity of each UploadPart, but there are minimum part size requirements and a limit on the number of parts that make this less flexible than one might imagine.

[rawtaz]

@instantlinux There is nothing that is hacker-proof. I'm not saying you're entirely off, and for your use case perhaps you're entirely right. I'm just pointing out that S3 and Object Locks isn't the only thing in this world that could address a ransomware attack vector (of which there are many different ones, targeted and more drive-by ones), which is what you seem to imply. There's other storages, run by yourself or someone else, there's snapshots, ways to sync, pros and cons with this, etc. If you think it is, okay, fine. I just don't think that's true and I'm just pointing that out. Perhaps we can settle on S3 Object Locking being one of the best/most secure countermeasures out there? I'm totally not disputing that, and hopefully we can get that supported by restic too.

[instantlinux]

You have a contribution from @MichaelEischer as #3246 which implements most of my request. What's the next step in getting that one merged?

[rawtaz]

You already know the next steps in a PR process - it's review, consideration of whether it's ready to be merged, possibly additional changes before merging, etc. Michael worked on that PR last night. Presumably then it's in the works and will be ready when it's ready.

[rawtaz]

Can someone see if this was indeed solved by the great work in #3246? You can download the latest master build from beta.restic.net .