Swift/ideas/object-lock

Problem

Some Object Storage providers support objects being locked against deletion or certain kinds of modification. The locks are intended to allow a user to set a retention period that cannot be overridden regardless of the level of permission used against the Object Storage API, so that even if a user's administrative credentials are leaked or otherwise broken, data written to the Object Store cannot be tampered with. Examples of use include immutable backups to protect against ransomware, and compliance with legal requirements for retention.

Proposed Solution

Object Locking has a defined set of behaviors in the S3 API, which we should replicate for the S3 compatible API in Swift. There would also need to be an equivalent set of attributes when using the Swift API, and should work in the same manner.

The S3 behaviour is documented here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html

This proposal suggests we use protected metadata on objects and containers to represent the same states and interpretation as the S3 API expects, which in turn are mapped in the S3 compatiblity layer to its API.

The Swift API would look something like:

• GET/POST/PUT /v1/{account}/{container} will need new headers for container-level configuration, matching S3
  • X-Container-Lock-Mode, X-Container-Lock-Days, X-Container-Lock-Years, X-Container-Lock-Enabled
• POST/PUT /v1/{account}/{container}/{object} will need additional headers for locks, matching S3
  • X-Object-Lock-Mode, X-Object-Lock-Days, X-Object-Lock-Years, X-Object-Lock-Enabled
  • X-Object-LegalHold-Enabled (for legal hold, which is in addition to retention)
• GET /v1/{account}/{container}/{object} will need additional headers for reading current state
  • X-Object-Lock-Mode, X-Object-Lock-LockedUntil, X-Object-Lock-Enabled
  • X-Object-LegalHold-Enabled (for legal hold, which is in addition)

Note that object reads return X-Object-Lock-LockedUntil rather than the retention period, as the retention period is relative to point the object was locked. It could return the object lock start date and retention as well, but it feels like LockedUntil is generally more useful.

S3 expects certain additional roles to exist for modes and legal holds. These should be configurable to specific Keystone roles, perhaps by policy.

Lock setting behavior

Swift does not write metadata updates synchronously to all copies of an object, but is instead eventually consistent. This means object locks set on an existing object will not be immediately applied to all copies, and consequently some DELETE actions would be permitted by some members of the service. Since this is core to how Swift metadata works, this proposal is not going to attempt to change that.

In the case a lock policy is set on the container, the locks should always be written to all copies on create, so no copy should exist that has inconsistent metadata for any duration.

In summary, lock-after-write we expect to be only eventually consistent, while locked-on-create (from container settings) should be consistent without any delay.