diff options
Diffstat (limited to '')
| -rw-r--r-- | doc/radosgw/s3-notification-compatibility.rst | 139 |
1 files changed, 139 insertions, 0 deletions
diff --git a/doc/radosgw/s3-notification-compatibility.rst b/doc/radosgw/s3-notification-compatibility.rst new file mode 100644 index 000000000..09054bed3 --- /dev/null +++ b/doc/radosgw/s3-notification-compatibility.rst @@ -0,0 +1,139 @@ +===================================== +S3 Bucket Notifications Compatibility +===================================== + +Ceph's `Bucket Notifications`_ and `PubSub Module`_ APIs follow `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below. + + +.. note:: + + Compatibility is different depending on which of the above mechanism is used + +Supported Destination +--------------------- + +AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations). +Currently, we support: **HTTP/S**, **Kafka** and **AMQP**. And also support pulling and acking of events stored in Ceph (as an intrenal destination). + +We are using the **SNS** ARNs to represent the **HTTP/S**, **Kafka** and **AMQP** destinations. + +Notification Configuration XML +------------------------------ + +Following tags (and the tags inside them) are not supported: + ++-----------------------------------+----------------------------------------------+ +| Tag | Remaks | ++===================================+==============================================+ +| ``<QueueConfiguration>`` | not needed, we treat all destinations as SNS | ++-----------------------------------+----------------------------------------------+ +| ``<CloudFunctionConfiguration>`` | not needed, we treat all destinations as SNS | ++-----------------------------------+----------------------------------------------+ + +REST API Extension +------------------ + +Ceph's bucket notification API has the following extensions: + +- Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb + + - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket + +- Getting the information on a specific notification (when more than one exists on a bucket) + + - In S3, it is only possible to fetch all notifications on a bucket + +- In addition to filtering based on prefix/suffix of object keys we support: + + - Filtering based on regular expression matching + + - Filtering based on metadata attributes attached to the object + + - Filtering based on object tags + +- Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP). + +- Filtering overlapping is allowed, so that same event could be sent as different notification + + +Unsupported Fields in the Event Record +-------------------------------------- + +The records sent for bucket notification follow format described in: `Event Message Structure`_. +However, the following fields may be sent empty, under the different deployment options (Notification/PubSub): + ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| Field | Notification | PubSub | Description | ++========================================+==============+===============+============================================================+ +| ``userIdentity.principalId`` | Supported | Not Supported | The identity of the user that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.sourceIPAddress`` | Not Supported | The IP address of the client that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.x-amz-request-id`` | Supported | Not Supported | The request id that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.x-amz-id-2`` | Supported | Not Supported | The IP address of the RGW on which the event was triggered | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``s3.object.size`` | Supported | Not Supported | The size of the object | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ + +Event Types +----------- + ++----------------------------------------------+-----------------+-------------------------------------------+ +| Event | Notification | PubSub | ++==============================================+=================+===========================================+ +| ``s3:ObjectCreated:*`` | Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:Put`` | Supported | Supported at ``s3:ObjectCreated:*`` level | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:Post`` | Supported | Not Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:Copy`` | Supported | Supported at ``s3:ObjectCreated:*`` level | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:CompleteMultipartUpload`` | Supported | Supported at ``s3:ObjectCreated:*`` level | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRemoved:*`` | Supported | Supported only the specific events below | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRemoved:Delete`` | Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRemoved:DeleteMarkerCreated`` | Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRestore:Post`` | Not applicable to Ceph | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRestore:Complete`` | Not applicable to Ceph | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ReducedRedundancyLostObject`` | Not applicable to Ceph | ++----------------------------------------------+-----------------+-------------------------------------------+ + +.. note:: + + The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object + +.. note:: + + In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process. + +Topic Configuration +------------------- +In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_. +Note that most of the API is not applicable to Ceph, and only the following actions are implemented: + + - ``CreateTopic`` + - ``DeleteTopic`` + - ``ListTopics`` + +We also have the following extensions to topic configuration: + + - In ``GetTopic`` we allow fetching a specific topic, instead of all user topics + - In ``CreateTopic`` + + - we allow setting endpoint attributes + - we allow setting opaque data thta will be sent to the endpoint in the notification + + +.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html +.. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html +.. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html +.. _`PubSub Module`: ../pubsub-module +.. _`Bucket Notifications`: ../notifications +.. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/master/examples/boto3 |