1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
|
=====================================
S3 Bucket Notifications Compatibility
=====================================
Ceph's `Bucket Notifications`_ and `PubSub Module`_ APIs follow `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below.
.. note::
Compatibility is different depending on which of the above mechanism is used
Supported Destination
---------------------
AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations).
Currently, we support: **HTTP/S**, **Kafka** and **AMQP**. And also support pulling and acking of events stored in Ceph (as an intrenal destination).
We are using the **SNS** ARNs to represent the **HTTP/S**, **Kafka** and **AMQP** destinations.
Notification Configuration XML
------------------------------
Following tags (and the tags inside them) are not supported:
+-----------------------------------+----------------------------------------------+
| Tag | Remaks |
+===================================+==============================================+
| ``<QueueConfiguration>`` | not needed, we treat all destinations as SNS |
+-----------------------------------+----------------------------------------------+
| ``<CloudFunctionConfiguration>`` | not needed, we treat all destinations as SNS |
+-----------------------------------+----------------------------------------------+
REST API Extension
------------------
Ceph's bucket notification API has the following extensions:
- Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb
- In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket
- Getting the information on a specific notification (when more than one exists on a bucket)
- In S3, it is only possible to fetch all notifications on a bucket
- In addition to filtering based on prefix/suffix of object keys we support:
- Filtering based on regular expression matching
- Filtering based on metadata attributes attached to the object
- Filtering based on object tags
- Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP).
- Filtering overlapping is allowed, so that same event could be sent as different notification
Unsupported Fields in the Event Record
--------------------------------------
The records sent for bucket notification follow format described in: `Event Message Structure`_.
However, the following fields may be sent empty, under the different deployment options (Notification/PubSub):
+----------------------------------------+--------------+---------------+------------------------------------------------------------+
| Field | Notification | PubSub | Description |
+========================================+==============+===============+============================================================+
| ``userIdentity.principalId`` | Supported | Not Supported | The identity of the user that triggered the event |
+----------------------------------------+--------------+---------------+------------------------------------------------------------+
| ``requestParameters.sourceIPAddress`` | Not Supported | The IP address of the client that triggered the event |
+----------------------------------------+--------------+---------------+------------------------------------------------------------+
| ``requestParameters.x-amz-request-id`` | Supported | Not Supported | The request id that triggered the event |
+----------------------------------------+--------------+---------------+------------------------------------------------------------+
| ``requestParameters.x-amz-id-2`` | Supported | Not Supported | The IP address of the RGW on which the event was triggered |
+----------------------------------------+--------------+---------------+------------------------------------------------------------+
| ``s3.object.size`` | Supported | Not Supported | The size of the object |
+----------------------------------------+--------------+---------------+------------------------------------------------------------+
Event Types
-----------
+----------------------------------------------+-----------------+-------------------------------------------+
| Event | Notification | PubSub |
+==============================================+=================+===========================================+
| ``s3:ObjectCreated:*`` | Supported |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectCreated:Put`` | Supported | Supported at ``s3:ObjectCreated:*`` level |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectCreated:Post`` | Supported | Not Supported |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectCreated:Copy`` | Supported | Supported at ``s3:ObjectCreated:*`` level |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectCreated:CompleteMultipartUpload`` | Supported | Supported at ``s3:ObjectCreated:*`` level |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectRemoved:*`` | Supported | Supported only the specific events below |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectRemoved:Delete`` | Supported |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectRemoved:DeleteMarkerCreated`` | Supported |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectRestore:Post`` | Not applicable to Ceph |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ObjectRestore:Complete`` | Not applicable to Ceph |
+----------------------------------------------+-----------------+-------------------------------------------+
| ``s3:ReducedRedundancyLostObject`` | Not applicable to Ceph |
+----------------------------------------------+-----------------+-------------------------------------------+
.. note::
The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object
.. note::
In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process.
Topic Configuration
-------------------
In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_.
Note that most of the API is not applicable to Ceph, and only the following actions are implemented:
- ``CreateTopic``
- ``DeleteTopic``
- ``ListTopics``
We also have the following extensions to topic configuration:
- In ``GetTopic`` we allow fetching a specific topic, instead of all user topics
- In ``CreateTopic``
- we allow setting endpoint attributes
- we allow setting opaque data thta will be sent to the endpoint in the notification
.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html
.. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
.. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
.. _`PubSub Module`: ../pubsub-module
.. _`Bucket Notifications`: ../notifications
.. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/master/examples/boto3
|
