diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-21 11:54:28 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-21 11:54:28 +0000 |
| commit | e6918187568dbd01842d8d1d2c808ce16a894239 (patch) | |
| tree | 64f88b554b444a49f656b6c656111a145cbbaa28 /doc/radosgw/s3 | |
| parent | Initial commit. (diff) | |
| download | ceph-e6918187568dbd01842d8d1d2c808ce16a894239.tar.xz ceph-e6918187568dbd01842d8d1d2c808ce16a894239.zip | |
Adding upstream version 18.2.2.upstream/18.2.2
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
| -rw-r--r-- | doc/radosgw/s3-notification-compatibility.rst | 149 | ||||
| -rw-r--r-- | doc/radosgw/s3.rst | 98 | ||||
| -rw-r--r-- | doc/radosgw/s3/authentication.rst | 235 | ||||
| -rw-r--r-- | doc/radosgw/s3/bucketops.rst | 706 | ||||
| -rw-r--r-- | doc/radosgw/s3/commons.rst | 113 | ||||
| -rw-r--r-- | doc/radosgw/s3/cpp.rst | 337 | ||||
| -rw-r--r-- | doc/radosgw/s3/csharp.rst | 199 | ||||
| -rw-r--r-- | doc/radosgw/s3/java.rst | 212 | ||||
| -rw-r--r-- | doc/radosgw/s3/objectops.rst | 558 | ||||
| -rw-r--r-- | doc/radosgw/s3/perl.rst | 192 | ||||
| -rw-r--r-- | doc/radosgw/s3/php.rst | 214 | ||||
| -rw-r--r-- | doc/radosgw/s3/python.rst | 197 | ||||
| -rw-r--r-- | doc/radosgw/s3/ruby.rst | 364 | ||||
| -rw-r--r-- | doc/radosgw/s3/serviceops.rst | 69 | ||||
| -rw-r--r-- | doc/radosgw/s3select.rst | 796 |
15 files changed, 4439 insertions, 0 deletions
diff --git a/doc/radosgw/s3-notification-compatibility.rst b/doc/radosgw/s3-notification-compatibility.rst new file mode 100644 index 000000000..1627ed0c4 --- /dev/null +++ b/doc/radosgw/s3-notification-compatibility.rst @@ -0,0 +1,149 @@ +===================================== +S3 Bucket Notifications Compatibility +===================================== + +Ceph's `Bucket Notifications`_ API follows `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below. + + +.. note:: + + Compatibility is different depending on which of the above mechanism is used + +Supported Destination +--------------------- + +AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations). +Currently, we support: **HTTP/S**, **Kafka** and **AMQP**. + +We are using the **SNS** ARNs to represent the **HTTP/S**, **Kafka** and **AMQP** destinations. + +Notification Configuration XML +------------------------------ + +Following tags (and the tags inside them) are not supported: + ++-----------------------------------+----------------------------------------------+ +| Tag | Remaks | ++===================================+==============================================+ +| ``<QueueConfiguration>`` | not needed, we treat all destinations as SNS | ++-----------------------------------+----------------------------------------------+ +| ``<CloudFunctionConfiguration>`` | not needed, we treat all destinations as SNS | ++-----------------------------------+----------------------------------------------+ + +REST API Extension +------------------ + +Ceph's bucket notification API has the following extensions: + +- Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb + + - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket + +- Getting the information on a specific notification (when more than one exists on a bucket) + + - In S3, it is only possible to fetch all notifications on a bucket + +- In addition to filtering based on prefix/suffix of object keys we support: + + - Filtering based on regular expression matching + + - Filtering based on metadata attributes attached to the object + + - Filtering based on object tags + +- Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP). + +- Filtering overlapping is allowed, so that same event could be sent as different notification + + +Unsupported Fields in the Event Record +-------------------------------------- + +The records sent for bucket notification follows the format described in: `Event Message Structure`_. +However, the ``requestParameters.sourceIPAddress`` field will be sent empty. + + +Event Types +----------- + ++--------------------------------------------------------+-----------------------------------------+ +| Event | Note | ++========================================================+=========================================+ +| ``s3:ObjectCreated:*`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectCreated:Put`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectCreated:Post`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectCreated:Copy`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectCreated:CompleteMultipartUpload`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectRemoved:*`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectRemoved:Delete`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectRemoved:DeleteMarkerCreated`` | Supported | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectLifecycle:Expiration:Current`` | Ceph extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectLifecycle:Expiration:NonCurrent`` | Ceph extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectLifecycle:Expiration:DeleteMarker`` | Ceph extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectLifecycle:Expiration:AbortMultipartUpload`` | Defined, Ceph extension (not generated) | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectLifecycle:Transition:Current`` | Ceph extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectLifecycle:Transition:NonCurrent`` | Ceph extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectSynced:*`` | Ceph extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectSynced:Create`` | Ceph Extension | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectSynced:Delete`` | Defined, Ceph extension (not generated) | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectSynced:DeletionMarkerCreated`` | Defined, Ceph extension (not generated) | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectRestore:Post`` | Not applicable | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ObjectRestore:Complete`` | Not applicable | ++--------------------------------------------------------+-----------------------------------------+ +| ``s3:ReducedRedundancyLostObject`` | Not applicable | ++--------------------------------------------------------+-----------------------------------------+ + +.. note:: + + The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object + +.. note:: + + In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process. + +.. note:: + + The ``s3:ObjectSynced:Create`` event is sent when an object successfully syncs to a zone. It must be explicitly set for each zone. + +Topic Configuration +------------------- +In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_. +Note that most of the API is not applicable to Ceph, and only the following actions are implemented: + + - ``CreateTopic`` + - ``DeleteTopic`` + - ``ListTopics`` + +We also have the following extensions to topic configuration: + + - In ``GetTopic`` we allow fetching a specific topic, instead of all user topics + - In ``CreateTopic`` + + - we allow setting endpoint attributes + - we allow setting opaque data that will be sent to the endpoint in the notification + + +.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html +.. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html +.. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html +.. _`Bucket Notifications`: ../notifications +.. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/main/examples/rgw/boto3 diff --git a/doc/radosgw/s3.rst b/doc/radosgw/s3.rst new file mode 100644 index 000000000..cb5eb3adb --- /dev/null +++ b/doc/radosgw/s3.rst @@ -0,0 +1,98 @@ +.. _radosgw s3: + +============================ + Ceph Object Gateway S3 API +============================ + +Ceph supports a RESTful API that is compatible with the basic data access model of the `Amazon S3 API`_. + +API +--- + +.. toctree:: + :maxdepth: 1 + + Common <s3/commons> + Authentication <s3/authentication> + Service Ops <s3/serviceops> + Bucket Ops <s3/bucketops> + Object Ops <s3/objectops> + C++ <s3/cpp> + C# <s3/csharp> + Java <s3/java> + Perl <s3/perl> + PHP <s3/php> + Python <s3/python> + Ruby <s3/ruby> + + +Features Support +---------------- + +The following table describes the support status for current Amazon S3 functional features: + ++---------------------------------+-----------------+----------------------------------------+ +| Feature | Status | Remarks | ++=================================+=================+========================================+ +| **List Buckets** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Delete Bucket** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Create Bucket** | Supported | Different set of canned ACLs | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Lifecycle** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Replication** | Partial | Permitted only across zones | ++---------------------------------+-----------------+----------------------------------------+ +| **Policy (Buckets, Objects)** | Supported | ACLs & bucket policies are supported | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Website** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket ACLs (Get, Put)** | Supported | Different set of canned ACLs | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Location** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Notification** | Supported | See `S3 Notification Compatibility`_ | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Object Versions** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Get Bucket Info (HEAD)** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Request Payment** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Put Object** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Delete Object** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Get Object** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Object ACLs (Get, Put)** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Get Object Info (HEAD)** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **POST Object** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Copy Object** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Multipart Uploads** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Object Tagging** | Supported | See :ref:`tag_policy` for Policy verbs | ++---------------------------------+-----------------+----------------------------------------+ +| **Bucket Tagging** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ +| **Storage Class** | Supported | See :ref:`storage_classes` | ++---------------------------------+-----------------+----------------------------------------+ + +Unsupported Header Fields +------------------------- + +The following common request header fields are not supported: + ++----------------------------+------------+ +| Name | Type | ++============================+============+ +| **x-amz-id-2** | Response | ++----------------------------+------------+ + +.. _Amazon S3 API: http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html +.. _S3 Notification Compatibility: ../s3-notification-compatibility diff --git a/doc/radosgw/s3/authentication.rst b/doc/radosgw/s3/authentication.rst new file mode 100644 index 000000000..64747cde2 --- /dev/null +++ b/doc/radosgw/s3/authentication.rst @@ -0,0 +1,235 @@ +========================= + Authentication and ACLs +========================= + +Requests to the RADOS Gateway (RGW) can be either authenticated or +unauthenticated. RGW assumes unauthenticated requests are sent by an anonymous +user. RGW supports canned ACLs. + +Authentication +-------------- +Authenticating a request requires including an access key and a Hash-based +Message Authentication Code (HMAC) in the request before it is sent to the +RGW server. RGW uses an S3-compatible authentication approach. + +:: + + HTTP/1.1 + PUT /buckets/bucket/object.mpeg + Host: cname.domain.com + Date: Mon, 2 Jan 2012 00:01:01 +0000 + Content-Encoding: mpeg + Content-Length: 9999999 + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +In the foregoing example, replace ``{access-key}`` with the value for your access +key ID followed by a colon (``:``). Replace ``{hash-of-header-and-secret}`` with +a hash of the header string and the secret corresponding to the access key ID. + +To generate the hash of the header string and secret, you must: + +#. Get the value of the header string. +#. Normalize the request header string into canonical form. +#. Generate an HMAC using a SHA-1 hashing algorithm. + See `RFC 2104`_ and `HMAC`_ for details. +#. Encode the ``hmac`` result as base-64. + +To normalize the header into canonical form: + +#. Get all fields beginning with ``x-amz-``. +#. Ensure that the fields are all lowercase. +#. Sort the fields lexicographically. +#. Combine multiple instances of the same field name into a + single field and separate the field values with a comma. +#. Replace white space and line breaks in field values with a single space. +#. Remove white space before and after colons. +#. Append a new line after each field. +#. Merge the fields back into the header. + +Replace the ``{hash-of-header-and-secret}`` with the base-64 encoded HMAC string. + +Authentication against OpenStack Keystone +----------------------------------------- + +In a radosgw instance that is configured with authentication against +OpenStack Keystone, it is possible to use Keystone as an authoritative +source for S3 API authentication. To do so, you must set: + +* the ``rgw keystone`` configuration options explained in :doc:`../keystone`, +* ``rgw s3 auth use keystone = true``. + +In addition, a user wishing to use the S3 API must obtain an AWS-style +access key and secret key. They can do so with the ``openstack ec2 +credentials create`` command:: + + $ openstack --os-interface public ec2 credentials create + +------------+---------------------------------------------------------------------------------------------------------------------------------------------+ + | Field | Value | + +------------+---------------------------------------------------------------------------------------------------------------------------------------------+ + | access | c921676aaabbccdeadbeef7e8b0eeb2c | + | links | {u'self': u'https://auth.example.com:5000/v3/users/7ecbebaffeabbddeadbeefa23267ccbb24/credentials/OS-EC2/c921676aaabbccdeadbeef7e8b0eeb2c'} | + | project_id | 5ed51981aab4679851adeadbeef6ebf7 | + | secret | ******************************** | + | trust_id | None | + | user_id | 7ecbebaffeabbddeadbeefa23267cc24 | + +------------+---------------------------------------------------------------------------------------------------------------------------------------------+ + +The thus-generated access and secret key can then be used for S3 API +access to radosgw. + +.. note:: Consider that most production radosgw deployments + authenticating against OpenStack Keystone are also set up + for :doc:`../multitenancy`, for which special + considerations apply with respect to S3 signed URLs and + public read ACLs. + +Access Control Lists (ACLs) +--------------------------- + +RGW supports S3-compatible ACL functionality. An ACL is a list of access grants +that specify which operations a user can perform on a bucket or on an object. +Each grant has a different meaning when applied to a bucket versus applied to +an object: + ++------------------+--------------------------------------------------------+----------------------------------------------+ +| Permission | Bucket | Object | ++==================+========================================================+==============================================+ +| ``READ`` | Grantee can list the objects in the bucket. | Grantee can read the object. | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``WRITE`` | Grantee can write or delete objects in the bucket. | N/A | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``READ_ACP`` | Grantee can read bucket ACL. | Grantee can read the object ACL. | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``WRITE_ACP`` | Grantee can write bucket ACL. | Grantee can write to the object ACL. | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``FULL_CONTROL`` | Grantee has full permissions for object in the bucket. | Grantee can read or write to the object ACL. | ++------------------+--------------------------------------------------------+----------------------------------------------+ + +Internally, S3 operations are mapped to ACL permissions thus: + ++---------------------------------------+---------------+ +| Operation | Permission | ++=======================================+===============+ +| ``s3:GetObject`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectTorrent`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersion`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersionTorrent`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectTagging`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersionTagging`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListAllMyBuckets`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListBucket`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListBucketMultipartUploads`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListBucketVersions`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListMultipartUploadParts`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:AbortMultipartUpload`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:CreateBucket`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteBucket`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteObject`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:s3DeleteObjectVersion`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:PutObject`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectVersionTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteObjectTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteObjectVersionTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:RestoreObject`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:GetAccelerateConfiguration`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketAcl`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketCORS`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketLocation`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketLogging`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketNotification`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketPolicy`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketRequestPayment`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketTagging`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketVersioning`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketWebsite`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetLifecycleConfiguration`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectAcl`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersionAcl`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetReplicationConfiguration`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketEncryption`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:DeleteBucketPolicy`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:DeleteBucketWebsite`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:DeleteReplicationConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutAccelerateConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketAcl`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketCORS`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketLogging`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketNotification`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketPolicy`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketRequestPayment`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketTagging`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutPutBucketVersioning`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketWebsite`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutLifecycleConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectAcl`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectVersionAcl`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutReplicationConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketEncryption`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ + +Some mappings, (e.g. ``s3:CreateBucket`` to ``WRITE``) are not +applicable to S3 operation, but are required to allow Swift and S3 to +access the same resources when things like Swift user ACLs are in +play. This is one of the many reasons that you should use S3 bucket +policies rather than S3 ACLs when possible. + + +.. _RFC 2104: http://www.ietf.org/rfc/rfc2104.txt +.. _HMAC: https://en.wikipedia.org/wiki/HMAC diff --git a/doc/radosgw/s3/bucketops.rst b/doc/radosgw/s3/bucketops.rst new file mode 100644 index 000000000..17da3a935 --- /dev/null +++ b/doc/radosgw/s3/bucketops.rst @@ -0,0 +1,706 @@ +=================== + Bucket Operations +=================== + +PUT Bucket +---------- +Creates a new bucket. To create a bucket, you must have a user ID and a valid AWS Access Key ID to authenticate requests. You may not +create buckets as an anonymous user. + +Constraints +~~~~~~~~~~~ +In general, bucket names should follow domain name constraints. + +- Bucket names must be unique. +- Bucket names cannot be formatted as IP address. +- Bucket names can be between 3 and 63 characters long. +- Bucket names must not contain uppercase characters or underscores. +- Bucket names must start with a lowercase letter or number. +- Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number. + +.. note:: The above constraints are relaxed if the option 'rgw_relaxed_s3_bucket_names' is set to true except that the bucket names must still be unique, cannot be formatted as IP address and can contain letters, numbers, periods, dashes and underscores for up to 255 characters long. + +Syntax +~~~~~~ + +:: + + PUT /{bucket} HTTP/1.1 + Host: cname.domain.com + x-amz-acl: public-read-write + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Parameters +~~~~~~~~~~ + + ++---------------+----------------------+-----------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++===============+======================+=============================================================================+============+ +| ``x-amz-acl`` | Canned ACLs. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++---------------+----------------------+-----------------------------------------------------------------------------+------------+ +| ``x-amz-bucket-object-lock-enabled`` | Enable object lock on bucket. | ``true``, ``false`` | No | ++--------------------------------------+-------------------------------+---------------------------------------------+------------+ + +Request Entities +~~~~~~~~~~~~~~~~ + ++-------------------------------+-----------+----------------------------------------------------------------+ +| Name | Type | Description | ++===============================+===========+================================================================+ +| ``CreateBucketConfiguration`` | Container | A container for the bucket configuration. | ++-------------------------------+-----------+----------------------------------------------------------------+ +| ``LocationConstraint`` | String | A zonegroup api name, with optional :ref:`s3_bucket_placement` | ++-------------------------------+-----------+----------------------------------------------------------------+ + + +HTTP Response +~~~~~~~~~~~~~ + +If the bucket name is unique, within constraints and unused, the operation will succeed. +If a bucket with the same name already exists and the user is the bucket owner, the operation will succeed. +If the bucket name is already in use, the operation will fail. + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``409`` | BucketAlreadyExists | Bucket already exists under different user's ownership. | ++---------------+-----------------------+----------------------------------------------------------+ + +DELETE Bucket +------------- + +Deletes a bucket. You can reuse bucket names following a successful bucket removal. + +Syntax +~~~~~~ + +:: + + DELETE /{bucket} HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+---------------+------------------+ +| HTTP Status | Status Code | Description | ++===============+===============+==================+ +| ``204`` | No Content | Bucket removed. | ++---------------+---------------+------------------+ + +GET Bucket +---------- +Returns a list of bucket objects. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?max-keys=25 HTTP/1.1 + Host: cname.domain.com + +Parameters +~~~~~~~~~~ + ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=====================+===========+=================================================================================================+ +| ``prefix`` | String | Only returns objects that contain the specified prefix. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``delimiter`` | String | The delimiter between the prefix and the rest of the object name. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``marker`` | String | A beginning index for the list of objects returned. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``max-keys`` | Integer | The maximum number of keys to return. Default is 1000. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``allow-unordered`` | Boolean | Non-standard extension. Allows results to be returned unordered. Cannot be used with delimiter. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+---------------+--------------------+ +| HTTP Status | Status Code | Description | ++===============+===============+====================+ +| ``200`` | OK | Buckets retrieved | ++---------------+---------------+--------------------+ + +Bucket Response Entities +~~~~~~~~~~~~~~~~~~~~~~~~ +``GET /{bucket}`` returns a container for buckets with the following fields. + ++------------------------+-----------+----------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+==================================================================================+ +| ``ListBucketResult`` | Entity | The container for the list of objects. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Name`` | String | The name of the bucket whose contents will be returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Prefix`` | String | A prefix for the object keys. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Marker`` | String | A beginning index for the list of objects returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``MaxKeys`` | Integer | The maximum number of keys returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Delimiter`` | String | If set, objects with the same prefix will appear in the ``CommonPrefixes`` list. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``IsTruncated`` | Boolean | If ``true``, only a subset of the bucket's contents were returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``CommonPrefixes`` | Container | If multiple objects contain the same prefix, they will appear in this list. | ++------------------------+-----------+----------------------------------------------------------------------------------+ + +Object Response Entities +~~~~~~~~~~~~~~~~~~~~~~~~ +The ``ListBucketResult`` contains objects, where each object is within a ``Contents`` container. + ++------------------------+-----------+------------------------------------------+ +| Name | Type | Description | ++========================+===========+==========================================+ +| ``Contents`` | Object | A container for the object. | ++------------------------+-----------+------------------------------------------+ +| ``Key`` | String | The object's key. | ++------------------------+-----------+------------------------------------------+ +| ``LastModified`` | Date | The object's last-modified date/time. | ++------------------------+-----------+------------------------------------------+ +| ``ETag`` | String | An MD-5 hash of the object. (entity tag) | ++------------------------+-----------+------------------------------------------+ +| ``Size`` | Integer | The object's size. | ++------------------------+-----------+------------------------------------------+ +| ``StorageClass`` | String | Should always return ``STANDARD``. | ++------------------------+-----------+------------------------------------------+ +| ``Type`` | String | ``Appendable`` or ``Normal``. | ++------------------------+-----------+------------------------------------------+ + +Get Bucket Location +------------------- +Retrieves the bucket's region. The user needs to be the bucket owner +to call this. A bucket can be constrained to a region by providing +``LocationConstraint`` during a PUT request. + +Syntax +~~~~~~ +Add the ``location`` subresource to bucket resource as shown below + +:: + + GET /{bucket}?location HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~~~~~~~~ + ++------------------------+-----------+------------------------------------------+ +| Name | Type | Description | ++========================+===========+==========================================+ +| ``LocationConstraint`` | String | The region where bucket resides, empty | +| | | string for default region | ++------------------------+-----------+------------------------------------------+ + + + +Get Bucket ACL +-------------- +Retrieves the bucket access control list. The user needs to be the bucket +owner or to have been granted ``READ_ACP`` permission on the bucket. + +Syntax +~~~~~~ +Add the ``acl`` subresource to the bucket request as shown below. + +:: + + GET /{bucket}?acl HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the response. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the bucket owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The bucket owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The bucket owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` bucket. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + +PUT Bucket ACL +-------------- +Sets an access control to an existing bucket. The user needs to be the bucket +owner or to have been granted ``WRITE_ACP`` permission on the bucket. + +Syntax +~~~~~~ +Add the ``acl`` subresource to the bucket request as shown below. + +:: + + PUT /{bucket}?acl HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the request. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the bucket owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The bucket owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The bucket owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` bucket. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + +List Bucket Multipart Uploads +----------------------------- + +``GET /?uploads`` returns a list of the current in-progress multipart uploads--i.e., the application initiates a multipart upload, but +the service hasn't completed all the uploads yet. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?uploads HTTP/1.1 + +Parameters +~~~~~~~~~~ + +You may specify parameters for ``GET /{bucket}?uploads``, but none of them are required. + ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+======================================================================================+ +| ``prefix`` | String | Returns in-progress uploads whose keys contains the specified prefix. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``delimiter`` | String | The delimiter between the prefix and the rest of the object name. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``key-marker`` | String | The beginning marker for the list of uploads. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``max-keys`` | Integer | The maximum number of in-progress uploads. The default is 1000. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``max-uploads`` | Integer | The maximum number of multipart uploads. The range from 1-1000. The default is 1000. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``upload-id-marker`` | String | Ignored if ``key-marker`` is not specified. Specifies the ``ID`` of first | +| | | upload to list in lexicographical order at or following the ``ID``. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=========================================+=============+==========================================================================================================+ +| ``ListMultipartUploadsResult`` | Container | A container for the results. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ListMultipartUploadsResult.Prefix`` | String | The prefix specified by the ``prefix`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Bucket`` | String | The bucket that will receive the bucket contents. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``KeyMarker`` | String | The key marker specified by the ``key-marker`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadIdMarker`` | String | The marker specified by the ``upload-id-marker`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``NextKeyMarker`` | String | The key marker to use in a subsequent request if ``IsTruncated`` is ``true``. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``NextUploadIdMarker`` | String | The upload ID marker to use in a subsequent request if ``IsTruncated`` is ``true``. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``MaxUploads`` | Integer | The max uploads specified by the ``max-uploads`` request parameter. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Delimiter`` | String | If set, objects with the same prefix will appear in the ``CommonPrefixes`` list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``IsTruncated`` | Boolean | If ``true``, only a subset of the bucket's upload contents were returned. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Upload`` | Container | A container for ``Key``, ``UploadId``, ``InitiatorOwner``, ``StorageClass``, and ``Initiated`` elements. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Key`` | String | The key of the object once the multipart upload is complete. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadId`` | String | The ``ID`` that identifies the multipart upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Initiator`` | Container | Contains the ``ID`` and ``DisplayName`` of the user who initiated the upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The initiator's display name. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ID`` | String | The initiator's ID. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the ``ID`` and ``DisplayName`` of the user who owns the uploaded object. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``StorageClass`` | String | The method used to store the resulting object. ``STANDARD`` or ``REDUCED_REDUNDANCY`` | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Initiated`` | Date | The date and time the user initiated the upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``CommonPrefixes`` | Container | If multiple objects contain the same prefix, they will appear in this list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``CommonPrefixes.Prefix`` | String | The substring of the key after the prefix as defined by the ``prefix`` request parameter. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ + +ENABLE/SUSPEND BUCKET VERSIONING +-------------------------------- + +``PUT /?versioning`` This subresource set the versioning state of an existing bucket. To set the versioning state, you must be the bucket owner. + +You can set the versioning state with one of the following values: + +- Enabled : Enables versioning for the objects in the bucket, All objects added to the bucket receive a unique version ID. +- Suspended : Disables versioning for the objects in the bucket, All objects added to the bucket receive the version ID null. + +If the versioning state has never been set on a bucket, it has no versioning state; a GET versioning request does not return a versioning state value. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?versioning HTTP/1.1 + +REQUEST ENTITIES +~~~~~~~~~~~~~~~~ + ++-----------------------------+-----------+---------------------------------------------------------------------------+ +| Name | Type | Description | ++=============================+===========+===========================================================================+ +| ``VersioningConfiguration`` | Container | A container for the request. | ++-----------------------------+-----------+---------------------------------------------------------------------------+ +| ``Status`` | String | Sets the versioning state of the bucket. Valid Values: Suspended/Enabled | ++-----------------------------+-----------+---------------------------------------------------------------------------+ + +PUT BUCKET OBJECT LOCK +-------------------------------- + +Places an Object Lock configuration on the specified bucket. The rule specified in the Object Lock configuration will be +applied by default to every new object placed in the specified bucket. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?object-lock HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++=============================+=============+========================================================================================+==========+ +| ``ObjectLockConfiguration`` | Container | A container for the request. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``ObjectLockEnabled`` | String | Indicates whether this bucket has an Object Lock configuration enabled. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Rule`` | Container | The Object Lock rule in place for the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``DefaultRetention`` | Container | The default retention period applied to new objects placed in the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Mode`` | String | The default Object Lock retention mode. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Days`` | Integer | The number of days specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Years`` | Integer | The number of years specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ + +HTTP Response +~~~~~~~~~~~~~ + +If the bucket object lock is not enabled when creating the bucket, the operation will fail. + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``400`` | MalformedXML | The XML is not well-formed | ++---------------+-----------------------+----------------------------------------------------------+ +| ``409`` | InvalidBucketState | The bucket object lock is not enabled | ++---------------+-----------------------+----------------------------------------------------------+ + +GET BUCKET OBJECT LOCK +-------------------------------- + +Gets the Object Lock configuration for a bucket. The rule specified in the Object Lock configuration will be applied by +default to every new object placed in the specified bucket. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?object-lock HTTP/1.1 + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++=============================+=============+========================================================================================+==========+ +| ``ObjectLockConfiguration`` | Container | A container for the request. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``ObjectLockEnabled`` | String | Indicates whether this bucket has an Object Lock configuration enabled. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Rule`` | Container | The Object Lock rule in place for the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``DefaultRetention`` | Container | The default retention period applied to new objects placed in the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Mode`` | String | The default Object Lock retention mode. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Days`` | Integer | The number of days specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Years`` | Integer | The number of years specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ + +Create Notification +------------------- + +Create a publisher for a specific bucket into a topic. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?notification HTTP/1.1 + + +Request Entities +~~~~~~~~~~~~~~~~ + +Parameters are XML encoded in the body of the request, in the following format: + +:: + + <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <TopicConfiguration> + <Id></Id> + <Topic></Topic> + <Event></Event> + <Filter> + <S3Key> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Key> + <S3Metadata> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Metadata> + <S3Tags> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Tags> + </Filter> + </TopicConfiguration> + </NotificationConfiguration> + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``NotificationConfiguration`` | Container | Holding list of ``TopicConfiguration`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TopicConfiguration`` | Container | Holding ``Id``, ``Topic`` and list of ``Event`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Id`` | String | Name of the notification | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Topic`` | String | Topic ARN. Topic must be created beforehand | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Event`` | String | List of supported events see: `S3 Notification Compatibility`_. Multiple ``Event`` | No | +| | | entities can be used. If omitted, all events are handled | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Filter`` | Container | Holding ``S3Key``, ``S3Metadata`` and ``S3Tags`` entities | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Key`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object key. | No | +| | | At most, 3 entities may be in the list, with ``Name`` be ``prefix``, ``suffix`` or | | +| | | ``regex``. All filter rules in the list must match for the filter to match. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Metadata`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object metadata. | No | +| | | All filter rules in the list must match the metadata defined on the object. However, | | +| | | the object still match if it has other metadata entries not listed in the filter. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Tags`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object tags. | No | +| | | All filter rules in the list must match the tags defined on the object. However, | | +| | | the object still match it it has other tags not listed in the filter. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Key.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be: ``prefix``, ``suffix`` | Yes | +| | | or ``regex``. The ``Value`` would hold the key prefix, key suffix or a regular | | +| | | expression for matching the key, accordingly. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Metadata.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the name of the metadata | Yes | +| | | attribute (e.g. ``x-amz-meta-xxx``). The ``Value`` would be the expected value for | | +| | | this attribute. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Tags.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the tag key, | Yes | +| | | and ``Value`` would be the tag value. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``400`` | MalformedXML | The XML is not well-formed | ++---------------+-----------------------+----------------------------------------------------------+ +| ``400`` | InvalidArgument | Missing Id; Missing/Invalid Topic ARN; Invalid Event | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchKey | The topic does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + + +Delete Notification +------------------- + +Delete a specific, or all, notifications from a bucket. + +.. note:: + + - Notification deletion is an extension to the S3 notification API + - When the bucket is deleted, any notification defined on it is also deleted + - Deleting an unknown notification (e.g. double delete) is not considered an error + +Syntax +~~~~~~ + +:: + + DELETE /{bucket}?notification[=<notification-id>] HTTP/1.1 + + +Parameters +~~~~~~~~~~ + ++------------------------+-----------+----------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+========================================================================================+ +| ``notification-id`` | String | Name of the notification. If not provided, all notifications on the bucket are deleted | ++------------------------+-----------+----------------------------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + +Get/List Notification +--------------------- + +Get a specific notification, or list all notifications configured on a bucket. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?notification[=<notification-id>] HTTP/1.1 + + +Parameters +~~~~~~~~~~ + ++------------------------+-----------+----------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+========================================================================================+ +| ``notification-id`` | String | Name of the notification. If not provided, all notifications on the bucket are listed | ++------------------------+-----------+----------------------------------------------------------------------------------------+ + +Response Entities +~~~~~~~~~~~~~~~~~ + +Response is XML encoded in the body of the request, in the following format: + +:: + + <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <TopicConfiguration> + <Id></Id> + <Topic></Topic> + <Event></Event> + <Filter> + <S3Key> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Key> + <S3Metadata> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Metadata> + <S3Tags> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Tags> + </Filter> + </TopicConfiguration> + </NotificationConfiguration> + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``NotificationConfiguration`` | Container | Holding list of ``TopicConfiguration`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TopicConfiguration`` | Container | Holding ``Id``, ``Topic`` and list of ``Event`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Id`` | String | Name of the notification | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Topic`` | String | Topic ARN | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Event`` | String | Handled event. Multiple ``Event`` entities may exist | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Filter`` | Container | Holding the filters configured for this notification | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchKey | The notification does not exist (if provided) | ++---------------+-----------------------+----------------------------------------------------------+ + +.. _S3 Notification Compatibility: ../../s3-notification-compatibility diff --git a/doc/radosgw/s3/commons.rst b/doc/radosgw/s3/commons.rst new file mode 100644 index 000000000..4b9b4a040 --- /dev/null +++ b/doc/radosgw/s3/commons.rst @@ -0,0 +1,113 @@ +================= + Common Entities +================= + +.. toctree:: + :maxdepth: -1 + +Bucket and Host Name +-------------------- +There are two different modes of accessing the buckets. The first (preferred) method +identifies the bucket as the top-level directory in the URI. :: + + GET /mybucket HTTP/1.1 + Host: cname.domain.com + +The second method identifies the bucket via a virtual bucket host name. For example:: + + GET / HTTP/1.1 + Host: mybucket.cname.domain.com + +To configure virtual hosted buckets, you can either set ``rgw_dns_name = cname.domain.com`` in ceph.conf, or add ``cname.domain.com`` to the list of ``hostnames`` in your zonegroup configuration. See `Ceph Object Gateway - Multisite Configuration`_ for more on zonegroups. + +.. tip:: We prefer the first method, because the second method requires expensive domain certification and DNS wild cards. + +.. tip:: You can define multiple hostname directly with the :confval:`rgw_dns_name` parameter. + +Common Request Headers +---------------------- + ++--------------------+------------------------------------------+ +| Request Header | Description | ++====================+==========================================+ +| ``CONTENT_LENGTH`` | Length of the request body. | ++--------------------+------------------------------------------+ +| ``DATE`` | Request time and date (in UTC). | ++--------------------+------------------------------------------+ +| ``HOST`` | The name of the host server. | ++--------------------+------------------------------------------+ +| ``AUTHORIZATION`` | Authorization token. | ++--------------------+------------------------------------------+ + +Common Response Status +---------------------- + ++---------------+-----------------------------------+ +| HTTP Status | Response Code | ++===============+===================================+ +| ``100`` | Continue | ++---------------+-----------------------------------+ +| ``200`` | Success | ++---------------+-----------------------------------+ +| ``201`` | Created | ++---------------+-----------------------------------+ +| ``202`` | Accepted | ++---------------+-----------------------------------+ +| ``204`` | NoContent | ++---------------+-----------------------------------+ +| ``206`` | Partial content | ++---------------+-----------------------------------+ +| ``304`` | NotModified | ++---------------+-----------------------------------+ +| ``400`` | InvalidArgument | ++---------------+-----------------------------------+ +| ``400`` | InvalidDigest | ++---------------+-----------------------------------+ +| ``400`` | BadDigest | ++---------------+-----------------------------------+ +| ``400`` | InvalidBucketName | ++---------------+-----------------------------------+ +| ``400`` | InvalidObjectName | ++---------------+-----------------------------------+ +| ``400`` | UnresolvableGrantByEmailAddress | ++---------------+-----------------------------------+ +| ``400`` | InvalidPart | ++---------------+-----------------------------------+ +| ``400`` | InvalidPartOrder | ++---------------+-----------------------------------+ +| ``400`` | RequestTimeout | ++---------------+-----------------------------------+ +| ``400`` | EntityTooLarge | ++---------------+-----------------------------------+ +| ``403`` | AccessDenied | ++---------------+-----------------------------------+ +| ``403`` | UserSuspended | ++---------------+-----------------------------------+ +| ``403`` | RequestTimeTooSkewed | ++---------------+-----------------------------------+ +| ``404`` | NoSuchKey | ++---------------+-----------------------------------+ +| ``404`` | NoSuchBucket | ++---------------+-----------------------------------+ +| ``404`` | NoSuchUpload | ++---------------+-----------------------------------+ +| ``405`` | MethodNotAllowed | ++---------------+-----------------------------------+ +| ``408`` | RequestTimeout | ++---------------+-----------------------------------+ +| ``409`` | BucketAlreadyExists | ++---------------+-----------------------------------+ +| ``409`` | BucketNotEmpty | ++---------------+-----------------------------------+ +| ``411`` | MissingContentLength | ++---------------+-----------------------------------+ +| ``412`` | PreconditionFailed | ++---------------+-----------------------------------+ +| ``416`` | InvalidRange | ++---------------+-----------------------------------+ +| ``422`` | UnprocessableEntity | ++---------------+-----------------------------------+ +| ``500`` | InternalError | ++---------------+-----------------------------------+ + +.. _`Ceph Object Gateway - Multisite Configuration`: ../../multisite diff --git a/doc/radosgw/s3/cpp.rst b/doc/radosgw/s3/cpp.rst new file mode 100644 index 000000000..089c9c53a --- /dev/null +++ b/doc/radosgw/s3/cpp.rst @@ -0,0 +1,337 @@ +.. _cpp: + +C++ S3 Examples +=============== + +Setup +----- + +The following contains includes and globals that will be used in later examples: + +.. code-block:: cpp + + #include "libs3.h" + #include <stdlib.h> + #include <iostream> + #include <fstream> + + const char access_key[] = "ACCESS_KEY"; + const char secret_key[] = "SECRET_KEY"; + const char host[] = "HOST"; + const char sample_bucket[] = "sample_bucket"; + const char sample_key[] = "hello.txt"; + const char sample_file[] = "resource/hello.txt"; + const char *security_token = NULL; + const char *auth_region = NULL; + + S3BucketContext bucketContext = + { + host, + sample_bucket, + S3ProtocolHTTP, + S3UriStylePath, + access_key, + secret_key, + security_token, + auth_region + }; + + S3Status responsePropertiesCallback( + const S3ResponseProperties *properties, + void *callbackData) + { + return S3StatusOK; + } + + static void responseCompleteCallback( + S3Status status, + const S3ErrorDetails *error, + void *callbackData) + { + return; + } + + S3ResponseHandler responseHandler = + { + &responsePropertiesCallback, + &responseCompleteCallback + }; + + +Creating (and Closing) a Connection +----------------------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: cpp + + S3_initialize("s3", S3_INIT_ALL, host); + // Do stuff... + S3_deinitialize(); + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name, owner ID, and display name +for each bucket. + +.. code-block:: cpp + + static S3Status listServiceCallback( + const char *ownerId, + const char *ownerDisplayName, + const char *bucketName, + int64_t creationDate, void *callbackData) + { + bool *header_printed = (bool*) callbackData; + if (!*header_printed) { + *header_printed = true; + printf("%-22s", " Bucket"); + printf(" %-20s %-12s", " Owner ID", "Display Name"); + printf("\n"); + printf("----------------------"); + printf(" --------------------" " ------------"); + printf("\n"); + } + + printf("%-22s", bucketName); + printf(" %-20s %-12s", ownerId ? ownerId : "", ownerDisplayName ? ownerDisplayName : ""); + printf("\n"); + + return S3StatusOK; + } + + S3ListServiceHandler listServiceHandler = + { + responseHandler, + &listServiceCallback + }; + bool header_printed = false; + S3_list_service(S3ProtocolHTTP, access_key, secret_key, security_token, host, + auth_region, NULL, 0, &listServiceHandler, &header_printed); + + +Creating a Bucket +----------------- + +This creates a new bucket. + +.. code-block:: cpp + + S3_create_bucket(S3ProtocolHTTP, access_key, secret_key, NULL, host, sample_bucket, S3CannedAclPrivate, NULL, NULL, &responseHandler, NULL); + + +Listing a Bucket's Content +-------------------------- + +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and +last modified date. + +.. code-block:: cpp + + static S3Status listBucketCallback( + int isTruncated, + const char *nextMarker, + int contentsCount, + const S3ListBucketContent *contents, + int commonPrefixesCount, + const char **commonPrefixes, + void *callbackData) + { + printf("%-22s", " Object Name"); + printf(" %-5s %-20s", "Size", " Last Modified"); + printf("\n"); + printf("----------------------"); + printf(" -----" " --------------------"); + printf("\n"); + + for (int i = 0; i < contentsCount; i++) { + char timebuf[256]; + char sizebuf[16]; + const S3ListBucketContent *content = &(contents[i]); + time_t t = (time_t) content->lastModified; + + strftime(timebuf, sizeof(timebuf), "%Y-%m-%dT%H:%M:%SZ", gmtime(&t)); + sprintf(sizebuf, "%5llu", (unsigned long long) content->size); + printf("%-22s %s %s\n", content->key, sizebuf, timebuf); + } + + return S3StatusOK; + } + + S3ListBucketHandler listBucketHandler = + { + responseHandler, + &listBucketCallback + }; + S3_list_bucket(&bucketContext, NULL, NULL, NULL, 0, NULL, 0, &listBucketHandler, NULL); + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: cpp + + S3_delete_bucket(S3ProtocolHTTP, S3UriStylePath, access_key, secret_key, 0, host, sample_bucket, NULL, NULL, 0, &responseHandler, NULL); + + +Creating an Object (from a file) +-------------------------------- + +This creates a file ``hello.txt``. + +.. code-block:: cpp + + #include <sys/stat.h> + typedef struct put_object_callback_data + { + FILE *infile; + uint64_t contentLength; + } put_object_callback_data; + + + static int putObjectDataCallback(int bufferSize, char *buffer, void *callbackData) + { + put_object_callback_data *data = (put_object_callback_data *) callbackData; + + int ret = 0; + + if (data->contentLength) { + int toRead = ((data->contentLength > (unsigned) bufferSize) ? (unsigned) bufferSize : data->contentLength); + ret = fread(buffer, 1, toRead, data->infile); + } + data->contentLength -= ret; + return ret; + } + + put_object_callback_data data; + struct stat statbuf; + if (stat(sample_file, &statbuf) == -1) { + fprintf(stderr, "\nERROR: Failed to stat file %s: ", sample_file); + perror(0); + exit(-1); + } + + int contentLength = statbuf.st_size; + data.contentLength = contentLength; + + if (!(data.infile = fopen(sample_file, "r"))) { + fprintf(stderr, "\nERROR: Failed to open input file %s: ", sample_file); + perror(0); + exit(-1); + } + + S3PutObjectHandler putObjectHandler = + { + responseHandler, + &putObjectDataCallback + }; + + S3_put_object(&bucketContext, sample_key, contentLength, NULL, NULL, 0, &putObjectHandler, &data); + fclose(data.infile); + + +Download an Object (to a file) +------------------------------ + +This downloads a file and prints the contents. + +.. code-block:: cpp + + static S3Status getObjectDataCallback(int bufferSize, const char *buffer, void *callbackData) + { + FILE *outfile = (FILE *) callbackData; + size_t wrote = fwrite(buffer, 1, bufferSize, outfile); + return ((wrote < (size_t) bufferSize) ? S3StatusAbortedByCallback : S3StatusOK); + } + + S3GetObjectHandler getObjectHandler = + { + responseHandler, + &getObjectDataCallback + }; + FILE *outfile = stdout; + S3_get_object(&bucketContext, sample_key, NULL, 0, 0, NULL, 0, &getObjectHandler, outfile); + + +Delete an Object +---------------- + +This deletes an object. + +.. code-block:: cpp + + S3ResponseHandler deleteResponseHandler = + { + 0, + &responseCompleteCallback + }; + S3_delete_object(&bucketContext, sample_key, 0, 0, &deleteResponseHandler, 0); + + +Change an Object's ACL +---------------------- + +This changes an object's ACL to grant full control to another user. + + +.. code-block:: cpp + + #include <string.h> + char ownerId[] = "owner"; + char ownerDisplayName[] = "owner"; + char granteeId[] = "grantee"; + char granteeDisplayName[] = "grantee"; + + S3AclGrant grants[] = { + { + S3GranteeTypeCanonicalUser, + {{}}, + S3PermissionFullControl + }, + { + S3GranteeTypeCanonicalUser, + {{}}, + S3PermissionReadACP + }, + { + S3GranteeTypeAllUsers, + {{}}, + S3PermissionRead + } + }; + + strncpy(grants[0].grantee.canonicalUser.id, ownerId, S3_MAX_GRANTEE_USER_ID_SIZE); + strncpy(grants[0].grantee.canonicalUser.displayName, ownerDisplayName, S3_MAX_GRANTEE_DISPLAY_NAME_SIZE); + + strncpy(grants[1].grantee.canonicalUser.id, granteeId, S3_MAX_GRANTEE_USER_ID_SIZE); + strncpy(grants[1].grantee.canonicalUser.displayName, granteeDisplayName, S3_MAX_GRANTEE_DISPLAY_NAME_SIZE); + + S3_set_acl(&bucketContext, sample_key, ownerId, ownerDisplayName, 3, grants, 0, &responseHandler, 0); + + +Generate Object Download URL (signed) +------------------------------------- + +This generates a signed download URL that will be valid for 5 minutes. + +.. code-block:: cpp + + #include <time.h> + char buffer[S3_MAX_AUTHENTICATED_QUERY_STRING_SIZE]; + int64_t expires = time(NULL) + 60 * 5; // Current time + 5 minutes + + S3_generate_authenticated_query_string(buffer, &bucketContext, sample_key, expires, NULL, "GET"); + diff --git a/doc/radosgw/s3/csharp.rst b/doc/radosgw/s3/csharp.rst new file mode 100644 index 000000000..af1c6e4b5 --- /dev/null +++ b/doc/radosgw/s3/csharp.rst @@ -0,0 +1,199 @@ +.. _csharp: + +C# S3 Examples +============== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: csharp + + using System; + using Amazon; + using Amazon.S3; + using Amazon.S3.Model; + + string accessKey = "put your access key here!"; + string secretKey = "put your secret key here!"; + + AmazonS3Config config = new AmazonS3Config(); + config.ServiceURL = "objects.dreamhost.com"; + + AmazonS3Client s3Client = new AmazonS3Client( + accessKey, + secretKey, + config + ); + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: csharp + + ListBucketsResponse response = client.ListBuckets(); + foreach (S3Bucket b in response.Buckets) + { + Console.WriteLine("{0}\t{1}", b.BucketName, b.CreationDate); + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: csharp + + PutBucketRequest request = new PutBucketRequest(); + request.BucketName = "my-new-bucket"; + client.PutBucket(request); + +Listing a Bucket's Content +-------------------------- + +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: csharp + + ListObjectsRequest request = new ListObjectsRequest(); + request.BucketName = "my-new-bucket"; + ListObjectsResponse response = client.ListObjects(request); + foreach (S3Object o in response.S3Objects) + { + Console.WriteLine("{0}\t{1}\t{2}", o.Key, o.Size, o.LastModified); + } + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: csharp + + DeleteBucketRequest request = new DeleteBucketRequest(); + request.BucketName = "my-new-bucket"; + client.DeleteBucket(request); + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. attention:: + + not available + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: csharp + + PutObjectRequest request = new PutObjectRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "hello.txt"; + request.ContentType = "text/plain"; + request.ContentBody = "Hello World!"; + client.PutObject(request); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and +``secret_plans.txt`` to be private. + +.. code-block:: csharp + + PutACLRequest request = new PutACLRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "hello.txt"; + request.CannedACL = S3CannedACL.PublicRead; + client.PutACL(request); + + PutACLRequest request2 = new PutACLRequest(); + request2.BucketName = "my-new-bucket"; + request2.Key = "secret_plans.txt"; + request2.CannedACL = S3CannedACL.Private; + client.PutACL(request2); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``C:\Users\larry\Documents`` + +.. code-block:: csharp + + GetObjectRequest request = new GetObjectRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "perl_poetry.pdf"; + GetObjectResponse response = client.GetObject(request); + response.WriteResponseStreamToFile("C:\\Users\\larry\\Documents\\perl_poetry.pdf"); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: csharp + + DeleteObjectRequest request = new DeleteObjectRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "goodbye.txt"; + client.DeleteObject(request); + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. note:: + + The C# S3 Library does not have a method for generating unsigned + URLs, so the following example only shows generating signed URLs. + +.. code-block:: csharp + + GetPreSignedUrlRequest request = new GetPreSignedUrlRequest(); + request.BucketName = "my-bucket-name"; + request.Key = "secret_plans.txt"; + request.Expires = DateTime.Now.AddHours(1); + request.Protocol = Protocol.HTTP; + string url = client.GetPreSignedURL(request); + Console.WriteLine(url); + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + diff --git a/doc/radosgw/s3/java.rst b/doc/radosgw/s3/java.rst new file mode 100644 index 000000000..057c09c2c --- /dev/null +++ b/doc/radosgw/s3/java.rst @@ -0,0 +1,212 @@ +.. _java: + +Java S3 Examples +================ + +Setup +----- + +The following examples may require some or all of the following java +classes to be imported: + +.. code-block:: java + + import java.io.ByteArrayInputStream; + import java.io.File; + import java.util.List; + import com.amazonaws.auth.AWSCredentials; + import com.amazonaws.auth.BasicAWSCredentials; + import com.amazonaws.util.StringUtils; + import com.amazonaws.services.s3.AmazonS3; + import com.amazonaws.services.s3.AmazonS3Client; + import com.amazonaws.services.s3.model.Bucket; + import com.amazonaws.services.s3.model.CannedAccessControlList; + import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest; + import com.amazonaws.services.s3.model.GetObjectRequest; + import com.amazonaws.services.s3.model.ObjectListing; + import com.amazonaws.services.s3.model.ObjectMetadata; + import com.amazonaws.services.s3.model.S3ObjectSummary; + + +If you are just testing the Ceph Object Storage services, consider +using HTTP protocol instead of HTTPS protocol. + +First, import the ``ClientConfiguration`` and ``Protocol`` classes. + +.. code-block:: java + + import com.amazonaws.ClientConfiguration; + import com.amazonaws.Protocol; + + +Then, define the client configuration, and add the client configuration +as an argument for the S3 client. + +.. code-block:: java + + AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); + + ClientConfiguration clientConfig = new ClientConfiguration(); + clientConfig.setProtocol(Protocol.HTTP); + + AmazonS3 conn = new AmazonS3Client(credentials, clientConfig); + conn.setEndpoint("endpoint.com"); + + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: java + + String accessKey = "insert your access key here!"; + String secretKey = "insert your secret key here!"; + + AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); + AmazonS3 conn = new AmazonS3Client(credentials); + conn.setEndpoint("objects.dreamhost.com"); + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: java + + List<Bucket> buckets = conn.listBuckets(); + for (Bucket bucket : buckets) { + System.out.println(bucket.getName() + "\t" + + StringUtils.fromDate(bucket.getCreationDate())); + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: java + + Bucket bucket = conn.createBucket("my-new-bucket"); + + +Listing a Bucket's Content +-------------------------- +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: java + + ObjectListing objects = conn.listObjects(bucket.getName()); + do { + for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) { + System.out.println(objectSummary.getKey() + "\t" + + objectSummary.getSize() + "\t" + + StringUtils.fromDate(objectSummary.getLastModified())); + } + objects = conn.listNextBatchOfObjects(objects); + } while (objects.isTruncated()); + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: java + + conn.deleteBucket(bucket.getName()); + + +Forced Delete for Non-empty Buckets +----------------------------------- +.. attention:: + not available + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: java + + ByteArrayInputStream input = new ByteArrayInputStream("Hello World!".getBytes()); + conn.putObject(bucket.getName(), "hello.txt", input, new ObjectMetadata()); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and +``secret_plans.txt`` to be private. + +.. code-block:: java + + conn.setObjectAcl(bucket.getName(), "hello.txt", CannedAccessControlList.PublicRead); + conn.setObjectAcl(bucket.getName(), "secret_plans.txt", CannedAccessControlList.Private); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``/home/larry/documents`` + +.. code-block:: java + + conn.getObject( + new GetObjectRequest(bucket.getName(), "perl_poetry.pdf"), + new File("/home/larry/documents/perl_poetry.pdf") + ); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: java + + conn.deleteObject(bucket.getName(), "goodbye.txt"); + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. note:: + The java library does not have a method for generating unsigned + URLs, so the example below just generates a signed URL. + +.. code-block:: java + + GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucket.getName(), "secret_plans.txt"); + System.out.println(conn.generatePresignedUrl(request)); + +The output will look something like this:: + + https://my-bucket-name.objects.dreamhost.com/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + diff --git a/doc/radosgw/s3/objectops.rst b/doc/radosgw/s3/objectops.rst new file mode 100644 index 000000000..2ac52607f --- /dev/null +++ b/doc/radosgw/s3/objectops.rst @@ -0,0 +1,558 @@ +Object Operations +================= + +Put Object +---------- +Adds an object to a bucket. You must have write permissions on the bucket to perform this operation. + + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object} HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================+============================================+===============================================================================+============+ +| **content-md5** | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **content-type** | A standard MIME type. | Any MIME type. Default: ``binary/octet-stream`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-meta-<...>** | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ + + +Copy Object +----------- +To copy an object, use ``PUT`` and specify a destination bucket and the object name. + +Syntax +~~~~~~ + +:: + + PUT /{dest-bucket}/{dest-object} HTTP/1.1 + x-amz-copy-source: {source-bucket}/{source-object} + +Request Headers +~~~~~~~~~~~~~~~ + ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================================+=================================================+========================+============+ +| **x-amz-copy-source** | The source bucket name + object name. | {bucket}/{obj} | Yes | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, | No | +| | | ``public-read``, | | +| | | ``public-read-write``, | | +| | | ``authenticated-read`` | | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-modified-since** | Copies only if modified since the timestamp. | Timestamp | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-unmodified-since** | Copies only if unmodified since the timestamp. | Timestamp | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-match** | Copies only if object ETag matches ETag. | Entity Tag | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-none-match** | Copies only if object ETag doesn't match. | Entity Tag | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ + +Response Entities +~~~~~~~~~~~~~~~~~ + ++------------------------+-------------+-----------------------------------------------+ +| Name | Type | Description | ++========================+=============+===============================================+ +| **CopyObjectResult** | Container | A container for the response elements. | ++------------------------+-------------+-----------------------------------------------+ +| **LastModified** | Date | The last modified date of the source object. | ++------------------------+-------------+-----------------------------------------------+ +| **Etag** | String | The ETag of the new object. | ++------------------------+-------------+-----------------------------------------------+ + +Remove Object +------------- + +Removes an object. Requires WRITE permission set on the containing bucket. + +Syntax +~~~~~~ + +:: + + DELETE /{bucket}/{object} HTTP/1.1 + + + +Get Object +---------- +Retrieves an object from a bucket within RADOS. + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object} HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| Name | Description | Valid Values | Required | ++===========================+================================================+================================+============+ +| **range** | The range of the object to retrieve. | Range: bytes=beginbyte-endbyte | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-modified-since** | Gets only if modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-unmodified-since** | Gets only if not modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ + +Response Headers +~~~~~~~~~~~~~~~~ + ++-------------------+--------------------------------------------------------------------------------------------+ +| Name | Description | ++===================+============================================================================================+ +| **Content-Range** | Data range, will only be returned if the range header field was specified in the request | ++-------------------+--------------------------------------------------------------------------------------------+ + +Get Object Info +--------------- + +Returns information about object. This request will return the same +header information as with the Get Object request, but will include +the metadata only, not the object data payload. + +Syntax +~~~~~~ + +:: + + HEAD /{bucket}/{object} HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| Name | Description | Valid Values | Required | ++===========================+================================================+================================+============+ +| **range** | The range of the object to retrieve. | Range: bytes=beginbyte-endbyte | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-modified-since** | Gets only if modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-unmodified-since** | Gets only if not modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ + +Get Object ACL +-------------- + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?acl HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the response. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the object owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The object owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The object owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` object. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + + + +Set Object ACL +-------------- + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?acl + +Request Entities +~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the response. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the object owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The object owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The object owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` object. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + + + +Initiate Multi-part Upload +-------------------------- + +Initiate a multi-part upload process. + +Syntax +~~~~~~ + +:: + + POST /{bucket}/{object}?uploads + +Request Headers +~~~~~~~~~~~~~~~ + ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================+============================================+===============================================================================+============+ +| **content-md5** | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **content-type** | A standard MIME type. | Any MIME type. Default: ``binary/octet-stream`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-meta-<...>** | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=========================================+=============+==========================================================================================================+ +| ``InitiatedMultipartUploadsResult`` | Container | A container for the results. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Bucket`` | String | The bucket that will receive the object contents. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Key`` | String | The key specified by the ``key`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadId`` | String | The ID specified by the ``upload-id`` request parameter identifying the multipart upload (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ + + +Multipart Upload Part +--------------------- + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?partNumber=&uploadId= HTTP/1.1 + +HTTP Response +~~~~~~~~~~~~~ + +The following HTTP response may be returned: + ++---------------+----------------+--------------------------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+================+==========================================================================+ +| **404** | NoSuchUpload | Specified upload-id does not match any initiated upload on this object | ++---------------+----------------+--------------------------------------------------------------------------+ + +List Multipart Upload Parts +--------------------------- + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?uploadId=123 HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=========================================+=============+==========================================================================================================+ +| ``ListPartsResult`` | Container | A container for the results. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Bucket`` | String | The bucket that will receive the object contents. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Key`` | String | The key specified by the ``key`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadId`` | String | The ID specified by the ``upload-id`` request parameter identifying the multipart upload (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Initiator`` | Container | Contains the ``ID`` and ``DisplayName`` of the user who initiated the upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ID`` | String | The initiator's ID. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The initiator's display name. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the ``ID`` and ``DisplayName`` of the user who owns the uploaded object. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``StorageClass`` | String | The method used to store the resulting object. ``STANDARD`` or ``REDUCED_REDUNDANCY`` | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``PartNumberMarker`` | String | The part marker to use in a subsequent request if ``IsTruncated`` is ``true``. Precedes the list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``NextPartNumberMarker`` | String | The next part marker to use in a subsequent request if ``IsTruncated`` is ``true``. The end of the list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``MaxParts`` | Integer | The max parts allowed in the response as specified by the ``max-parts`` request parameter. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``IsTruncated`` | Boolean | If ``true``, only a subset of the object's upload contents were returned. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Part`` | Container | A container for ``LastModified``, ``PartNumber``, ``ETag`` and ``Size`` elements. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``LastModified`` | Date | Date and time at which the part was uploaded. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``PartNumber`` | Integer | The identification number of the part. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ETag`` | String | The part's entity tag. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Size`` | Integer | The size of the uploaded part. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ + + + +Complete Multipart Upload +------------------------- +Assembles uploaded parts and creates a new object, thereby completing a multipart upload. + +Syntax +~~~~~~ + +:: + + POST /{bucket}/{object}?uploadId= HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| Name | Type | Description | Required | ++==================================+=============+=====================================================+==========+ +| ``CompleteMultipartUpload`` | Container | A container consisting of one or more parts. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| ``Part`` | Container | A container for the ``PartNumber`` and ``ETag``. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| ``PartNumber`` | Integer | The identifier of the part. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| ``ETag`` | String | The part's entity tag. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-------------------------------------+-------------+-------------------------------------------------------+ +| Name | Type | Description | ++=====================================+=============+=======================================================+ +| **CompleteMultipartUploadResult** | Container | A container for the response. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **Location** | URI | The resource identifier (path) of the new object. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **Bucket** | String | The name of the bucket that contains the new object. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **Key** | String | The object's key. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **ETag** | String | The entity tag of the new object. | ++-------------------------------------+-------------+-------------------------------------------------------+ + +Abort Multipart Upload +---------------------- + +Syntax +~~~~~~ + +:: + + DELETE /{bucket}/{object}?uploadId= HTTP/1.1 + + + +Append Object +------------- +Append data to an object. You must have write permissions on the bucket to perform this operation. +It is used to upload files in appending mode. The type of the objects created by the Append Object +operation is Appendable Object, and the type of the objects uploaded with the Put Object operation is Normal Object. +**Append Object can't be used if bucket versioning is enabled or suspended.** +**Synced object will become normal in multisite, but you can still append to the original object.** +**Compression and encryption features are disabled for Appendable objects.** + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?append&position= HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================+============================================+===============================================================================+============+ +| **content-md5** | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **content-type** | A standard MIME type. | Any MIME type. Default: ``binary/octet-stream`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-meta-<...>** | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ + +Response Headers +~~~~~~~~~~~~~~~~ + ++--------------------------------+------------------------------------------------------------------+ +| Name | Description | ++================================+==================================================================+ +| **x-rgw-next-append-position** | Next position to append object | ++--------------------------------+------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + +The following HTTP response may be returned: + ++---------------+----------------------------+---------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+============================+===================================================+ +| **409** | PositionNotEqualToLength | Specified position does not match object length | ++---------------+----------------------------+---------------------------------------------------+ +| **409** | ObjectNotAppendable | Specified object can not be appended | ++---------------+----------------------------+---------------------------------------------------+ +| **409** | InvalidBucketstate | Bucket versioning is enabled or suspended | ++---------------+----------------------------+---------------------------------------------------+ + + +Put Object Retention +-------------------- +Places an Object Retention configuration on an object. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?retention&versionId= HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++=====================+=============+===============================================================================+============+ +| ``Retention`` | Container | A container for the request. | Yes | ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| ``Mode`` | String | Retention mode for the specified object. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ +| ``RetainUntilDate`` | Timestamp | Retention date. Format: 2020-01-05T00:00:00.000Z | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ + + +Get Object Retention +-------------------- +Gets an Object Retention configuration on an object. + + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?retention&versionId= HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++=====================+=============+===============================================================================+============+ +| ``Retention`` | Container | A container for the request. | Yes | ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| ``Mode`` | String | Retention mode for the specified object. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ +| ``RetainUntilDate`` | Timestamp | Retention date. Format: 2020-01-05T00:00:00.000Z | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ + + +Put Object Legal Hold +--------------------- +Applies a Legal Hold configuration to the specified object. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?legal-hold&versionId= HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++================+=============+========================================================================================+============+ +| ``LegalHold`` | Container | A container for the request. | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| ``Status`` | String | Indicates whether the specified object has a Legal Hold in place. Valid Values: ON/OFF | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ + + +Get Object Legal Hold +--------------------- +Gets an object's current Legal Hold status. + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?legal-hold&versionId= HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++================+=============+========================================================================================+============+ +| ``LegalHold`` | Container | A container for the request. | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| ``Status`` | String | Indicates whether the specified object has a Legal Hold in place. Valid Values: ON/OFF | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ + diff --git a/doc/radosgw/s3/perl.rst b/doc/radosgw/s3/perl.rst new file mode 100644 index 000000000..f12e5c698 --- /dev/null +++ b/doc/radosgw/s3/perl.rst @@ -0,0 +1,192 @@ +.. _perl: + +Perl S3 Examples +================ + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: perl + + use Amazon::S3; + my $access_key = 'put your access key here!'; + my $secret_key = 'put your secret key here!'; + + my $conn = Amazon::S3->new({ + aws_access_key_id => $access_key, + aws_secret_access_key => $secret_key, + host => 'objects.dreamhost.com', + secure => 1, + retry => 1, + }); + + +Listing Owned Buckets +--------------------- + +This gets a list of `Amazon::S3::Bucket`_ objects that you own. +We'll also print out the bucket name and creation date of each bucket. + +.. code-block:: perl + + my @buckets = @{$conn->buckets->{buckets} || []}; + foreach my $bucket (@buckets) { + print $bucket->bucket . "\t" . $bucket->creation_date . "\n"; + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: perl + + my $bucket = $conn->add_bucket({ bucket => 'my-new-bucket' }); + + +Listing a Bucket's Content +-------------------------- + +This gets a list of hashes with info about each object in the bucket. +We'll also print out each object's name, the file size, and last +modified date. + +.. code-block:: perl + + my @keys = @{$bucket->list_all->{keys} || []}; + foreach my $key (@keys) { + print "$key->{key}\t$key->{size}\t$key->{last_modified}\n"; + } + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: perl + + $conn->delete_bucket($bucket); + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. attention:: + + not available in the `Amazon::S3`_ perl module + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: perl + + $bucket->add_key( + 'hello.txt', 'Hello World!', + { content_type => 'text/plain' }, + ); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable and +``secret_plans.txt`` to be private. + +.. code-block:: perl + + $bucket->set_acl({ + key => 'hello.txt', + acl_short => 'public-read', + }); + $bucket->set_acl({ + key => 'secret_plans.txt', + acl_short => 'private', + }); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: perl + + $bucket->get_key_filename('perl_poetry.pdf', undef, + '/home/larry/documents/perl_poetry.pdf'); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: perl + + $bucket->delete_key('goodbye.txt'); + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +Then this generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. note:: + The `Amazon::S3`_ module does not have a way to generate download + URLs, so we are going to be using another module instead. Unfortunately, + most modules for generating these URLs assume that you are using Amazon, + so we have had to go with using a more obscure module, `Muck::FS::S3`_. This + should be the same as Amazon's sample S3 perl module, but this sample + module is not in CPAN. So, you can either use CPAN to install + `Muck::FS::S3`_, or install Amazon's sample S3 module manually. If you go + the manual route, you can remove ``Muck::FS::`` from the example below. + +.. code-block:: perl + + use Muck::FS::S3::QueryStringAuthGenerator; + my $generator = Muck::FS::S3::QueryStringAuthGenerator->new( + $access_key, + $secret_key, + 0, # 0 means use 'http'. set this to 1 for 'https' + 'objects.dreamhost.com', + ); + + my $hello_url = $generator->make_bare_url($bucket->bucket, 'hello.txt'); + print $hello_url . "\n"; + + $generator->expires_in(3600); # 1 hour = 3600 seconds + my $plans_url = $generator->get($bucket->bucket, 'secret_plans.txt'); + print $plans_url . "\n"; + +The output will look something like this:: + + http://objects.dreamhost.com:80/my-bucket-name/hello.txt + http://objects.dreamhost.com:80/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + + +.. _`Amazon::S3`: http://search.cpan.org/~tima/Amazon-S3-0.441/lib/Amazon/S3.pm +.. _`Amazon::S3::Bucket`: http://search.cpan.org/~tima/Amazon-S3-0.441/lib/Amazon/S3/Bucket.pm +.. _`Muck::FS::S3`: http://search.cpan.org/~mike/Muck-0.02/ + diff --git a/doc/radosgw/s3/php.rst b/doc/radosgw/s3/php.rst new file mode 100644 index 000000000..4878a3489 --- /dev/null +++ b/doc/radosgw/s3/php.rst @@ -0,0 +1,214 @@ +.. _php: + +PHP S3 Examples +=============== + +Installing AWS PHP SDK +---------------------- + +This installs AWS PHP SDK using composer (see here_ how to install composer). + +.. _here: https://getcomposer.org/download/ + +.. code-block:: bash + + $ composer install aws/aws-sdk-php + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. note:: + + The client initialization requires a region so we use ``''``. + +.. code-block:: php + + <?php + + use Aws\S3\S3Client; + + define('AWS_KEY', 'place access key here'); + define('AWS_SECRET_KEY', 'place secret key here'); + $ENDPOINT = 'http://objects.dreamhost.com'; + + // require the amazon sdk from your composer vendor dir + require __DIR__.'/vendor/autoload.php'; + + // Instantiate the S3 class and point it at the desired host + $client = new S3Client([ + 'region' => '', + 'version' => '2006-03-01', + 'endpoint' => $ENDPOINT, + 'credentials' => [ + 'key' => AWS_KEY, + 'secret' => AWS_SECRET_KEY + ], + // Set the S3 class to use objects.dreamhost.com/bucket + // instead of bucket.objects.dreamhost.com + 'use_path_style_endpoint' => true + ]); + +Listing Owned Buckets +--------------------- +This gets a ``AWS\Result`` instance that is more convenient to visit using array access way. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: php + + <?php + $listResponse = $client->listBuckets(); + $buckets = $listResponse['Buckets']; + foreach ($buckets as $bucket) { + echo $bucket['Name'] . "\t" . $bucket['CreationDate'] . "\n"; + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` and returns a +``AWS\Result`` object. + +.. code-block:: php + + <?php + $client->createBucket(['Bucket' => 'my-new-bucket']); + + +List a Bucket's Content +----------------------- + +This gets a ``AWS\Result`` instance that is more convenient to visit using array access way. +This then prints out each object's name, the file size, and last modified date. + +.. code-block:: php + + <?php + $objectsListResponse = $client->listObjects(['Bucket' => $bucketname]); + $objects = $objectsListResponse['Contents'] ?? []; + foreach ($objects as $object) { + echo $object['Key'] . "\t" . $object['Size'] . "\t" . $object['LastModified'] . "\n"; + } + +.. note:: + + If there are more than 1000 objects in this bucket, + you need to check $objectsListResponse['isTruncated'] + and run again with the name of the last key listed. + Keep doing this until isTruncated is not true. + +The output will look something like this if the bucket has some files:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +This deletes the bucket called ``my-old-bucket`` and returns a +``AWS\Result`` object + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: php + + <?php + $client->deleteBucket(['Bucket' => 'my-old-bucket']); + + +Creating an Object +------------------ + +This creates an object ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: php + + <?php + $client->putObject([ + 'Bucket' => 'my-bucket-name', + 'Key' => 'hello.txt', + 'Body' => "Hello World!" + ]); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable and +``secret_plans.txt`` to be private. + +.. code-block:: php + + <?php + $client->putObjectAcl([ + 'Bucket' => 'my-bucket-name', + 'Key' => 'hello.txt', + 'ACL' => 'public-read' + ]); + $client->putObjectAcl([ + 'Bucket' => 'my-bucket-name', + 'Key' => 'secret_plans.txt', + 'ACL' => 'private' + ]); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: php + + <?php + $client->deleteObject(['Bucket' => 'my-bucket-name', 'Key' => 'goodbye.txt']); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: php + + <?php + $object = $client->getObject(['Bucket' => 'my-bucket-name', 'Key' => 'poetry.pdf']); + file_put_contents('/home/larry/documents/poetry.pdf', $object['Body']->getContents()); + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. +This works because we made ``hello.txt`` public by setting +the ACL above. This then generates a signed download URL +for ``secret_plans.txt`` that will work for 1 hour. +Signed download URLs will work for the time period even +if the object is private (when the time period is up, +the URL will stop working). + +.. code-block:: php + + <?php + $hello_url = $client->getObjectUrl('my-bucket-name', 'hello.txt'); + echo $hello_url."\n"; + + $secret_plans_cmd = $client->getCommand('GetObject', ['Bucket' => 'my-bucket-name', 'Key' => 'secret_plans.txt']); + $request = $client->createPresignedRequest($secret_plans_cmd, '+1 hour'); + echo $request->getUri()."\n"; + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=sandboxAccessKey%2F20190116%2F%2Fs3%2Faws4_request&X-Amz-Date=20190116T125520Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Signature=61921f07c73d7695e47a2192cf55ae030f34c44c512b2160bb5a936b2b48d923 + diff --git a/doc/radosgw/s3/python.rst b/doc/radosgw/s3/python.rst new file mode 100644 index 000000000..35f682893 --- /dev/null +++ b/doc/radosgw/s3/python.rst @@ -0,0 +1,197 @@ +.. _python: + +Python S3 Examples +================== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: python + + import boto + import boto.s3.connection + access_key = 'put your access key here!' + secret_key = 'put your secret key here!' + + conn = boto.connect_s3( + aws_access_key_id = access_key, + aws_secret_access_key = secret_key, + host = 'objects.dreamhost.com', + #is_secure=False, # uncomment if you are not using ssl + calling_format = boto.s3.connection.OrdinaryCallingFormat(), + ) + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: python + + for bucket in conn.get_all_buckets(): + print("{name}\t{created}".format( + name = bucket.name, + created = bucket.creation_date, + )) + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: python + + bucket = conn.create_bucket('my-new-bucket') + + +Listing a Bucket's Content +-------------------------- + +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: python + + for key in bucket.list(): + print("{name}\t{size}\t{modified}".format( + name = key.name, + size = key.size, + modified = key.last_modified, + )) + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: python + + conn.delete_bucket(bucket.name) + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. attention:: + + not available in python + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: python + + key = bucket.new_key('hello.txt') + key.set_contents_from_string('Hello World!') + + +Uploading an Object or a File +----------------------------- + +This creates a file ``logo.png`` with the contents from the file ``"logo.png"`` + +.. code-block:: python + + key = bucket.new_key('logo.png') + key.set_contents_from_filename('logo.png') + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and +``secret_plans.txt`` to be private. + +.. code-block:: python + + hello_key = bucket.get_key('hello.txt') + hello_key.set_canned_acl('public-read') + plans_key = bucket.get_key('secret_plans.txt') + plans_key.set_canned_acl('private') + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: python + + key = bucket.get_key('perl_poetry.pdf') + key.get_contents_to_filename('/home/larry/documents/perl_poetry.pdf') + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: python + + bucket.delete_key('goodbye.txt') + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. code-block:: python + + hello_key = bucket.get_key('hello.txt') + hello_url = hello_key.generate_url(0, query_auth=False, force_http=True) + print(hello_url) + + plans_key = bucket.get_key('secret_plans.txt') + plans_url = plans_key.generate_url(3600, query_auth=True, force_http=True) + print(plans_url) + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + +Using S3 API Extensions +----------------------- + +To use the boto3 client to tests the RadosGW extensions to the S3 API, the `extensions file`_ should be placed under: ``~/.aws/models/s3/2006-03-01/`` directory. +For example, unordered list of objects could be fetched using: + +.. code-block:: python + + print(conn.list_objects(Bucket='my-new-bucket', AllowUnordered=True)) + + +Without the extensions file, in the above example, boto3 would complain that the ``AllowUnordered`` argument is invalid. + + +.. _extensions file: https://github.com/ceph/ceph/blob/main/examples/rgw/boto3/service-2.sdk-extras.json diff --git a/doc/radosgw/s3/ruby.rst b/doc/radosgw/s3/ruby.rst new file mode 100644 index 000000000..435b3c630 --- /dev/null +++ b/doc/radosgw/s3/ruby.rst @@ -0,0 +1,364 @@ +.. _ruby: + +Ruby `AWS::SDK`_ Examples (aws-sdk gem ~>2) +=========================================== + +Settings +--------------------- + +You can setup the connection on global way: + +.. code-block:: ruby + + Aws.config.update( + endpoint: 'https://objects.dreamhost.com.', + access_key_id: 'my-access-key', + secret_access_key: 'my-secret-key', + force_path_style: true, + region: 'us-east-1' + ) + + +and instantiate a client object: + +.. code-block:: ruby + + s3_client = Aws::S3::Client.new + +Listing Owned Buckets +--------------------- + +This gets a list of buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: ruby + + s3_client.list_buckets.buckets.each do |bucket| + puts "#{bucket.name}\t#{bucket.creation_date}" + end + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: ruby + + s3_client.create_bucket(bucket: 'my-new-bucket') + +If you want a private bucket: + +`acl` option accepts: # private, public-read, public-read-write, authenticated-read + +.. code-block:: ruby + + s3_client.create_bucket(bucket: 'my-new-bucket', acl: 'private') + + +Listing a Bucket's Content +-------------------------- + +This gets a list of hashes with the contents of each object +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: ruby + + s3_client.get_objects(bucket: 'my-new-bucket').contents.each do |object| + puts "#{object.key}\t#{object.size}\t#{object.last-modified}" + end + +The output will look something like this if the bucket has some files:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: ruby + + s3_client.delete_bucket(bucket: 'my-new-bucket') + + +Forced Delete for Non-empty Buckets +----------------------------------- +First, you need to clear the bucket: + +.. code-block:: ruby + + Aws::S3::Bucket.new('my-new-bucket', client: s3_client).clear! + +after, you can destroy the bucket + +.. code-block:: ruby + + s3_client.delete_bucket(bucket: 'my-new-bucket') + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: ruby + + s3_client.put_object( + key: 'hello.txt', + body: 'Hello World!', + bucket: 'my-new-bucket', + content_type: 'text/plain' + ) + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and ``secret_plans.txt`` +to be private. + +.. code-block:: ruby + + s3_client.put_object_acl(bucket: 'my-new-bucket', key: 'hello.txt', acl: 'public-read') + + s3_client.put_object_acl(bucket: 'my-new-bucket', key: 'private.txt', acl: 'private') + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: ruby + + s3_client.get_object(bucket: 'my-new-bucket', key: 'poetry.pdf', response_target: '/home/larry/documents/poetry.pdf') + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: ruby + + s3_client.delete_object(key: 'goodbye.txt', bucket: 'my-new-bucket') + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. code-block:: ruby + + puts Aws::S3::Object.new( + key: 'hello.txt', + bucket_name: 'my-new-bucket', + client: s3_client + ).public_url + + puts Aws::S3::Object.new( + key: 'secret_plans.txt', + bucket_name: 'hermes_ceph_gem', + client: s3_client + ).presigned_url(:get, expires_in: 60 * 60) + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + +.. _`AWS::SDK`: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html + + + +Ruby `AWS::S3`_ Examples (aws-s3 gem) +===================================== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: ruby + + AWS::S3::Base.establish_connection!( + :server => 'objects.dreamhost.com', + :use_ssl => true, + :access_key_id => 'my-access-key', + :secret_access_key => 'my-secret-key' + ) + + +Listing Owned Buckets +--------------------- + +This gets a list of `AWS::S3::Bucket`_ objects that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: ruby + + AWS::S3::Service.buckets.each do |bucket| + puts "#{bucket.name}\t#{bucket.creation_date}" + end + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: ruby + + AWS::S3::Bucket.create('my-new-bucket') + + +Listing a Bucket's Content +-------------------------- + +This gets a list of hashes with the contents of each object +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: ruby + + new_bucket = AWS::S3::Bucket.find('my-new-bucket') + new_bucket.each do |object| + puts "#{object.key}\t#{object.about['content-length']}\t#{object.about['last-modified']}" + end + +The output will look something like this if the bucket has some files:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: ruby + + AWS::S3::Bucket.delete('my-new-bucket') + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. code-block:: ruby + + AWS::S3::Bucket.delete('my-new-bucket', :force => true) + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: ruby + + AWS::S3::S3Object.store( + 'hello.txt', + 'Hello World!', + 'my-new-bucket', + :content_type => 'text/plain' + ) + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and ``secret_plans.txt`` +to be private. + +.. code-block:: ruby + + policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket') + policy.grants = [ AWS::S3::ACL::Grant.grant(:public_read) ] + AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy) + + policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket') + policy.grants = [] + AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy) + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: ruby + + open('/home/larry/documents/poetry.pdf', 'w') do |file| + AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk| + file.write(chunk) + end + end + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: ruby + + AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket') + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. code-block:: ruby + + puts AWS::S3::S3Object.url_for( + 'hello.txt', + 'my-new-bucket', + :authenticated => false + ) + + puts AWS::S3::S3Object.url_for( + 'secret_plans.txt', + 'my-new-bucket', + :expires_in => 60 * 60 + ) + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + +.. _`AWS::S3`: http://amazon.rubyforge.org/ +.. _`AWS::S3::Bucket`: http://amazon.rubyforge.org/doc/ + diff --git a/doc/radosgw/s3/serviceops.rst b/doc/radosgw/s3/serviceops.rst new file mode 100644 index 000000000..54b6ca375 --- /dev/null +++ b/doc/radosgw/s3/serviceops.rst @@ -0,0 +1,69 @@ +Service Operations +================== + +List Buckets +------------ +``GET /`` returns a list of buckets created by the user making the request. ``GET /`` only +returns buckets created by an authenticated user. You cannot make an anonymous request. + +Syntax +~~~~~~ +:: + + GET / HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~ + ++----------------------------+-------------+-----------------------------------------------------------------+ +| Name | Type | Description | ++============================+=============+=================================================================+ +| ``Buckets`` | Container | Container for list of buckets. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``Bucket`` | Container | Container for bucket information. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``Name`` | String | Bucket name. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``CreationDate`` | Date | UTC time when the bucket was created. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``ListAllMyBucketsResult`` | Container | A container for the result. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``Owner`` | Container | A container for the bucket owner's ``ID`` and ``DisplayName``. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``ID`` | String | The bucket owner's ID. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``DisplayName`` | String | The bucket owner's display name. | ++----------------------------+-------------+-----------------------------------------------------------------+ + + +Get Usage Stats +--------------- + +Gets usage stats per user, similar to the admin command :ref:`rgw_user_usage_stats`. + +Syntax +~~~~~~ +:: + + GET /?usage HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~ + ++----------------------------+-------------+-----------------------------------------------------------------+ +| Name | Type | Description | ++============================+=============+=================================================================+ +| ``Summary`` | Container | Summary of total stats by user. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``TotalBytes`` | Integer | Bytes used by user | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``TotalBytesRounded`` | Integer | Bytes rounded to the nearest 4k boundary | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``TotalEntries`` | Integer | Total object entries | ++----------------------------+-------------+-----------------------------------------------------------------+ diff --git a/doc/radosgw/s3select.rst b/doc/radosgw/s3select.rst new file mode 100644 index 000000000..d46d4e96f --- /dev/null +++ b/doc/radosgw/s3select.rst @@ -0,0 +1,796 @@ +=============== + Ceph s3 select +=============== + +.. contents:: + +Overview +-------- + +The **S3 Select** engine creates an efficient pipe between clients and Ceph +back end nodes. The S3 Select engine works best when implemented as closely as +possible to back end storage. + +The S3 Select engine makes it possible to use an SQL-like syntax to select a +restricted subset of data stored in an S3 object. The S3 Select engine +facilitates the use of higher level, analytic applications (for example: +SPARK-SQL). The ability of the S3 Select engine to target a proper subset of +structed data within an S3 object decreases latency and increases throughput. + +For example: assume that a user needs to extract a single column that is +filtered by another column, and that these colums are stored in a CSV file in +an S3 object that is several GB in size. The following query performs this +extraction: ``select customer-id from s3Object where age>30 and age<65;`` + +Without the use of S3 Select, the whole S3 object must be retrieved from an OSD +via RGW before the data is filtered and extracted. Significant network and CPU +overhead are saved by "pushing down" the query into radosgw. + +**The bigger the object and the more accurate the query, +the better the performance of s3select**. + +Basic Workflow +-------------- + +S3 Select queries are sent to RGW via `AWS-CLI +<https://docs.aws.amazon.com/cli/latest/reference/s3api/select-object-content.html>`_ + +S3 Select passes the authentication and permission parameters as an incoming +message (POST). ``RGWSelectObj_ObjStore_S3::send_response_data`` is the entry +point and handles each fetched chunk according to the object key that was +input. ``send_response_data`` is the first to handle the input query: it +extracts the query and other CLI parameters. + +RGW executes an S3 Select query on each new fetched chunk (up to 4 MB). The +current implementation supports CSV objects. CSV rows are sometimes "cut" in +the middle by the limits of the chunks, and those broken-lines (the first or +last per chunk) are skipped while processing the query. Such broken lines are +stored and later merged with the next broken line (which belongs to the next +chunk), and only then processed. + +For each processed chunk, an output message is formatted according to `aws +specification +<https://docs.aws.amazon.com/amazons3/latest/api/archive-restobjectselectcontent.html#archive-restobjectselectcontent-responses>`_ +and sent back to the client. RGW supports the following response: +``{:event-type,records} {:content-type,application/octet-stream} +{:message-type,event}``. For aggregation queries, the last chunk should be +identified as the end of input. + + +Basic Functionalities +~~~~~~~~~~~~~~~~~~~~~ + +**S3select** has a definite set of functionalities compliant with AWS. + +The implemented software architecture supports basic arithmetic expressions, +logical and compare expressions, including nested function calls and casting +operators, which enables the user great flexibility. + +review the below s3-select-feature-table_. + + +Error Handling +~~~~~~~~~~~~~~ + +Upon an error being detected, RGW returns 400-Bad-Request and a specific error message sends back to the client. +Currently, there are 2 main types of error. + +**Syntax error**: the s3select parser rejects user requests that are not aligned with parser syntax definitions, as +described in this documentation. +Upon Syntax Error, the engine creates an error message that points to the location of the error. +RGW sends back the error message in a specific error response. + +**Processing Time error**: the runtime engine may detect errors that occur only on processing time, for that type of +error, a different error message would describe that. +RGW sends back the error message in a specific error response. + +.. _s3-select-feature-table: + +Features Support +---------------- + +Currently only part of `AWS select command +<https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference-select.html>`_ +is implemented, table below describes what is currently supported. + +The following table describes the current implementation for s3-select +functionalities: + ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Feature | Detailed | Example / Description | ++=================================+=================+=======================================================================+ +| Arithmetic operators | ^ * % / + - ( ) | select (int(_1)+int(_2))*int(_9) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| | ``%`` modulo | select count(*) from s3object where cast(_1 as int)%2 = 0; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| | ``^`` power-of | select cast(2^10 as int) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Compare operators | > < >= <= = != | select _1,_2 from s3object where (int(_1)+int(_3))>int(_5); | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | AND OR NOT | select count(*) from s3object where not (int(_1)>123 and int(_5)<200);| ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | is null | return true/false for null indication in expression | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | is not null | return true/false for null indication in expression | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator and NULL | unknown state | review null-handle_ observe how logical operator result with null. | +| | | the following query return **0**. | +| | | | +| | | select count(*) from s3object where null and (3>2); | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Arithmetic operator with NULL | unknown state | review null-handle_ observe the results of binary operations with NULL| +| | | the following query return **0**. | +| | | | +| | | select count(*) from s3object where (null+1) and (3>2); | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| compare with NULL | unknown state | review null-handle_ observe results of compare operations with NULL | +| | | the following query return **0**. | +| | | | +| | | select count(*) from s3object where (null*1.5) != 3; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| missing column | unknown state | select count(*) from s3object where _1 is null; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| query is filtering rows where predicate | select count(*) from s3object where (_1 > 12 and _2 = 0) is not null; | +| is returning non null results. | | +| this predicate will return null | | +| upon _1 or _2 is null | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| projection column | similar to | select case | +| | switch/case | cast(_1 as int) + 1 | +| | default | when 2 then "a" | +| | | when 3 then "b" | +| | | else "c" end from s3object; | +| | | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| projection column | similar to | select case | +| | if/then/else | when (1+1=(2+1)*3) then 'case_1' | +| | | when ((4*3)=(12)) then 'case_2' | +| | | else 'case_else' end, | +| | | age*2 from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | ``coalesce {expression,expression ...} :: return first non-null argument`` | +| | | +| | select coalesce(nullif(5,5),nullif(1,1.0),age+12) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | ``nullif {expr1,expr2} ::return null in case both arguments are equal,`` | +| | ``or else the first one`` | +| | | +| | select nullif(cast(_1 as int),cast(_2 as int)) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | ``{expression} in ( .. {expression} ..)`` | +| | | +| | select count(*) from s3object | +| | where 'ben' in (trim(_5),substring(_1,char_length(_1)-3,3),last_name); | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | ``{expression} between {expression} and {expression}`` | +| | | +| | select count(*) from s3object | +| | where substring(_3,char_length(_3),1) between "x" and trim(_1) | +| | and substring(_3,char_length(_3)-1,1) = ":"; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| logical operator | ``{expression} like {match-pattern}`` | +| | | +| | select count(*) from s3object where first_name like '%de_'; | +| | | +| | select count(*) from s3object where _1 like \"%a[r-s]\; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| | ``{expression} like {match-pattern} escape {char}`` | +| | | +| logical operator | select count(*) from s3object where "jok_ai" like "%#_ai" escape "#"; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| true / false | select (cast(_1 as int)>123 = true) from s3object | +| predicate as a projection | where address like '%new-york%'; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| an alias to | select (_1 like "_3_") as *likealias*,_1 from s3object | +| predicate as a projection | where *likealias* = true and cast(_1 as int) between 800 and 900; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| casting operator | select cast(123 as int)%2 from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| casting operator | select cast(123.456 as float)%2 from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| casting operator | select cast('ABC0-9' as string),cast(substr('ab12cd',3,2) as int)*4 from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| casting operator | select cast(5 as bool) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| casting operator | select cast(substring('publish on 2007-01-01',12,10) as timestamp) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| non AWS casting operator | select int(_1),int( 1.2 + 3.4) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| non AWS casting operator | select float(1.2) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| not AWS casting operator | select to_timestamp('1999-10-10T12:23:44Z') from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Aggregation Function | sum | select sum(int(_1)) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Aggregation Function | avg | select avg(cast(_1 a float) + cast(_2 as int)) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Aggregation Function | min | select min( int(_1) * int(_5) ) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Aggregation Function | max | select max(float(_1)),min(int(_5)) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Aggregation Function | count | select count(*) from s3object where (int(_1)+int(_3))>int(_5); | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Timestamp Functions | extract | select count(*) from s3object where | +| | | extract(year from to_timestamp(_2)) > 1950 | +| | | and extract(year from to_timestamp(_1)) < 1960; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Timestamp Functions | date_add | select count(0) from s3object where | +| | | date_diff(year,to_timestamp(_1),date_add(day,366, | +| | | to_timestamp(_1))) = 1; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Timestamp Functions | date_diff | select count(0) from s3object where | +| | | date_diff(month,to_timestamp(_1),to_timestamp(_2))) = 2; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Timestamp Functions | utcnow | select count(0) from s3object where | +| | | date_diff(hours,utcnow(),date_add(day,1,utcnow())) = 24; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Timestamp Functions | to_string | select to_string( | +| | | to_timestamp("2009-09-17T17:56:06.234567Z"), | +| | | "yyyyMMdd-H:m:s") from s3object; | +| | | | +| | | ``result: "20090917-17:56:6"`` | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | substring | select count(0) from s3object where | +| | | int(substring(_1,1,4))>1950 and int(substring(_1,1,4))<1960; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| substring with ``from`` negative number is valid | select substring("123456789" from -4) from s3object; | +| considered as first | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| substring with ``from`` zero ``for`` out-of-bound | select substring("123456789" from 0 for 100) from s3object; | +| number is valid just as (first,last) | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | trim | select trim(' foobar ') from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | trim | select trim(trailing from ' foobar ') from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | trim | select trim(leading from ' foobar ') from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | trim | select trim(both '12' from '1112211foobar22211122') from s3objects; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | lower/upper | select lower('ABcD12#$e') from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| String Functions | char_length | select count(*) from s3object where char_length(_3)=3; | +| | character_length| | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Complex queries | select sum(cast(_1 as int)), | +| | max(cast(_3 as int)), | +| | substring('abcdefghijklm',(2-1)*3+sum(cast(_1 as int))/sum(cast(_1 as int))+1, | +| | (count() + count(0))/count(0)) from s3object; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| alias support | | select int(_1) as a1, int(_2) as a2 , (a1+a2) as a3 | +| | | from s3object where a3>100 and a3<300; | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ + +.. _null-handle: + +NULL +~~~~ +NULL is a legit value in ceph-s3select systems similar to other DB systems, i.e. systems needs to handle the case where a value is NULL. + +The definition of NULL in our context, is missing/unknown, in that sense **NULL can not produce a value on ANY arithmetic operations** ( a + NULL will produce NULL value). + +The Same is with arithmetic comparison, **any comparison to NULL is NULL**, i.e. unknown. +Below is a truth table contains the NULL use-case. + ++---------------------------------+-----------------------------+ +| A is NULL | Result (NULL=UNKNOWN) | ++=================================+=============================+ +| NOT A | NULL | ++---------------------------------+-----------------------------+ +| A OR False | NULL | ++---------------------------------+-----------------------------+ +| A OR True | True | ++---------------------------------+-----------------------------+ +| A OR A | NULL | ++---------------------------------+-----------------------------+ +| A AND False | False | ++---------------------------------+-----------------------------+ +| A AND True | NULL | ++---------------------------------+-----------------------------+ +| A and A | NULL | ++---------------------------------+-----------------------------+ + +S3-select Function Interfaces +----------------------------- + +Timestamp Functions +~~~~~~~~~~~~~~~~~~~ +The timestamp functionalities as described in `AWS-specs +<https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference-date.html>`_ +is fully implemented. + + ``to_timestamp( string )`` : The casting operator converts string to timestamp + basic type. to_timestamp operator is able to convert the following + ``YYYY-MM-DDTHH:mm:ss.SSSSSS+/-HH:mm`` , ``YYYY-MM-DDTHH:mm:ss.SSSSSSZ`` , + ``YYYY-MM-DDTHH:mm:ss+/-HH:mm`` , ``YYYY-MM-DDTHH:mm:ssZ`` , + ``YYYY-MM-DDTHH:mm+/-HH:mm`` , ``YYYY-MM-DDTHH:mmZ`` , ``YYYY-MM-DDT`` or + ``YYYYT`` string formats into timestamp. Where time (or part of it) is + missing in the string format, zero's are replacing the missing parts. And for + missing month and day, 1 is default value for them. Timezone part is in + format ``+/-HH:mm`` or ``Z`` , where the letter "Z" indicates Coordinated + Universal Time (UTC). Value of timezone can range between -12:00 and +14:00. + + ``extract(date-part from timestamp)`` : The function extracts date-part from + input timestamp and returns it as integer. Supported date-part : year, month, + week, day, hour, minute, second, timezone_hour, timezone_minute. + + ``date_add(date-part, quantity, timestamp)`` : The function adds quantity + (integer) to date-part of timestamp and returns result as timestamp. It also + includes timezone in calculation. Supported data-part : year, month, day, + hour, minute, second. + + ``date_diff(date-part, timestamp, timestamp)`` : The function returns an + integer, a calculated result for difference between 2 timestamps according to + date-part. It includes timezone in calculation. supported date-part : year, + month, day, hour, minute, second. + + ``utcnow()`` : return timestamp of current time. + + ``to_string(timestamp, format_pattern)`` : returns a string representation of + the input timestamp in the given input string format. + +to_string parameters +~~~~~~~~~~~~~~~~~~~~ + ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| Format | Example | Description | ++==============+=================+===================================================================================+ +| yy | 69 | 2-digit year | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| y | 1969 | 4-digit year | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| yyyy | 1969 | Zero-padded 4-digit year | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| M | 1 | Month of year | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| MM | 01 | Zero-padded month of year | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| MMM | Jan | Abbreviated month year name | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| MMMM | January | Full month of year name | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| MMMMM | J | Month of year first letter (NOTE: not valid for use with to_timestamp function) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| d | 2 | Day of month (1-31) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| dd | 02 | Zero-padded day of month (01-31) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| a | AM | AM or PM of day | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| h | 3 | Hour of day (1-12) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| hh | 03 | Zero-padded hour of day (01-12) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| H | 3 | Hour of day (0-23) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| HH | 03 | Zero-padded hour of day (00-23) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| m | 4 | Minute of hour (0-59) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| mm | 04 | Zero-padded minute of hour (00-59) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| s | 5 | Second of minute (0-59) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| ss | 05 | Zero-padded second of minute (00-59) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| S | 0 | Fraction of second (precision: 0.1, range: 0.0-0.9) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| SS | 6 | Fraction of second (precision: 0.01, range: 0.0-0.99) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| SSS | 60 | Fraction of second (precision: 0.001, range: 0.0-0.999) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| SSSSSS | 60000000 | Fraction of second (maximum precision: 1 nanosecond, range: 0.0-0999999999) | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| n | 60000000 | Nano of second | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| X | +07 or Z | Offset in hours or "Z" if the offset is 0 | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| XX or XXXX| +0700 or Z | Offset in hours and minutes or "Z" if the offset is 0 | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| XXX or XXXXX | +07:00 or Z | Offset in hours and minutes or "Z" if the offset is 0 | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| X | 7 | Offset in hours | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| xx or xxxx | 700 | Offset in hours and minutes | ++--------------+-----------------+-----------------------------------------------------------------------------------+ +| xxx or xxxxx | +07:00 | Offset in hours and minutes | ++--------------+-----------------+-----------------------------------------------------------------------------------+ + + +Aggregation Functions +~~~~~~~~~~~~~~~~~~~~~ + +``count()`` : return integer according to number of rows matching condition(if such exist). + +``sum(expression)`` : return a summary of expression per all rows matching condition(if such exist). + +``avg(expression)`` : return a average of expression per all rows matching condition(if such exist). + +``max(expression)`` : return the maximal result for all expressions matching condition(if such exist). + +``min(expression)`` : return the minimal result for all expressions matching condition(if such exist). + +String Functions +~~~~~~~~~~~~~~~~ + +``substring(string,from,to)`` : substring( string ``from`` start [ ``for`` length ] ) +return a string extract from input string according to from,to inputs. +``substring(string from )`` +``substring(string from for)`` + +``char_length`` : return a number of characters in string (``character_length`` does the same). + +``trim`` : trim ( [[``leading`` | ``trailing`` | ``both`` remove_chars] ``from``] string ) +trims leading/trailing(or both) characters from target string, the default is blank character. + +``upper\lower`` : converts characters into lowercase/uppercase. + +SQL Limit Operator +~~~~~~~~~~~~~~~~~~ + +The SQL LIMIT operator is used to limit the number of rows processed by the query. +Upon reaching the limit set by the user, the RGW stops fetching additional chunks. +TODO : add examples, for aggregation and non-aggregation queries. + +Alias +~~~~~ +**Alias** programming-construct is an essential part of s3-select language, it enables much better programming especially with objects containing many columns or in the case of complex queries. + +Upon parsing the statement containing alias construct, it replaces alias with reference to correct projection column, on query execution time the reference is evaluated as any other expression. + +There is a risk that self(or cyclic) reference may occur causing stack-overflow(endless-loop), for that concern upon evaluating an alias, it is validated for cyclic reference. + +Alias also maintains a result cache, meaning that successive uses of a given alias do not evaluate the expression again. The result is instead returned from the cache. + +With each new row the cache is invalidated as the results may then differ. + +Testing +~~~~~~~ + +``s3select`` contains several testing frameworks which provide a large coverage for its functionalities. + +(1) Tests comparison against a trusted engine, meaning, C/C++ compiler is a trusted expression evaluator, +since the syntax for arithmetical and logical expressions are identical (s3select compare to C) +the framework runs equal expressions and validates their results. +A dedicated expression generator produces different sets of expressions per each new test session. + +(2) Compares results of queries whose syntax is different but which are semantically equivalent. +This kind of test validates that different runtime flows produce an identical result +on each run with a different, random dataset. + +For example, on a dataset which contains a random numbers(1-1000) +the following queries will produce identical results. +``select count(*) from s3object where char_length(_3)=3;`` +``select count(*) from s3object where cast(_3 as int)>99 and cast(_3 as int)<1000;`` + +(3) Constant dataset, the conventional way of testing. A query is processing a constant dataset, its result is validated against constant results. + +Additional Syntax Support +~~~~~~~~~~~~~~~~~~~~~~~~~ + +S3select syntax supports table-alias ``select s._1 from s3object s where s._2 = ‘4’;`` + +S3select syntax supports case insensitive ``Select SUM(Cast(_1 as int)) FROM S3Object;`` + +S3select syntax supports statements without closing semicolon ``select count(*) from s3object`` + + +Sending Query to RGW +-------------------- + +Any HTTP client can send an ``s3-select`` request to RGW, which must be compliant with `AWS Request syntax <https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html#API_SelectObjectContent_RequestSyntax>`_. + + + +When sending an ``s3-select`` request to RGW using AWS CLI, clients must follow `AWS command reference <https://docs.aws.amazon.com/cli/latest/reference/s3api/select-object-content.html>`_. +Below is an example: + +:: + + aws --endpoint-url http://localhost:8000 s3api select-object-content + --bucket {BUCKET-NAME} + --expression-type 'SQL' + --scan-range '{"Start" : 1000, "End" : 1000000}' + --input-serialization + '{"CSV": {"FieldDelimiter": "," , "QuoteCharacter": "\"" , "RecordDelimiter" : "\n" , "QuoteEscapeCharacter" : "\\" , "FileHeaderInfo": "USE" }, "CompressionType": "NONE"}' + --output-serialization '{"CSV": {"FieldDelimiter": ":", "RecordDelimiter":"\t", "QuoteFields": "ALWAYS"}}' + --key {OBJECT-NAME} + --request-progress '{"Enabled": True}' + --expression "select count(0) from s3object where int(_1)<10;" output.csv + +Input Serialization +~~~~~~~~~~~~~~~~~~~ + +**FileHeaderInfo** -> (string) +Describes the first line of input. Valid values are: + +**NONE** : The first line is not a header. +**IGNORE** : The first line is a header, but you can't use the header values to indicate the column in an expression. +it's possible to use column position (such as _1, _2, …) to indicate the column (``SELECT s._1 FROM S3OBJECT s``). +**USE** : First line is a header, and you can use the header value to identify a column in an expression (``SELECT column_name FROM S3OBJECT``). + +**QuoteEscapeCharacter** -> (string) +A single character used for escaping the quotation mark character inside an already escaped value. + +**RecordDelimiter** -> (string) +A single character is used to separate individual records in the input. Instead of the default value, you can specify an arbitrary delimiter. + +**FieldDelimiter** -> (string) +A single character is used to separate individual fields in a record. You can specify an arbitrary delimiter. + +Output Serialization +~~~~~~~~~~~~~~~~~~~~ + +**AWS CLI example** + + aws s3api select-object-content \ + --bucket "mybucket" \ + --key keyfile1 \ + --expression "SELECT * FROM s3object s" \ + --expression-type 'SQL' \ + --request-progress '{"Enabled": false}' \ + --input-serialization '{"CSV": {"FieldDelimiter": ","}, "CompressionType": "NONE"}' \ + --output-serialization '{"CSV": {"FieldDelimiter": ":", "RecordDelimiter":"\\t", "QuoteFields": "ALWAYS"}}' /dev/stdout + + **QuoteFields** -> (string) + Indicates whether to use quotation marks around output fields. + **ALWAYS**: Always use quotation marks for output fields. + **ASNEEDED** (not implemented): Use quotation marks for output fields when needed. + + **RecordDelimiter** -> (string) + A single character is used to separate individual records in the output. Instead of the default value, you can specify an + arbitrary delimiter. + + **FieldDelimiter** -> (string) + The value used to separate individual fields in a record. You can specify an arbitrary delimiter. + +Scan Range Option +~~~~~~~~~~~~~~~~~ + + The scan range option to AWS-CLI enables the client to scan and process only a selected part of the object. + This option reduces input/output operations and bandwidth by skipping parts of the object that are not of interest. + TODO : different data-sources (CSV, JSON, Parquet) + +CSV Parsing Behavior +-------------------- + + The ``s3-select`` engine contains a CSV parser, which parses s3-objects as follows. + - Each row ends with ``row-delimiter``. + - ``field-separator`` separates adjacent columns, successive instances of ``field separator`` define a NULL column. + - ``quote-character`` overrides ``field separator``, meaning that ``field separator`` is treated like any character between quotes. + - ``escape character`` disables interpretation of special characters, except for ``row delimiter``. + + Below are examples of CSV parsing rules. + ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Feature | Description | input ==> tokens | ++=================================+=================+=======================================================================+ +| NULL | successive | ,,1,,2, ==> {null}{null}{1}{null}{2}{null} | +| | field delimiter | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| QUOTE | quote character | 11,22,"a,b,c,d",last ==> {11}{22}{"a,b,c,d"}{last} | +| | overrides | | +| | field delimiter | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| Escape | escape char | 11,22,str=\\"abcd\\"\\,str2=\\"123\\",last | +| | overrides | ==> {11}{22}{str="abcd",str2="123"}{last} | +| | meta-character. | | +| | escape removed | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| row delimiter | no close quote, | 11,22,a="str,44,55,66 | +| | row delimiter is| ==> {11}{22}{a="str,44,55,66} | +| | closing line | | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ +| csv header info | FileHeaderInfo | "**USE**" value means each token on first line is column-name, | +| | tag | "**IGNORE**" value means to skip the first line | ++---------------------------------+-----------------+-----------------------------------------------------------------------+ + +JSON +-------------------- + +A JSON reader has been integrated with the ``s3select-engine``, which allows the client to use SQL statements to scan and extract information from JSON documents. +It should be noted that the data readers and parsers for CSV, Parquet, and JSON documents are separated from the SQL engine itself, so all of these readers use the same SQL engine. + +It's important to note that values in a JSON document can be nested in various ways, such as within objects or arrays. +These objects and arrays can be nested within each other without any limitations. +When using SQL to query a specific value in a JSON document, the client must specify the location of the value +via a path in the SELECT statement. + +The SQL engine processes the SELECT statement in a row-based fashion. +It uses the columns specified in the statement to perform its projection calculation, and each row contains values for these columns. +In other words, the SQL engine processes each row one at a time (and aggregates results), using the values in the columns to perform SQL calculations. +However, the generic structure of a JSON document does not have a row-and-column structure like CSV or Parquet. +Instead, it is the SQL statement itself that defines the rows and columns when querying a JSON document. + +When querying JSON documents using SQL, the FROM clause in the SELECT statement defines the row boundaries. +A row in a JSON document should be similar to how the row delimiter is used to define rows when querying CSV objects, and how row groups are used to define rows when querying Parquet objects. +The statement "SELECT ... FROM s3object[*].aaa.bb.cc" instructs the reader to search for the path "aaa.bb.cc" and defines the row boundaries based on the occurrence of this path. +A row begins when the reader encounters the path, and it ends when the reader exits the innermost part of the path, which in this case is the object "cc". + +NOTE : The semantics of querying JSON document may change and may not be the same as the current methodology described. + +TODO : relevant example for object and array values. + +A JSON Query Example +-------------------- + +:: + + { + "firstName": "Joe", + "lastName": "Jackson", + "gender": "male", + "age": "twenty", + "address": { + "streetAddress": "101", + "city": "San Diego", + "state": "CA" + }, + + "firstName": "Joe_2", + "lastName": "Jackson_2", + "gender": "male", + "age": 21, + "address": { + "streetAddress": "101", + "city": "San Diego", + "state": "CA" + }, + + "phoneNumbers": [ + { "type": "home1", "number": "734928_1","addr": 11 }, + { "type": "home2", "number": "734928_2","addr": 22 }, + { "type": "home3", "number": "734928_3","addr": 33 }, + { "type": "home4", "number": "734928_4","addr": 44 }, + { "type": "home5", "number": "734928_5","addr": 55 }, + { "type": "home6", "number": "734928_6","addr": 66 }, + { "type": "home7", "number": "734928_7","addr": 77 }, + { "type": "home8", "number": "734928_8","addr": 88 }, + { "type": "home9", "number": "734928_9","addr": 99 }, + { "type": "home10", "number": "734928_10","addr": 100 } + ], + + "key_after_array": "XXX", + + "description" : { + "main_desc" : "value_1", + "second_desc" : "value_2" + } + } + + # the from-clause define a single row. + # _1 points to root object level. + # _1.age appears twice in Documnet-row, the last value is used for the operation. + query = "select _1.firstname,_1.key_after_array,_1.age+4,_1.description.main_desc,_1.description.second_desc from s3object[*];"; + expected_result = Joe_2,XXX,25,value_1,value_2 + + + # the from-clause points the phonenumbers array (it defines the _1) + # each element in phoneNumbers array define a row. + # in this case each element is an object contains 3 keys/values. + # the query "can not access" values outside phonenumbers array, the query can access only values appears on _1.phonenumbers path. + query = "select cast(substring(_1.number,1,6) as int) *10 from s3object[*].phonenumbers where _1.type='home2';"; + expected_result = 7349280 + + +BOTO3 +----- + +using BOTO3 is "natural" and easy due to AWS-cli support. + +:: + + import pprint + + def run_s3select(bucket,key,query,column_delim=",",row_delim="\n",quot_char='"',esc_char='\\',csv_header_info="NONE"): + + s3 = boto3.client('s3', + endpoint_url=endpoint, + aws_access_key_id=access_key, + region_name=region_name, + aws_secret_access_key=secret_key) + + result = "" + try: + r = s3.select_object_content( + Bucket=bucket, + Key=key, + ExpressionType='SQL', + InputSerialization = {"CSV": {"RecordDelimiter" : row_delim, "FieldDelimiter" : column_delim,"QuoteEscapeCharacter": esc_char, "QuoteCharacter": quot_char, "FileHeaderInfo": csv_header_info}, "CompressionType": "NONE"}, + OutputSerialization = {"CSV": {}}, + Expression=query, + RequestProgress = {"Enabled": progress}) + + except ClientError as c: + result += str(c) + return result + + for event in r['Payload']: + if 'Records' in event: + result = "" + records = event['Records']['Payload'].decode('utf-8') + result += records + if 'Progress' in event: + print("progress") + pprint.pprint(event['Progress'],width=1) + if 'Stats' in event: + print("Stats") + pprint.pprint(event['Stats'],width=1) + if 'End' in event: + print("End") + pprint.pprint(event['End'],width=1) + + return result + + + + + run_s3select( + "my_bucket", + "my_csv_object", + "select int(_1) as a1, int(_2) as a2 , (a1+a2) as a3 from s3object where a3>100 and a3<300;") + + +S3 SELECT Responses +------------------- + +Error Response +~~~~~~~~~~~~~~ + +:: + + <?xml version="1.0" encoding="UTF-8"?> + <Error> + <Code>NoSuchKey</Code> + <Message>The resource you requested does not exist</Message> + <Resource>/mybucket/myfoto.jpg</Resource> + <RequestId>4442587FB7D0A2F9</RequestId> + </Error> + +Report Response +~~~~~~~~~~~~~~~ +:: + + HTTP/1.1 200 + <?xml version="1.0" encoding="UTF-8"?> + <Payload> + <Records> + <Payload>blob</Payload> + </Records> + <Stats> + <Details> + <BytesProcessed>long</BytesProcessed> + <BytesReturned>long</BytesReturned> + <BytesScanned>long</BytesScanned> + </Details> + </Stats> + <Progress> + <Details> + <BytesProcessed>long</BytesProcessed> + <BytesReturned>long</BytesReturned> + <BytesScanned>long</BytesScanned> + </Details> + </Progress> + <Cont> + </Cont> + <End> + </End> + </Payload> + +Response Description +~~~~~~~~~~~~~~~~~~~~ + +For CEPH S3 Select, responses can be messages of the following types: + +**Records message**: Can contain a single record, partial records, or multiple records. Depending on the size of the result, a response can contain one or more of these messages. + +**Error message**: Upon an error being detected, RGW returns 400 Bad Request, and a specific error message sends back to the client, according to its type. + +**Continuation message**: Ceph S3 periodically sends this message to keep the TCP connection open. +These messages appear in responses at random. The client must detect the message type and process it accordingly. + +**Progress message**: Ceph S3 periodically sends this message if requested. It contains information about the progress of a query that has started but has not yet been completed. + +**Stats message**: Ceph S3 sends this message at the end of the request. It contains statistics about the query. + +**End message**: Indicates that the request is complete, and no more messages will be sent. You should not assume that request is complete until the client receives an End message. |