diff options
Diffstat (limited to 'doc/radosgw/s3')
| -rw-r--r-- | doc/radosgw/s3/authentication.rst | 231 | ||||
| -rw-r--r-- | doc/radosgw/s3/bucketops.rst | 701 | ||||
| -rw-r--r-- | doc/radosgw/s3/commons.rst | 111 | ||||
| -rw-r--r-- | doc/radosgw/s3/cpp.rst | 337 | ||||
| -rw-r--r-- | doc/radosgw/s3/csharp.rst | 199 | ||||
| -rw-r--r-- | doc/radosgw/s3/java.rst | 212 | ||||
| -rw-r--r-- | doc/radosgw/s3/objectops.rst | 558 | ||||
| -rw-r--r-- | doc/radosgw/s3/perl.rst | 192 | ||||
| -rw-r--r-- | doc/radosgw/s3/php.rst | 208 | ||||
| -rw-r--r-- | doc/radosgw/s3/python.rst | 171 | ||||
| -rw-r--r-- | doc/radosgw/s3/ruby.rst | 364 | ||||
| -rw-r--r-- | doc/radosgw/s3/serviceops.rst | 39 |
12 files changed, 3323 insertions, 0 deletions
diff --git a/doc/radosgw/s3/authentication.rst b/doc/radosgw/s3/authentication.rst new file mode 100644 index 00000000..10143290 --- /dev/null +++ b/doc/radosgw/s3/authentication.rst @@ -0,0 +1,231 @@ +========================= + Authentication and ACLs +========================= + +Requests to the RADOS Gateway (RGW) can be either authenticated or +unauthenticated. RGW assumes unauthenticated requests are sent by an anonymous +user. RGW supports canned ACLs. + +Authentication +-------------- +Authenticating a request requires including an access key and a Hash-based +Message Authentication Code (HMAC) in the request before it is sent to the +RGW server. RGW uses an S3-compatible authentication approach. + +:: + + HTTP/1.1 + PUT /buckets/bucket/object.mpeg + Host: cname.domain.com + Date: Mon, 2 Jan 2012 00:01:01 +0000 + Content-Encoding: mpeg + Content-Length: 9999999 + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +In the foregoing example, replace ``{access-key}`` with the value for your access +key ID followed by a colon (``:``). Replace ``{hash-of-header-and-secret}`` with +a hash of the header string and the secret corresponding to the access key ID. + +To generate the hash of the header string and secret, you must: + +#. Get the value of the header string. +#. Normalize the request header string into canonical form. +#. Generate an HMAC using a SHA-1 hashing algorithm. + See `RFC 2104`_ and `HMAC`_ for details. +#. Encode the ``hmac`` result as base-64. + +To normalize the header into canonical form: + +#. Get all fields beginning with ``x-amz-``. +#. Ensure that the fields are all lowercase. +#. Sort the fields lexicographically. +#. Combine multiple instances of the same field name into a + single field and separate the field values with a comma. +#. Replace white space and line breaks in field values with a single space. +#. Remove white space before and after colons. +#. Append a new line after each field. +#. Merge the fields back into the header. + +Replace the ``{hash-of-header-and-secret}`` with the base-64 encoded HMAC string. + +Authentication against OpenStack Keystone +----------------------------------------- + +In a radosgw instance that is configured with authentication against +OpenStack Keystone, it is possible to use Keystone as an authoritative +source for S3 API authentication. To do so, you must set: + +* the ``rgw keystone`` configuration options explained in :doc:`../keystone`, +* ``rgw s3 auth use keystone = true``. + +In addition, a user wishing to use the S3 API must obtain an AWS-style +access key and secret key. They can do so with the ``openstack ec2 +credentials create`` command:: + + $ openstack --os-interface public ec2 credentials create + +------------+---------------------------------------------------------------------------------------------------------------------------------------------+ + | Field | Value | + +------------+---------------------------------------------------------------------------------------------------------------------------------------------+ + | access | c921676aaabbccdeadbeef7e8b0eeb2c | + | links | {u'self': u'https://auth.example.com:5000/v3/users/7ecbebaffeabbddeadbeefa23267ccbb24/credentials/OS-EC2/c921676aaabbccdeadbeef7e8b0eeb2c'} | + | project_id | 5ed51981aab4679851adeadbeef6ebf7 | + | secret | ******************************** | + | trust_id | None | + | user_id | 7ecbebaffeabbddeadbeefa23267cc24 | + +------------+---------------------------------------------------------------------------------------------------------------------------------------------+ + +The thus-generated access and secret key can then be used for S3 API +access to radosgw. + +.. note:: Consider that most production radosgw deployments + authenticating against OpenStack Keystone are also set up + for :doc:`../multitenancy`, for which special + considerations apply with respect to S3 signed URLs and + public read ACLs. + +Access Control Lists (ACLs) +--------------------------- + +RGW supports S3-compatible ACL functionality. An ACL is a list of access grants +that specify which operations a user can perform on a bucket or on an object. +Each grant has a different meaning when applied to a bucket versus applied to +an object: + ++------------------+--------------------------------------------------------+----------------------------------------------+ +| Permission | Bucket | Object | ++==================+========================================================+==============================================+ +| ``READ`` | Grantee can list the objects in the bucket. | Grantee can read the object. | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``WRITE`` | Grantee can write or delete objects in the bucket. | N/A | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``READ_ACP`` | Grantee can read bucket ACL. | Grantee can read the object ACL. | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``WRITE_ACP`` | Grantee can write bucket ACL. | Grantee can write to the object ACL. | ++------------------+--------------------------------------------------------+----------------------------------------------+ +| ``FULL_CONTROL`` | Grantee has full permissions for object in the bucket. | Grantee can read or write to the object ACL. | ++------------------+--------------------------------------------------------+----------------------------------------------+ + +Internally, S3 operations are mapped to ACL permissions thus: + ++---------------------------------------+---------------+ +| Operation | Permission | ++=======================================+===============+ +| ``s3:GetObject`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectTorrent`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersion`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersionTorrent`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectTagging`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersionTagging`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListAllMyBuckets`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListBucket`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListBucketMultipartUploads`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListBucketVersions`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:ListMultipartUploadParts`` | ``READ`` | ++---------------------------------------+---------------+ +| ``s3:AbortMultipartUpload`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:CreateBucket`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteBucket`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteObject`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:s3DeleteObjectVersion`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:PutObject`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectVersionTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteObjectTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:DeleteObjectVersionTagging`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:RestoreObject`` | ``WRITE`` | ++---------------------------------------+---------------+ +| ``s3:GetAccelerateConfiguration`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketAcl`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketCORS`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketLocation`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketLogging`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketNotification`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketPolicy`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketRequestPayment`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketTagging`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketVersioning`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetBucketWebsite`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetLifecycleConfiguration`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectAcl`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetObjectVersionAcl`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:GetReplicationConfiguration`` | ``READ_ACP`` | ++---------------------------------------+---------------+ +| ``s3:DeleteBucketPolicy`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:DeleteBucketWebsite`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:DeleteReplicationConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutAccelerateConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketAcl`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketCORS`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketLogging`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketNotification`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketPolicy`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketRequestPayment`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketTagging`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutPutBucketVersioning`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutBucketWebsite`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutLifecycleConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectAcl`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutObjectVersionAcl`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ +| ``s3:PutReplicationConfiguration`` | ``WRITE_ACP`` | ++---------------------------------------+---------------+ + +Some mappings, (e.g. ``s3:CreateBucket`` to ``WRITE``) are not +applicable to S3 operation, but are required to allow Swift and S3 to +access the same resources when things like Swift user ACLs are in +play. This is one of the many reasons that you should use S3 bucket +policies rather than S3 ACLs when possible. + + +.. _RFC 2104: http://www.ietf.org/rfc/rfc2104.txt +.. _HMAC: https://en.wikipedia.org/wiki/HMAC diff --git a/doc/radosgw/s3/bucketops.rst b/doc/radosgw/s3/bucketops.rst new file mode 100644 index 00000000..5d770492 --- /dev/null +++ b/doc/radosgw/s3/bucketops.rst @@ -0,0 +1,701 @@ +=================== + Bucket Operations +=================== + +PUT Bucket +---------- +Creates a new bucket. To create a bucket, you must have a user ID and a valid AWS Access Key ID to authenticate requests. You may not +create buckets as an anonymous user. + +Constraints +~~~~~~~~~~~ +In general, bucket names should follow domain name constraints. + +- Bucket names must be unique. +- Bucket names must begin and end with a lowercase letter. +- Bucket names may contain a dash (-). + +Syntax +~~~~~~ + +:: + + PUT /{bucket} HTTP/1.1 + Host: cname.domain.com + x-amz-acl: public-read-write + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Parameters +~~~~~~~~~~ + + ++---------------+----------------------+-----------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++===============+======================+=============================================================================+============+ +| ``x-amz-acl`` | Canned ACLs. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++---------------+----------------------+-----------------------------------------------------------------------------+------------+ +| ``x-amz-bucket-object-lock-enabled`` | Enable object lock on bucket. | ``true``, ``false`` | No | ++--------------------------------------+-------------------------------+---------------------------------------------+------------+ + +Request Entities +~~~~~~~~~~~~~~~~ + ++-------------------------------+-----------+----------------------------------------------------------------+ +| Name | Type | Description | ++===============================+===========+================================================================+ +| ``CreateBucketConfiguration`` | Container | A container for the bucket configuration. | ++-------------------------------+-----------+----------------------------------------------------------------+ +| ``LocationConstraint`` | String | A zonegroup api name, with optional :ref:`s3_bucket_placement` | ++-------------------------------+-----------+----------------------------------------------------------------+ + + +HTTP Response +~~~~~~~~~~~~~ + +If the bucket name is unique, within constraints and unused, the operation will succeed. +If a bucket with the same name already exists and the user is the bucket owner, the operation will succeed. +If the bucket name is already in use, the operation will fail. + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``409`` | BucketAlreadyExists | Bucket already exists under different user's ownership. | ++---------------+-----------------------+----------------------------------------------------------+ + +DELETE Bucket +------------- + +Deletes a bucket. You can reuse bucket names following a successful bucket removal. + +Syntax +~~~~~~ + +:: + + DELETE /{bucket} HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+---------------+------------------+ +| HTTP Status | Status Code | Description | ++===============+===============+==================+ +| ``204`` | No Content | Bucket removed. | ++---------------+---------------+------------------+ + +GET Bucket +---------- +Returns a list of bucket objects. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?max-keys=25 HTTP/1.1 + Host: cname.domain.com + +Parameters +~~~~~~~~~~ + ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=====================+===========+=================================================================================================+ +| ``prefix`` | String | Only returns objects that contain the specified prefix. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``delimiter`` | String | The delimiter between the prefix and the rest of the object name. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``marker`` | String | A beginning index for the list of objects returned. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``max-keys`` | Integer | The maximum number of keys to return. Default is 1000. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ +| ``allow-unordered`` | Boolean | Non-standard extension. Allows results to be returned unordered. Cannot be used with delimiter. | ++---------------------+-----------+-------------------------------------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+---------------+--------------------+ +| HTTP Status | Status Code | Description | ++===============+===============+====================+ +| ``200`` | OK | Buckets retrieved | ++---------------+---------------+--------------------+ + +Bucket Response Entities +~~~~~~~~~~~~~~~~~~~~~~~~ +``GET /{bucket}`` returns a container for buckets with the following fields. + ++------------------------+-----------+----------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+==================================================================================+ +| ``ListBucketResult`` | Entity | The container for the list of objects. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Name`` | String | The name of the bucket whose contents will be returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Prefix`` | String | A prefix for the object keys. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Marker`` | String | A beginning index for the list of objects returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``MaxKeys`` | Integer | The maximum number of keys returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``Delimiter`` | String | If set, objects with the same prefix will appear in the ``CommonPrefixes`` list. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``IsTruncated`` | Boolean | If ``true``, only a subset of the bucket's contents were returned. | ++------------------------+-----------+----------------------------------------------------------------------------------+ +| ``CommonPrefixes`` | Container | If multiple objects contain the same prefix, they will appear in this list. | ++------------------------+-----------+----------------------------------------------------------------------------------+ + +Object Response Entities +~~~~~~~~~~~~~~~~~~~~~~~~ +The ``ListBucketResult`` contains objects, where each object is within a ``Contents`` container. + ++------------------------+-----------+------------------------------------------+ +| Name | Type | Description | ++========================+===========+==========================================+ +| ``Contents`` | Object | A container for the object. | ++------------------------+-----------+------------------------------------------+ +| ``Key`` | String | The object's key. | ++------------------------+-----------+------------------------------------------+ +| ``LastModified`` | Date | The object's last-modified date/time. | ++------------------------+-----------+------------------------------------------+ +| ``ETag`` | String | An MD-5 hash of the object. (entity tag) | ++------------------------+-----------+------------------------------------------+ +| ``Size`` | Integer | The object's size. | ++------------------------+-----------+------------------------------------------+ +| ``StorageClass`` | String | Should always return ``STANDARD``. | ++------------------------+-----------+------------------------------------------+ +| ``Type`` | String | ``Appendable`` or ``Normal``. | ++------------------------+-----------+------------------------------------------+ + +Get Bucket Location +------------------- +Retrieves the bucket's region. The user needs to be the bucket owner +to call this. A bucket can be constrained to a region by providing +``LocationConstraint`` during a PUT request. + +Syntax +~~~~~~ +Add the ``location`` subresource to bucket resource as shown below + +:: + + GET /{bucket}?location HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~~~~~~~~ + ++------------------------+-----------+------------------------------------------+ +| Name | Type | Description | ++========================+===========+==========================================+ +| ``LocationConstraint`` | String | The region where bucket resides, empty | +| | | string for default region | ++------------------------+-----------+------------------------------------------+ + + + +Get Bucket ACL +-------------- +Retrieves the bucket access control list. The user needs to be the bucket +owner or to have been granted ``READ_ACP`` permission on the bucket. + +Syntax +~~~~~~ +Add the ``acl`` subresource to the bucket request as shown below. + +:: + + GET /{bucket}?acl HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the response. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the bucket owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The bucket owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The bucket owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` bucket. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + +PUT Bucket ACL +-------------- +Sets an access control to an existing bucket. The user needs to be the bucket +owner or to have been granted ``WRITE_ACP`` permission on the bucket. + +Syntax +~~~~~~ +Add the ``acl`` subresource to the bucket request as shown below. + +:: + + PUT /{bucket}?acl HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the request. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the bucket owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The bucket owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The bucket owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` bucket. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + +List Bucket Multipart Uploads +----------------------------- + +``GET /?uploads`` returns a list of the current in-progress multipart uploads--i.e., the application initiates a multipart upload, but +the service hasn't completed all the uploads yet. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?uploads HTTP/1.1 + +Parameters +~~~~~~~~~~ + +You may specify parameters for ``GET /{bucket}?uploads``, but none of them are required. + ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+======================================================================================+ +| ``prefix`` | String | Returns in-progress uploads whose keys contains the specified prefix. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``delimiter`` | String | The delimiter between the prefix and the rest of the object name. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``key-marker`` | String | The beginning marker for the list of uploads. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``max-keys`` | Integer | The maximum number of in-progress uploads. The default is 1000. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``max-uploads`` | Integer | The maximum number of multipart uploads. The range from 1-1000. The default is 1000. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ +| ``upload-id-marker`` | String | Ignored if ``key-marker`` is not specified. Specifies the ``ID`` of first | +| | | upload to list in lexicographical order at or following the ``ID``. | ++------------------------+-----------+--------------------------------------------------------------------------------------+ + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=========================================+=============+==========================================================================================================+ +| ``ListMultipartUploadsResult`` | Container | A container for the results. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ListMultipartUploadsResult.Prefix`` | String | The prefix specified by the ``prefix`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Bucket`` | String | The bucket that will receive the bucket contents. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``KeyMarker`` | String | The key marker specified by the ``key-marker`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadIdMarker`` | String | The marker specified by the ``upload-id-marker`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``NextKeyMarker`` | String | The key marker to use in a subsequent request if ``IsTruncated`` is ``true``. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``NextUploadIdMarker`` | String | The upload ID marker to use in a subsequent request if ``IsTruncated`` is ``true``. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``MaxUploads`` | Integer | The max uploads specified by the ``max-uploads`` request parameter. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Delimiter`` | String | If set, objects with the same prefix will appear in the ``CommonPrefixes`` list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``IsTruncated`` | Boolean | If ``true``, only a subset of the bucket's upload contents were returned. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Upload`` | Container | A container for ``Key``, ``UploadId``, ``InitiatorOwner``, ``StorageClass``, and ``Initiated`` elements. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Key`` | String | The key of the object once the multipart upload is complete. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadId`` | String | The ``ID`` that identifies the multipart upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Initiator`` | Container | Contains the ``ID`` and ``DisplayName`` of the user who initiated the upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The initiator's display name. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ID`` | String | The initiator's ID. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the ``ID`` and ``DisplayName`` of the user who owns the uploaded object. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``StorageClass`` | String | The method used to store the resulting object. ``STANDARD`` or ``REDUCED_REDUNDANCY`` | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Initiated`` | Date | The date and time the user initiated the upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``CommonPrefixes`` | Container | If multiple objects contain the same prefix, they will appear in this list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``CommonPrefixes.Prefix`` | String | The substring of the key after the prefix as defined by the ``prefix`` request parameter. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ + +ENABLE/SUSPEND BUCKET VERSIONING +-------------------------------- + +``PUT /?versioning`` This subresource set the versioning state of an existing bucket. To set the versioning state, you must be the bucket owner. + +You can set the versioning state with one of the following values: + +- Enabled : Enables versioning for the objects in the bucket, All objects added to the bucket receive a unique version ID. +- Suspended : Disables versioning for the objects in the bucket, All objects added to the bucket receive the version ID null. + +If the versioning state has never been set on a bucket, it has no versioning state; a GET versioning request does not return a versioning state value. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?versioning HTTP/1.1 + +REQUEST ENTITIES +~~~~~~~~~~~~~~~~ + ++-----------------------------+-----------+---------------------------------------------------------------------------+ +| Name | Type | Description | ++=============================+===========+===========================================================================+ +| ``VersioningConfiguration`` | Container | A container for the request. | ++-----------------------------+-----------+---------------------------------------------------------------------------+ +| ``Status`` | String | Sets the versioning state of the bucket. Valid Values: Suspended/Enabled | ++-----------------------------+-----------+---------------------------------------------------------------------------+ + +PUT BUCKET OBJECT LOCK +-------------------------------- + +Places an Object Lock configuration on the specified bucket. The rule specified in the Object Lock configuration will be +applied by default to every new object placed in the specified bucket. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?object-lock HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++=============================+=============+========================================================================================+==========+ +| ``ObjectLockConfiguration`` | Container | A container for the request. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``ObjectLockEnabled`` | String | Indicates whether this bucket has an Object Lock configuration enabled. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Rule`` | Container | The Object Lock rule in place for the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``DefaultRetention`` | Container | The default retention period applied to new objects placed in the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Mode`` | String | The default Object Lock retention mode. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Days`` | Integer | The number of days specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Years`` | Integer | The number of years specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ + +HTTP Response +~~~~~~~~~~~~~ + +If the bucket object lock is not enabled when creating the bucket, the operation will fail. + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``400`` | MalformedXML | The XML is not well-formed | ++---------------+-----------------------+----------------------------------------------------------+ +| ``409`` | InvalidBucketState | The bucket object lock is not enabled | ++---------------+-----------------------+----------------------------------------------------------+ + +GET BUCKET OBJECT LOCK +-------------------------------- + +Gets the Object Lock configuration for a bucket. The rule specified in the Object Lock configuration will be applied by +default to every new object placed in the specified bucket. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?object-lock HTTP/1.1 + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++=============================+=============+========================================================================================+==========+ +| ``ObjectLockConfiguration`` | Container | A container for the request. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``ObjectLockEnabled`` | String | Indicates whether this bucket has an Object Lock configuration enabled. | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Rule`` | Container | The Object Lock rule in place for the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``DefaultRetention`` | Container | The default retention period applied to new objects placed in the specified bucket. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Mode`` | String | The default Object Lock retention mode. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Days`` | Integer | The number of days specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ +| ``Years`` | Integer | The number of years specified for the default retention period. | No | ++-----------------------------+-------------+----------------------------------------------------------------------------------------+----------+ + +Create Notification +------------------- + +Create a publisher for a specific bucket into a topic. + +Syntax +~~~~~~ + +:: + + PUT /<bucket name>?notification HTTP/1.1 + + +Request Entities +~~~~~~~~~~~~~~~~ + +Parameters are XML encoded in the body of the request, in the following format: + +:: + + <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <TopicConfiguration> + <Id></Id> + <Topic></Topic> + <Event></Event> + <Filter> + <S3Key> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Key> + <S3Metadata> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Metadata> + <S3Tags> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Tags> + </Filter> + </TopicConfiguration> + </NotificationConfiguration> + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``NotificationConfiguration`` | Container | Holding list of ``TopicConfiguration`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TopicConfiguration`` | Container | Holding ``Id``, ``Topic`` and list of ``Event`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Id`` | String | Name of the notification | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Topic`` | String | Topic ARN. Topic must be created beforehand | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Event`` | String | List of supported events see: `S3 Notification Compatibility`_. Multiple ``Event`` | No | +| | | entities can be used. If omitted, all events are handled | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Filter`` | Container | Holding ``S3Key``, ``S3Metadata`` and ``S3Tags`` entities | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Key`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object key. | No | +| | | At most, 3 entities may be in the list, with ``Name`` be ``prefix``, ``suffix`` or | | +| | | ``regex``. All filter rules in the list must match for the filter to match. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Metadata`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object metadata. | No | +| | | All filter rules in the list must match the metadata defined on the object. However, | | +| | | the object still match if it has other metadata entries not listed in the filter. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Tags`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object tags. | No | +| | | All filter rules in the list must match the tags defined on the object. However, | | +| | | the object still match it it has other tags not listed in the filter. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Key.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be: ``prefix``, ``suffix`` | Yes | +| | | or ``regex``. The ``Value`` would hold the key prefix, key suffix or a regular | | +| | | expression for matching the key, accordingly. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Metadata.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the name of the metadata | Yes | +| | | attribute (e.g. ``x-amz-meta-xxx``). The ``Value`` would be the expected value for | | +| | | this attribute. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Tags.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the tag key, | Yes | +| | | and ``Value`` would be the tag value. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``400`` | MalformedXML | The XML is not well-formed | ++---------------+-----------------------+----------------------------------------------------------+ +| ``400`` | InvalidArgument | Missing Id; Missing/Invalid Topic ARN; Invalid Event | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchKey | The topic does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + + +Delete Notification +------------------- + +Delete a specific, or all, notifications from a bucket. + +.. note:: + + - Notification deletion is an extension to the S3 notification API + - When the bucket is deleted, any notification defined on it is also deleted + - Deleting an unkown notification (e.g. double delete) is not considered an error + +Syntax +~~~~~~ + +:: + + DELETE /bucket?notification[=<notification-id>] HTTP/1.1 + + +Parameters +~~~~~~~~~~ + ++------------------------+-----------+----------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+========================================================================================+ +| ``notification-id`` | String | Name of the notification. If not provided, all notifications on the bucket are deleted | ++------------------------+-----------+----------------------------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + +Get/List Notification +--------------------- + +Get a specific notification, or list all notifications configured on a bucket. + +Syntax +~~~~~~ + +:: + + GET /bucket?notification[=<notification-id>] HTTP/1.1 + + +Parameters +~~~~~~~~~~ + ++------------------------+-----------+----------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+========================================================================================+ +| ``notification-id`` | String | Name of the notification. If not provided, all notifications on the bucket are listed | ++------------------------+-----------+----------------------------------------------------------------------------------------+ + +Response Entities +~~~~~~~~~~~~~~~~~ + +Response is XML encoded in the body of the request, in the following format: + +:: + + <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <TopicConfiguration> + <Id></Id> + <Topic></Topic> + <Event></Event> + <Filter> + <S3Key> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Key> + <S3Metadata> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Metadata> + <S3Tags> + <FilterRule> + <Name></Name> + <Value></Value> + </FilterRule> + </S3Tags> + </Filter> + </TopicConfiguration> + </NotificationConfiguration> + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``NotificationConfiguration`` | Container | Holding list of ``TopicConfiguration`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TopicConfiguration`` | Container | Holding ``Id``, ``Topic`` and list of ``Event`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Id`` | String | Name of the notification | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Topic`` | String | Topic ARN | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Event`` | String | Handled event. Multiple ``Event`` entities may exist | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Filter`` | Container | Holding the filters configured for this notification | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchKey | The notification does not exist (if provided) | ++---------------+-----------------------+----------------------------------------------------------+ + +.. _S3 Notification Compatibility: ../../s3-notification-compatibility diff --git a/doc/radosgw/s3/commons.rst b/doc/radosgw/s3/commons.rst new file mode 100644 index 00000000..ca848bc2 --- /dev/null +++ b/doc/radosgw/s3/commons.rst @@ -0,0 +1,111 @@ +================= + Common Entities +================= + +.. toctree:: + :maxdepth: -1 + +Bucket and Host Name +-------------------- +There are two different modes of accessing the buckets. The first (preferred) method +identifies the bucket as the top-level directory in the URI. :: + + GET /mybucket HTTP/1.1 + Host: cname.domain.com + +The second method identifies the bucket via a virtual bucket host name. For example:: + + GET / HTTP/1.1 + Host: mybucket.cname.domain.com + +To configure virtual hosted buckets, you can either set ``rgw_dns_name = cname.domain.com`` in ceph.conf, or add ``cname.domain.com`` to the list of ``hostnames`` in your zonegroup configuration. See `Ceph Object Gateway - Multisite Configuration`_ for more on zonegroups. + +.. tip:: We prefer the first method, because the second method requires expensive domain certification and DNS wild cards. + +Common Request Headers +---------------------- + ++--------------------+------------------------------------------+ +| Request Header | Description | ++====================+==========================================+ +| ``CONTENT_LENGTH`` | Length of the request body. | ++--------------------+------------------------------------------+ +| ``DATE`` | Request time and date (in UTC). | ++--------------------+------------------------------------------+ +| ``HOST`` | The name of the host server. | ++--------------------+------------------------------------------+ +| ``AUTHORIZATION`` | Authorization token. | ++--------------------+------------------------------------------+ + +Common Response Status +---------------------- + ++---------------+-----------------------------------+ +| HTTP Status | Response Code | ++===============+===================================+ +| ``100`` | Continue | ++---------------+-----------------------------------+ +| ``200`` | Success | ++---------------+-----------------------------------+ +| ``201`` | Created | ++---------------+-----------------------------------+ +| ``202`` | Accepted | ++---------------+-----------------------------------+ +| ``204`` | NoContent | ++---------------+-----------------------------------+ +| ``206`` | Partial content | ++---------------+-----------------------------------+ +| ``304`` | NotModified | ++---------------+-----------------------------------+ +| ``400`` | InvalidArgument | ++---------------+-----------------------------------+ +| ``400`` | InvalidDigest | ++---------------+-----------------------------------+ +| ``400`` | BadDigest | ++---------------+-----------------------------------+ +| ``400`` | InvalidBucketName | ++---------------+-----------------------------------+ +| ``400`` | InvalidObjectName | ++---------------+-----------------------------------+ +| ``400`` | UnresolvableGrantByEmailAddress | ++---------------+-----------------------------------+ +| ``400`` | InvalidPart | ++---------------+-----------------------------------+ +| ``400`` | InvalidPartOrder | ++---------------+-----------------------------------+ +| ``400`` | RequestTimeout | ++---------------+-----------------------------------+ +| ``400`` | EntityTooLarge | ++---------------+-----------------------------------+ +| ``403`` | AccessDenied | ++---------------+-----------------------------------+ +| ``403`` | UserSuspended | ++---------------+-----------------------------------+ +| ``403`` | RequestTimeTooSkewed | ++---------------+-----------------------------------+ +| ``404`` | NoSuchKey | ++---------------+-----------------------------------+ +| ``404`` | NoSuchBucket | ++---------------+-----------------------------------+ +| ``404`` | NoSuchUpload | ++---------------+-----------------------------------+ +| ``405`` | MethodNotAllowed | ++---------------+-----------------------------------+ +| ``408`` | RequestTimeout | ++---------------+-----------------------------------+ +| ``409`` | BucketAlreadyExists | ++---------------+-----------------------------------+ +| ``409`` | BucketNotEmpty | ++---------------+-----------------------------------+ +| ``411`` | MissingContentLength | ++---------------+-----------------------------------+ +| ``412`` | PreconditionFailed | ++---------------+-----------------------------------+ +| ``416`` | InvalidRange | ++---------------+-----------------------------------+ +| ``422`` | UnprocessableEntity | ++---------------+-----------------------------------+ +| ``500`` | InternalError | ++---------------+-----------------------------------+ + +.. _`Ceph Object Gateway - Multisite Configuration`: ../../multisite diff --git a/doc/radosgw/s3/cpp.rst b/doc/radosgw/s3/cpp.rst new file mode 100644 index 00000000..089c9c53 --- /dev/null +++ b/doc/radosgw/s3/cpp.rst @@ -0,0 +1,337 @@ +.. _cpp: + +C++ S3 Examples +=============== + +Setup +----- + +The following contains includes and globals that will be used in later examples: + +.. code-block:: cpp + + #include "libs3.h" + #include <stdlib.h> + #include <iostream> + #include <fstream> + + const char access_key[] = "ACCESS_KEY"; + const char secret_key[] = "SECRET_KEY"; + const char host[] = "HOST"; + const char sample_bucket[] = "sample_bucket"; + const char sample_key[] = "hello.txt"; + const char sample_file[] = "resource/hello.txt"; + const char *security_token = NULL; + const char *auth_region = NULL; + + S3BucketContext bucketContext = + { + host, + sample_bucket, + S3ProtocolHTTP, + S3UriStylePath, + access_key, + secret_key, + security_token, + auth_region + }; + + S3Status responsePropertiesCallback( + const S3ResponseProperties *properties, + void *callbackData) + { + return S3StatusOK; + } + + static void responseCompleteCallback( + S3Status status, + const S3ErrorDetails *error, + void *callbackData) + { + return; + } + + S3ResponseHandler responseHandler = + { + &responsePropertiesCallback, + &responseCompleteCallback + }; + + +Creating (and Closing) a Connection +----------------------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: cpp + + S3_initialize("s3", S3_INIT_ALL, host); + // Do stuff... + S3_deinitialize(); + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name, owner ID, and display name +for each bucket. + +.. code-block:: cpp + + static S3Status listServiceCallback( + const char *ownerId, + const char *ownerDisplayName, + const char *bucketName, + int64_t creationDate, void *callbackData) + { + bool *header_printed = (bool*) callbackData; + if (!*header_printed) { + *header_printed = true; + printf("%-22s", " Bucket"); + printf(" %-20s %-12s", " Owner ID", "Display Name"); + printf("\n"); + printf("----------------------"); + printf(" --------------------" " ------------"); + printf("\n"); + } + + printf("%-22s", bucketName); + printf(" %-20s %-12s", ownerId ? ownerId : "", ownerDisplayName ? ownerDisplayName : ""); + printf("\n"); + + return S3StatusOK; + } + + S3ListServiceHandler listServiceHandler = + { + responseHandler, + &listServiceCallback + }; + bool header_printed = false; + S3_list_service(S3ProtocolHTTP, access_key, secret_key, security_token, host, + auth_region, NULL, 0, &listServiceHandler, &header_printed); + + +Creating a Bucket +----------------- + +This creates a new bucket. + +.. code-block:: cpp + + S3_create_bucket(S3ProtocolHTTP, access_key, secret_key, NULL, host, sample_bucket, S3CannedAclPrivate, NULL, NULL, &responseHandler, NULL); + + +Listing a Bucket's Content +-------------------------- + +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and +last modified date. + +.. code-block:: cpp + + static S3Status listBucketCallback( + int isTruncated, + const char *nextMarker, + int contentsCount, + const S3ListBucketContent *contents, + int commonPrefixesCount, + const char **commonPrefixes, + void *callbackData) + { + printf("%-22s", " Object Name"); + printf(" %-5s %-20s", "Size", " Last Modified"); + printf("\n"); + printf("----------------------"); + printf(" -----" " --------------------"); + printf("\n"); + + for (int i = 0; i < contentsCount; i++) { + char timebuf[256]; + char sizebuf[16]; + const S3ListBucketContent *content = &(contents[i]); + time_t t = (time_t) content->lastModified; + + strftime(timebuf, sizeof(timebuf), "%Y-%m-%dT%H:%M:%SZ", gmtime(&t)); + sprintf(sizebuf, "%5llu", (unsigned long long) content->size); + printf("%-22s %s %s\n", content->key, sizebuf, timebuf); + } + + return S3StatusOK; + } + + S3ListBucketHandler listBucketHandler = + { + responseHandler, + &listBucketCallback + }; + S3_list_bucket(&bucketContext, NULL, NULL, NULL, 0, NULL, 0, &listBucketHandler, NULL); + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: cpp + + S3_delete_bucket(S3ProtocolHTTP, S3UriStylePath, access_key, secret_key, 0, host, sample_bucket, NULL, NULL, 0, &responseHandler, NULL); + + +Creating an Object (from a file) +-------------------------------- + +This creates a file ``hello.txt``. + +.. code-block:: cpp + + #include <sys/stat.h> + typedef struct put_object_callback_data + { + FILE *infile; + uint64_t contentLength; + } put_object_callback_data; + + + static int putObjectDataCallback(int bufferSize, char *buffer, void *callbackData) + { + put_object_callback_data *data = (put_object_callback_data *) callbackData; + + int ret = 0; + + if (data->contentLength) { + int toRead = ((data->contentLength > (unsigned) bufferSize) ? (unsigned) bufferSize : data->contentLength); + ret = fread(buffer, 1, toRead, data->infile); + } + data->contentLength -= ret; + return ret; + } + + put_object_callback_data data; + struct stat statbuf; + if (stat(sample_file, &statbuf) == -1) { + fprintf(stderr, "\nERROR: Failed to stat file %s: ", sample_file); + perror(0); + exit(-1); + } + + int contentLength = statbuf.st_size; + data.contentLength = contentLength; + + if (!(data.infile = fopen(sample_file, "r"))) { + fprintf(stderr, "\nERROR: Failed to open input file %s: ", sample_file); + perror(0); + exit(-1); + } + + S3PutObjectHandler putObjectHandler = + { + responseHandler, + &putObjectDataCallback + }; + + S3_put_object(&bucketContext, sample_key, contentLength, NULL, NULL, 0, &putObjectHandler, &data); + fclose(data.infile); + + +Download an Object (to a file) +------------------------------ + +This downloads a file and prints the contents. + +.. code-block:: cpp + + static S3Status getObjectDataCallback(int bufferSize, const char *buffer, void *callbackData) + { + FILE *outfile = (FILE *) callbackData; + size_t wrote = fwrite(buffer, 1, bufferSize, outfile); + return ((wrote < (size_t) bufferSize) ? S3StatusAbortedByCallback : S3StatusOK); + } + + S3GetObjectHandler getObjectHandler = + { + responseHandler, + &getObjectDataCallback + }; + FILE *outfile = stdout; + S3_get_object(&bucketContext, sample_key, NULL, 0, 0, NULL, 0, &getObjectHandler, outfile); + + +Delete an Object +---------------- + +This deletes an object. + +.. code-block:: cpp + + S3ResponseHandler deleteResponseHandler = + { + 0, + &responseCompleteCallback + }; + S3_delete_object(&bucketContext, sample_key, 0, 0, &deleteResponseHandler, 0); + + +Change an Object's ACL +---------------------- + +This changes an object's ACL to grant full control to another user. + + +.. code-block:: cpp + + #include <string.h> + char ownerId[] = "owner"; + char ownerDisplayName[] = "owner"; + char granteeId[] = "grantee"; + char granteeDisplayName[] = "grantee"; + + S3AclGrant grants[] = { + { + S3GranteeTypeCanonicalUser, + {{}}, + S3PermissionFullControl + }, + { + S3GranteeTypeCanonicalUser, + {{}}, + S3PermissionReadACP + }, + { + S3GranteeTypeAllUsers, + {{}}, + S3PermissionRead + } + }; + + strncpy(grants[0].grantee.canonicalUser.id, ownerId, S3_MAX_GRANTEE_USER_ID_SIZE); + strncpy(grants[0].grantee.canonicalUser.displayName, ownerDisplayName, S3_MAX_GRANTEE_DISPLAY_NAME_SIZE); + + strncpy(grants[1].grantee.canonicalUser.id, granteeId, S3_MAX_GRANTEE_USER_ID_SIZE); + strncpy(grants[1].grantee.canonicalUser.displayName, granteeDisplayName, S3_MAX_GRANTEE_DISPLAY_NAME_SIZE); + + S3_set_acl(&bucketContext, sample_key, ownerId, ownerDisplayName, 3, grants, 0, &responseHandler, 0); + + +Generate Object Download URL (signed) +------------------------------------- + +This generates a signed download URL that will be valid for 5 minutes. + +.. code-block:: cpp + + #include <time.h> + char buffer[S3_MAX_AUTHENTICATED_QUERY_STRING_SIZE]; + int64_t expires = time(NULL) + 60 * 5; // Current time + 5 minutes + + S3_generate_authenticated_query_string(buffer, &bucketContext, sample_key, expires, NULL, "GET"); + diff --git a/doc/radosgw/s3/csharp.rst b/doc/radosgw/s3/csharp.rst new file mode 100644 index 00000000..af1c6e4b --- /dev/null +++ b/doc/radosgw/s3/csharp.rst @@ -0,0 +1,199 @@ +.. _csharp: + +C# S3 Examples +============== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: csharp + + using System; + using Amazon; + using Amazon.S3; + using Amazon.S3.Model; + + string accessKey = "put your access key here!"; + string secretKey = "put your secret key here!"; + + AmazonS3Config config = new AmazonS3Config(); + config.ServiceURL = "objects.dreamhost.com"; + + AmazonS3Client s3Client = new AmazonS3Client( + accessKey, + secretKey, + config + ); + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: csharp + + ListBucketsResponse response = client.ListBuckets(); + foreach (S3Bucket b in response.Buckets) + { + Console.WriteLine("{0}\t{1}", b.BucketName, b.CreationDate); + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: csharp + + PutBucketRequest request = new PutBucketRequest(); + request.BucketName = "my-new-bucket"; + client.PutBucket(request); + +Listing a Bucket's Content +-------------------------- + +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: csharp + + ListObjectsRequest request = new ListObjectsRequest(); + request.BucketName = "my-new-bucket"; + ListObjectsResponse response = client.ListObjects(request); + foreach (S3Object o in response.S3Objects) + { + Console.WriteLine("{0}\t{1}\t{2}", o.Key, o.Size, o.LastModified); + } + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: csharp + + DeleteBucketRequest request = new DeleteBucketRequest(); + request.BucketName = "my-new-bucket"; + client.DeleteBucket(request); + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. attention:: + + not available + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: csharp + + PutObjectRequest request = new PutObjectRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "hello.txt"; + request.ContentType = "text/plain"; + request.ContentBody = "Hello World!"; + client.PutObject(request); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and +``secret_plans.txt`` to be private. + +.. code-block:: csharp + + PutACLRequest request = new PutACLRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "hello.txt"; + request.CannedACL = S3CannedACL.PublicRead; + client.PutACL(request); + + PutACLRequest request2 = new PutACLRequest(); + request2.BucketName = "my-new-bucket"; + request2.Key = "secret_plans.txt"; + request2.CannedACL = S3CannedACL.Private; + client.PutACL(request2); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``C:\Users\larry\Documents`` + +.. code-block:: csharp + + GetObjectRequest request = new GetObjectRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "perl_poetry.pdf"; + GetObjectResponse response = client.GetObject(request); + response.WriteResponseStreamToFile("C:\\Users\\larry\\Documents\\perl_poetry.pdf"); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: csharp + + DeleteObjectRequest request = new DeleteObjectRequest(); + request.BucketName = "my-new-bucket"; + request.Key = "goodbye.txt"; + client.DeleteObject(request); + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. note:: + + The C# S3 Library does not have a method for generating unsigned + URLs, so the following example only shows generating signed URLs. + +.. code-block:: csharp + + GetPreSignedUrlRequest request = new GetPreSignedUrlRequest(); + request.BucketName = "my-bucket-name"; + request.Key = "secret_plans.txt"; + request.Expires = DateTime.Now.AddHours(1); + request.Protocol = Protocol.HTTP; + string url = client.GetPreSignedURL(request); + Console.WriteLine(url); + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + diff --git a/doc/radosgw/s3/java.rst b/doc/radosgw/s3/java.rst new file mode 100644 index 00000000..057c09c2 --- /dev/null +++ b/doc/radosgw/s3/java.rst @@ -0,0 +1,212 @@ +.. _java: + +Java S3 Examples +================ + +Setup +----- + +The following examples may require some or all of the following java +classes to be imported: + +.. code-block:: java + + import java.io.ByteArrayInputStream; + import java.io.File; + import java.util.List; + import com.amazonaws.auth.AWSCredentials; + import com.amazonaws.auth.BasicAWSCredentials; + import com.amazonaws.util.StringUtils; + import com.amazonaws.services.s3.AmazonS3; + import com.amazonaws.services.s3.AmazonS3Client; + import com.amazonaws.services.s3.model.Bucket; + import com.amazonaws.services.s3.model.CannedAccessControlList; + import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest; + import com.amazonaws.services.s3.model.GetObjectRequest; + import com.amazonaws.services.s3.model.ObjectListing; + import com.amazonaws.services.s3.model.ObjectMetadata; + import com.amazonaws.services.s3.model.S3ObjectSummary; + + +If you are just testing the Ceph Object Storage services, consider +using HTTP protocol instead of HTTPS protocol. + +First, import the ``ClientConfiguration`` and ``Protocol`` classes. + +.. code-block:: java + + import com.amazonaws.ClientConfiguration; + import com.amazonaws.Protocol; + + +Then, define the client configuration, and add the client configuration +as an argument for the S3 client. + +.. code-block:: java + + AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); + + ClientConfiguration clientConfig = new ClientConfiguration(); + clientConfig.setProtocol(Protocol.HTTP); + + AmazonS3 conn = new AmazonS3Client(credentials, clientConfig); + conn.setEndpoint("endpoint.com"); + + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: java + + String accessKey = "insert your access key here!"; + String secretKey = "insert your secret key here!"; + + AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); + AmazonS3 conn = new AmazonS3Client(credentials); + conn.setEndpoint("objects.dreamhost.com"); + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: java + + List<Bucket> buckets = conn.listBuckets(); + for (Bucket bucket : buckets) { + System.out.println(bucket.getName() + "\t" + + StringUtils.fromDate(bucket.getCreationDate())); + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: java + + Bucket bucket = conn.createBucket("my-new-bucket"); + + +Listing a Bucket's Content +-------------------------- +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: java + + ObjectListing objects = conn.listObjects(bucket.getName()); + do { + for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) { + System.out.println(objectSummary.getKey() + "\t" + + objectSummary.getSize() + "\t" + + StringUtils.fromDate(objectSummary.getLastModified())); + } + objects = conn.listNextBatchOfObjects(objects); + } while (objects.isTruncated()); + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: java + + conn.deleteBucket(bucket.getName()); + + +Forced Delete for Non-empty Buckets +----------------------------------- +.. attention:: + not available + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: java + + ByteArrayInputStream input = new ByteArrayInputStream("Hello World!".getBytes()); + conn.putObject(bucket.getName(), "hello.txt", input, new ObjectMetadata()); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and +``secret_plans.txt`` to be private. + +.. code-block:: java + + conn.setObjectAcl(bucket.getName(), "hello.txt", CannedAccessControlList.PublicRead); + conn.setObjectAcl(bucket.getName(), "secret_plans.txt", CannedAccessControlList.Private); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``/home/larry/documents`` + +.. code-block:: java + + conn.getObject( + new GetObjectRequest(bucket.getName(), "perl_poetry.pdf"), + new File("/home/larry/documents/perl_poetry.pdf") + ); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: java + + conn.deleteObject(bucket.getName(), "goodbye.txt"); + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. note:: + The java library does not have a method for generating unsigned + URLs, so the example below just generates a signed URL. + +.. code-block:: java + + GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucket.getName(), "secret_plans.txt"); + System.out.println(conn.generatePresignedUrl(request)); + +The output will look something like this:: + + https://my-bucket-name.objects.dreamhost.com/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + diff --git a/doc/radosgw/s3/objectops.rst b/doc/radosgw/s3/objectops.rst new file mode 100644 index 00000000..2ac52607 --- /dev/null +++ b/doc/radosgw/s3/objectops.rst @@ -0,0 +1,558 @@ +Object Operations +================= + +Put Object +---------- +Adds an object to a bucket. You must have write permissions on the bucket to perform this operation. + + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object} HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================+============================================+===============================================================================+============+ +| **content-md5** | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **content-type** | A standard MIME type. | Any MIME type. Default: ``binary/octet-stream`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-meta-<...>** | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ + + +Copy Object +----------- +To copy an object, use ``PUT`` and specify a destination bucket and the object name. + +Syntax +~~~~~~ + +:: + + PUT /{dest-bucket}/{dest-object} HTTP/1.1 + x-amz-copy-source: {source-bucket}/{source-object} + +Request Headers +~~~~~~~~~~~~~~~ + ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================================+=================================================+========================+============+ +| **x-amz-copy-source** | The source bucket name + object name. | {bucket}/{obj} | Yes | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, | No | +| | | ``public-read``, | | +| | | ``public-read-write``, | | +| | | ``authenticated-read`` | | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-modified-since** | Copies only if modified since the timestamp. | Timestamp | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-unmodified-since** | Copies only if unmodified since the timestamp. | Timestamp | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-match** | Copies only if object ETag matches ETag. | Entity Tag | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ +| **x-amz-copy-if-none-match** | Copies only if object ETag doesn't match. | Entity Tag | No | ++--------------------------------------+-------------------------------------------------+------------------------+------------+ + +Response Entities +~~~~~~~~~~~~~~~~~ + ++------------------------+-------------+-----------------------------------------------+ +| Name | Type | Description | ++========================+=============+===============================================+ +| **CopyObjectResult** | Container | A container for the response elements. | ++------------------------+-------------+-----------------------------------------------+ +| **LastModified** | Date | The last modified date of the source object. | ++------------------------+-------------+-----------------------------------------------+ +| **Etag** | String | The ETag of the new object. | ++------------------------+-------------+-----------------------------------------------+ + +Remove Object +------------- + +Removes an object. Requires WRITE permission set on the containing bucket. + +Syntax +~~~~~~ + +:: + + DELETE /{bucket}/{object} HTTP/1.1 + + + +Get Object +---------- +Retrieves an object from a bucket within RADOS. + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object} HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| Name | Description | Valid Values | Required | ++===========================+================================================+================================+============+ +| **range** | The range of the object to retrieve. | Range: bytes=beginbyte-endbyte | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-modified-since** | Gets only if modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-unmodified-since** | Gets only if not modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ + +Response Headers +~~~~~~~~~~~~~~~~ + ++-------------------+--------------------------------------------------------------------------------------------+ +| Name | Description | ++===================+============================================================================================+ +| **Content-Range** | Data range, will only be returned if the range header field was specified in the request | ++-------------------+--------------------------------------------------------------------------------------------+ + +Get Object Info +--------------- + +Returns information about object. This request will return the same +header information as with the Get Object request, but will include +the metadata only, not the object data payload. + +Syntax +~~~~~~ + +:: + + HEAD /{bucket}/{object} HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| Name | Description | Valid Values | Required | ++===========================+================================================+================================+============+ +| **range** | The range of the object to retrieve. | Range: bytes=beginbyte-endbyte | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-modified-since** | Gets only if modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-unmodified-since** | Gets only if not modified since the timestamp. | Timestamp | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ +| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | ++---------------------------+------------------------------------------------+--------------------------------+------------+ + +Get Object ACL +-------------- + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?acl HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the response. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the object owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The object owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The object owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` object. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + + + +Set Object ACL +-------------- + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?acl + +Request Entities +~~~~~~~~~~~~~~~~ + ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++===========================+=============+==============================================================================================+ +| ``AccessControlPolicy`` | Container | A container for the response. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``AccessControlList`` | Container | A container for the ACL information. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the object owner's ``ID`` and ``DisplayName``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``ID`` | String | The object owner's ID. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The object owner's display name. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grant`` | Container | A container for ``Grantee`` and ``Permission``. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Grantee`` | Container | A container for the ``DisplayName`` and ``ID`` of the user receiving a grant of permission. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ +| ``Permission`` | String | The permission given to the ``Grantee`` object. | ++---------------------------+-------------+----------------------------------------------------------------------------------------------+ + + + +Initiate Multi-part Upload +-------------------------- + +Initiate a multi-part upload process. + +Syntax +~~~~~~ + +:: + + POST /{bucket}/{object}?uploads + +Request Headers +~~~~~~~~~~~~~~~ + ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================+============================================+===============================================================================+============+ +| **content-md5** | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **content-type** | A standard MIME type. | Any MIME type. Default: ``binary/octet-stream`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-meta-<...>** | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=========================================+=============+==========================================================================================================+ +| ``InitiatedMultipartUploadsResult`` | Container | A container for the results. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Bucket`` | String | The bucket that will receive the object contents. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Key`` | String | The key specified by the ``key`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadId`` | String | The ID specified by the ``upload-id`` request parameter identifying the multipart upload (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ + + +Multipart Upload Part +--------------------- + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?partNumber=&uploadId= HTTP/1.1 + +HTTP Response +~~~~~~~~~~~~~ + +The following HTTP response may be returned: + ++---------------+----------------+--------------------------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+================+==========================================================================+ +| **404** | NoSuchUpload | Specified upload-id does not match any initiated upload on this object | ++---------------+----------------+--------------------------------------------------------------------------+ + +List Multipart Upload Parts +--------------------------- + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?uploadId=123 HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| Name | Type | Description | ++=========================================+=============+==========================================================================================================+ +| ``ListPartsResult`` | Container | A container for the results. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Bucket`` | String | The bucket that will receive the object contents. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Key`` | String | The key specified by the ``key`` request parameter (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``UploadId`` | String | The ID specified by the ``upload-id`` request parameter identifying the multipart upload (if any). | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Initiator`` | Container | Contains the ``ID`` and ``DisplayName`` of the user who initiated the upload. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ID`` | String | The initiator's ID. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``DisplayName`` | String | The initiator's display name. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Owner`` | Container | A container for the ``ID`` and ``DisplayName`` of the user who owns the uploaded object. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``StorageClass`` | String | The method used to store the resulting object. ``STANDARD`` or ``REDUCED_REDUNDANCY`` | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``PartNumberMarker`` | String | The part marker to use in a subsequent request if ``IsTruncated`` is ``true``. Precedes the list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``NextPartNumberMarker`` | String | The next part marker to use in a subsequent request if ``IsTruncated`` is ``true``. The end of the list. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``MaxParts`` | Integer | The max parts allowed in the response as specified by the ``max-parts`` request parameter. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``IsTruncated`` | Boolean | If ``true``, only a subset of the object's upload contents were returned. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Part`` | Container | A container for ``LastModified``, ``PartNumber``, ``ETag`` and ``Size`` elements. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``LastModified`` | Date | Date and time at which the part was uploaded. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``PartNumber`` | Integer | The identification number of the part. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``ETag`` | String | The part's entity tag. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ +| ``Size`` | Integer | The size of the uploaded part. | ++-----------------------------------------+-------------+----------------------------------------------------------------------------------------------------------+ + + + +Complete Multipart Upload +------------------------- +Assembles uploaded parts and creates a new object, thereby completing a multipart upload. + +Syntax +~~~~~~ + +:: + + POST /{bucket}/{object}?uploadId= HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| Name | Type | Description | Required | ++==================================+=============+=====================================================+==========+ +| ``CompleteMultipartUpload`` | Container | A container consisting of one or more parts. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| ``Part`` | Container | A container for the ``PartNumber`` and ``ETag``. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| ``PartNumber`` | Integer | The identifier of the part. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ +| ``ETag`` | String | The part's entity tag. | Yes | ++----------------------------------+-------------+-----------------------------------------------------+----------+ + + +Response Entities +~~~~~~~~~~~~~~~~~ + ++-------------------------------------+-------------+-------------------------------------------------------+ +| Name | Type | Description | ++=====================================+=============+=======================================================+ +| **CompleteMultipartUploadResult** | Container | A container for the response. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **Location** | URI | The resource identifier (path) of the new object. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **Bucket** | String | The name of the bucket that contains the new object. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **Key** | String | The object's key. | ++-------------------------------------+-------------+-------------------------------------------------------+ +| **ETag** | String | The entity tag of the new object. | ++-------------------------------------+-------------+-------------------------------------------------------+ + +Abort Multipart Upload +---------------------- + +Syntax +~~~~~~ + +:: + + DELETE /{bucket}/{object}?uploadId= HTTP/1.1 + + + +Append Object +------------- +Append data to an object. You must have write permissions on the bucket to perform this operation. +It is used to upload files in appending mode. The type of the objects created by the Append Object +operation is Appendable Object, and the type of the objects uploaded with the Put Object operation is Normal Object. +**Append Object can't be used if bucket versioning is enabled or suspended.** +**Synced object will become normal in multisite, but you can still append to the original object.** +**Compression and encryption features are disabled for Appendable objects.** + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?append&position= HTTP/1.1 + +Request Headers +~~~~~~~~~~~~~~~ + ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| Name | Description | Valid Values | Required | ++======================+============================================+===============================================================================+============+ +| **content-md5** | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **content-type** | A standard MIME type. | Any MIME type. Default: ``binary/octet-stream`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-meta-<...>** | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ +| **x-amz-acl** | A canned ACL. | ``private``, ``public-read``, ``public-read-write``, ``authenticated-read`` | No | ++----------------------+--------------------------------------------+-------------------------------------------------------------------------------+------------+ + +Response Headers +~~~~~~~~~~~~~~~~ + ++--------------------------------+------------------------------------------------------------------+ +| Name | Description | ++================================+==================================================================+ +| **x-rgw-next-append-position** | Next position to append object | ++--------------------------------+------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + +The following HTTP response may be returned: + ++---------------+----------------------------+---------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+============================+===================================================+ +| **409** | PositionNotEqualToLength | Specified position does not match object length | ++---------------+----------------------------+---------------------------------------------------+ +| **409** | ObjectNotAppendable | Specified object can not be appended | ++---------------+----------------------------+---------------------------------------------------+ +| **409** | InvalidBucketstate | Bucket versioning is enabled or suspended | ++---------------+----------------------------+---------------------------------------------------+ + + +Put Object Retention +-------------------- +Places an Object Retention configuration on an object. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?retention&versionId= HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++=====================+=============+===============================================================================+============+ +| ``Retention`` | Container | A container for the request. | Yes | ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| ``Mode`` | String | Retention mode for the specified object. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ +| ``RetainUntilDate`` | Timestamp | Retention date. Format: 2020-01-05T00:00:00.000Z | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ + + +Get Object Retention +-------------------- +Gets an Object Retention configuration on an object. + + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?retention&versionId= HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++=====================+=============+===============================================================================+============+ +| ``Retention`` | Container | A container for the request. | Yes | ++---------------------+-------------+-------------------------------------------------------------------------------+------------+ +| ``Mode`` | String | Retention mode for the specified object. Valid Values: GOVERNANCE/COMPLIANCE | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ +| ``RetainUntilDate`` | Timestamp | Retention date. Format: 2020-01-05T00:00:00.000Z | Yes | ++---------------------+-------------+--------------------------------------------------------------------------------------------+ + + +Put Object Legal Hold +--------------------- +Applies a Legal Hold configuration to the specified object. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}/{object}?legal-hold&versionId= HTTP/1.1 + +Request Entities +~~~~~~~~~~~~~~~~ + ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++================+=============+========================================================================================+============+ +| ``LegalHold`` | Container | A container for the request. | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| ``Status`` | String | Indicates whether the specified object has a Legal Hold in place. Valid Values: ON/OFF | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ + + +Get Object Legal Hold +--------------------- +Gets an object's current Legal Hold status. + +Syntax +~~~~~~ + +:: + + GET /{bucket}/{object}?legal-hold&versionId= HTTP/1.1 + +Response Entities +~~~~~~~~~~~~~~~~~ + ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| Name | Type | Description | Required | ++================+=============+========================================================================================+============+ +| ``LegalHold`` | Container | A container for the request. | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ +| ``Status`` | String | Indicates whether the specified object has a Legal Hold in place. Valid Values: ON/OFF | Yes | ++----------------+-------------+----------------------------------------------------------------------------------------+------------+ + diff --git a/doc/radosgw/s3/perl.rst b/doc/radosgw/s3/perl.rst new file mode 100644 index 00000000..f12e5c69 --- /dev/null +++ b/doc/radosgw/s3/perl.rst @@ -0,0 +1,192 @@ +.. _perl: + +Perl S3 Examples +================ + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: perl + + use Amazon::S3; + my $access_key = 'put your access key here!'; + my $secret_key = 'put your secret key here!'; + + my $conn = Amazon::S3->new({ + aws_access_key_id => $access_key, + aws_secret_access_key => $secret_key, + host => 'objects.dreamhost.com', + secure => 1, + retry => 1, + }); + + +Listing Owned Buckets +--------------------- + +This gets a list of `Amazon::S3::Bucket`_ objects that you own. +We'll also print out the bucket name and creation date of each bucket. + +.. code-block:: perl + + my @buckets = @{$conn->buckets->{buckets} || []}; + foreach my $bucket (@buckets) { + print $bucket->bucket . "\t" . $bucket->creation_date . "\n"; + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: perl + + my $bucket = $conn->add_bucket({ bucket => 'my-new-bucket' }); + + +Listing a Bucket's Content +-------------------------- + +This gets a list of hashes with info about each object in the bucket. +We'll also print out each object's name, the file size, and last +modified date. + +.. code-block:: perl + + my @keys = @{$bucket->list_all->{keys} || []}; + foreach my $key (@keys) { + print "$key->{key}\t$key->{size}\t$key->{last_modified}\n"; + } + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: perl + + $conn->delete_bucket($bucket); + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. attention:: + + not available in the `Amazon::S3`_ perl module + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: perl + + $bucket->add_key( + 'hello.txt', 'Hello World!', + { content_type => 'text/plain' }, + ); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable and +``secret_plans.txt`` to be private. + +.. code-block:: perl + + $bucket->set_acl({ + key => 'hello.txt', + acl_short => 'public-read', + }); + $bucket->set_acl({ + key => 'secret_plans.txt', + acl_short => 'private', + }); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: perl + + $bucket->get_key_filename('perl_poetry.pdf', undef, + '/home/larry/documents/perl_poetry.pdf'); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: perl + + $bucket->delete_key('goodbye.txt'); + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +Then this generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. note:: + The `Amazon::S3`_ module does not have a way to generate download + URLs, so we are going to be using another module instead. Unfortunately, + most modules for generating these URLs assume that you are using Amazon, + so we have had to go with using a more obscure module, `Muck::FS::S3`_. This + should be the same as Amazon's sample S3 perl module, but this sample + module is not in CPAN. So, you can either use CPAN to install + `Muck::FS::S3`_, or install Amazon's sample S3 module manually. If you go + the manual route, you can remove ``Muck::FS::`` from the example below. + +.. code-block:: perl + + use Muck::FS::S3::QueryStringAuthGenerator; + my $generator = Muck::FS::S3::QueryStringAuthGenerator->new( + $access_key, + $secret_key, + 0, # 0 means use 'http'. set this to 1 for 'https' + 'objects.dreamhost.com', + ); + + my $hello_url = $generator->make_bare_url($bucket->bucket, 'hello.txt'); + print $hello_url . "\n"; + + $generator->expires_in(3600); # 1 hour = 3600 seconds + my $plans_url = $generator->get($bucket->bucket, 'secret_plans.txt'); + print $plans_url . "\n"; + +The output will look something like this:: + + http://objects.dreamhost.com:80/my-bucket-name/hello.txt + http://objects.dreamhost.com:80/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + + +.. _`Amazon::S3`: http://search.cpan.org/~tima/Amazon-S3-0.441/lib/Amazon/S3.pm +.. _`Amazon::S3::Bucket`: http://search.cpan.org/~tima/Amazon-S3-0.441/lib/Amazon/S3/Bucket.pm +.. _`Muck::FS::S3`: http://search.cpan.org/~mike/Muck-0.02/ + diff --git a/doc/radosgw/s3/php.rst b/doc/radosgw/s3/php.rst new file mode 100644 index 00000000..c5079f5f --- /dev/null +++ b/doc/radosgw/s3/php.rst @@ -0,0 +1,208 @@ +.. _php: + +PHP S3 Examples +=============== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: php + + <?php + define('AWS_KEY', 'place access key here'); + define('AWS_SECRET_KEY', 'place secret key here'); + define('AWS_CANONICAL_ID', 'your DHO Username'); + define('AWS_CANONICAL_NAME', 'Also your DHO Username!'); + $HOST = 'objects.dreamhost.com'; + + // require the amazon sdk for php library + require_once 'AWSSDKforPHP/sdk.class.php'; + + // Instantiate the S3 class and point it at the desired host + $Connection = new AmazonS3(array( + 'key' => AWS_KEY, + 'secret' => AWS_SECRET_KEY, + 'canonical_id' => AWS_CANONICAL_ID, + 'canonical_name' => AWS_CANONICAL_NAME, + )); + $Connection->set_hostname($HOST); + $Connection->allow_hostname_override(false); + + // Set the S3 class to use objects.dreamhost.com/bucket + // instead of bucket.objects.dreamhost.com + $Connection->enable_path_style(); + + +Listing Owned Buckets +--------------------- +This gets a list of CFSimpleXML objects representing buckets that you +own. This also prints out the bucket name and creation date of each +bucket. + +.. code-block:: php + + <?php + $ListResponse = $Connection->list_buckets(); + $Buckets = $ListResponse->body->Buckets->Bucket; + foreach ($Buckets as $Bucket) { + echo $Bucket->Name . "\t" . $Bucket->CreationDate . "\n"; + } + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` and returns a +``CFResponse`` object. + +.. note:: + + This command requires a region as the second argument, + so we use ``AmazonS3::REGION_US_E1``, because this constant is ``''`` + +.. code-block:: php + + <?php + $Connection->create_bucket('my-new-bucket', AmazonS3::REGION_US_E1); + + +List a Bucket's Content +----------------------- + +This gets an array of ``CFSimpleXML`` objects representing the objects +in the bucket. This then prints out each object's name, the file size, +and last modified date. + +.. code-block:: php + + <?php + $ObjectsListResponse = $Connection->list_objects($bucketname); + $Objects = $ObjectsListResponse->body->Contents; + foreach ($Objects as $Object) { + echo $Object->Key . "\t" . $Object->Size . "\t" . $Object->LastModified . "\n"; + } + +.. note:: + + If there are more than 1000 objects in this bucket, + you need to check $ObjectListResponse->body->isTruncated + and run again with the name of the last key listed. + Keep doing this until isTruncated is not true. + +The output will look something like this if the bucket has some files:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +This deletes the bucket called ``my-old-bucket`` and returns a +``CFResponse`` object + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: php + + <?php + $Connection->delete_bucket('my-old-bucket'); + + +Forced Delete for Non-empty Buckets +----------------------------------- + +This will delete the bucket even if it is not empty. + +.. code-block:: php + + <?php + $Connection->delete_bucket('my-old-bucket', 1); + + +Creating an Object +------------------ + +This creates an object ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: php + + <?php + $Connection->create_object('my-bucket-name', 'hello.txt', array( + 'body' => "Hello World!", + )); + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable and +``secret_plans.txt`` to be private. + +.. code-block:: php + + <?php + $Connection->set_object_acl('my-bucket-name', 'hello.txt', AmazonS3::ACL_PUBLIC); + $Connection->set_object_acl('my-bucket-name', 'secret_plans.txt', AmazonS3::ACL_PRIVATE); + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: php + + <?php + $Connection->delete_object('my-bucket-name', 'goodbye.txt'); + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: php + + <?php + $FileHandle = fopen('/home/larry/documents/poetry.pdf', 'w+'); + $Connection->get_object('my-bucket-name', 'poetry.pdf', array( + 'fileDownload' => $FileHandle, + )); + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. +This works because we made ``hello.txt`` public by setting +the ACL above. This then generates a signed download URL +for ``secret_plans.txt`` that will work for 1 hour. +Signed download URLs will work for the time period even +if the object is private (when the time period is up, +the URL will stop working). + +.. code-block:: php + + <?php + my $plans_url = $Connection->get_object_url('my-bucket-name', 'hello.txt'); + echo $plans_url . "\n"; + my $secret_url = $Connection->get_object_url('my-bucket-name', 'secret_plans.txt', '1 hour'); + echo $secret_url . "\n"; + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + diff --git a/doc/radosgw/s3/python.rst b/doc/radosgw/s3/python.rst new file mode 100644 index 00000000..a2c6a59d --- /dev/null +++ b/doc/radosgw/s3/python.rst @@ -0,0 +1,171 @@ +.. _python: + +Python S3 Examples +================== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: python + + import boto + import boto.s3.connection + access_key = 'put your access key here!' + secret_key = 'put your secret key here!' + + conn = boto.connect_s3( + aws_access_key_id = access_key, + aws_secret_access_key = secret_key, + host = 'objects.dreamhost.com', + #is_secure=False, # uncomment if you are not using ssl + calling_format = boto.s3.connection.OrdinaryCallingFormat(), + ) + + +Listing Owned Buckets +--------------------- + +This gets a list of Buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: python + + for bucket in conn.get_all_buckets(): + print "{name}\t{created}".format( + name = bucket.name, + created = bucket.creation_date, + ) + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: python + + bucket = conn.create_bucket('my-new-bucket') + + +Listing a Bucket's Content +-------------------------- + +This gets a list of objects in the bucket. +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: python + + for key in bucket.list(): + print "{name}\t{size}\t{modified}".format( + name = key.name, + size = key.size, + modified = key.last_modified, + ) + +The output will look something like this:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- + +.. note:: + + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: python + + conn.delete_bucket(bucket.name) + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. attention:: + + not available in python + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: python + + key = bucket.new_key('hello.txt') + key.set_contents_from_string('Hello World!') + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and +``secret_plans.txt`` to be private. + +.. code-block:: python + + hello_key = bucket.get_key('hello.txt') + hello_key.set_canned_acl('public-read') + plans_key = bucket.get_key('secret_plans.txt') + plans_key.set_canned_acl('private') + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``perl_poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: python + + key = bucket.get_key('perl_poetry.pdf') + key.get_contents_to_filename('/home/larry/documents/perl_poetry.pdf') + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: python + + bucket.delete_key('goodbye.txt') + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. code-block:: python + + hello_key = bucket.get_key('hello.txt') + hello_url = hello_key.generate_url(0, query_auth=False, force_http=True) + print hello_url + + plans_key = bucket.get_key('secret_plans.txt') + plans_url = plans_key.generate_url(3600, query_auth=True, force_http=True) + print plans_url + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + diff --git a/doc/radosgw/s3/ruby.rst b/doc/radosgw/s3/ruby.rst new file mode 100644 index 00000000..435b3c63 --- /dev/null +++ b/doc/radosgw/s3/ruby.rst @@ -0,0 +1,364 @@ +.. _ruby: + +Ruby `AWS::SDK`_ Examples (aws-sdk gem ~>2) +=========================================== + +Settings +--------------------- + +You can setup the connection on global way: + +.. code-block:: ruby + + Aws.config.update( + endpoint: 'https://objects.dreamhost.com.', + access_key_id: 'my-access-key', + secret_access_key: 'my-secret-key', + force_path_style: true, + region: 'us-east-1' + ) + + +and instantiate a client object: + +.. code-block:: ruby + + s3_client = Aws::S3::Client.new + +Listing Owned Buckets +--------------------- + +This gets a list of buckets that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: ruby + + s3_client.list_buckets.buckets.each do |bucket| + puts "#{bucket.name}\t#{bucket.creation_date}" + end + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: ruby + + s3_client.create_bucket(bucket: 'my-new-bucket') + +If you want a private bucket: + +`acl` option accepts: # private, public-read, public-read-write, authenticated-read + +.. code-block:: ruby + + s3_client.create_bucket(bucket: 'my-new-bucket', acl: 'private') + + +Listing a Bucket's Content +-------------------------- + +This gets a list of hashes with the contents of each object +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: ruby + + s3_client.get_objects(bucket: 'my-new-bucket').contents.each do |object| + puts "#{object.key}\t#{object.size}\t#{object.last-modified}" + end + +The output will look something like this if the bucket has some files:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: ruby + + s3_client.delete_bucket(bucket: 'my-new-bucket') + + +Forced Delete for Non-empty Buckets +----------------------------------- +First, you need to clear the bucket: + +.. code-block:: ruby + + Aws::S3::Bucket.new('my-new-bucket', client: s3_client).clear! + +after, you can destroy the bucket + +.. code-block:: ruby + + s3_client.delete_bucket(bucket: 'my-new-bucket') + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: ruby + + s3_client.put_object( + key: 'hello.txt', + body: 'Hello World!', + bucket: 'my-new-bucket', + content_type: 'text/plain' + ) + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and ``secret_plans.txt`` +to be private. + +.. code-block:: ruby + + s3_client.put_object_acl(bucket: 'my-new-bucket', key: 'hello.txt', acl: 'public-read') + + s3_client.put_object_acl(bucket: 'my-new-bucket', key: 'private.txt', acl: 'private') + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: ruby + + s3_client.get_object(bucket: 'my-new-bucket', key: 'poetry.pdf', response_target: '/home/larry/documents/poetry.pdf') + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: ruby + + s3_client.delete_object(key: 'goodbye.txt', bucket: 'my-new-bucket') + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. code-block:: ruby + + puts Aws::S3::Object.new( + key: 'hello.txt', + bucket_name: 'my-new-bucket', + client: s3_client + ).public_url + + puts Aws::S3::Object.new( + key: 'secret_plans.txt', + bucket_name: 'hermes_ceph_gem', + client: s3_client + ).presigned_url(:get, expires_in: 60 * 60) + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + +.. _`AWS::SDK`: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html + + + +Ruby `AWS::S3`_ Examples (aws-s3 gem) +===================================== + +Creating a Connection +--------------------- + +This creates a connection so that you can interact with the server. + +.. code-block:: ruby + + AWS::S3::Base.establish_connection!( + :server => 'objects.dreamhost.com', + :use_ssl => true, + :access_key_id => 'my-access-key', + :secret_access_key => 'my-secret-key' + ) + + +Listing Owned Buckets +--------------------- + +This gets a list of `AWS::S3::Bucket`_ objects that you own. +This also prints out the bucket name and creation date of each bucket. + +.. code-block:: ruby + + AWS::S3::Service.buckets.each do |bucket| + puts "#{bucket.name}\t#{bucket.creation_date}" + end + +The output will look something like this:: + + mahbuckat1 2011-04-21T18:05:39.000Z + mahbuckat2 2011-04-21T18:05:48.000Z + mahbuckat3 2011-04-21T18:07:18.000Z + + +Creating a Bucket +----------------- + +This creates a new bucket called ``my-new-bucket`` + +.. code-block:: ruby + + AWS::S3::Bucket.create('my-new-bucket') + + +Listing a Bucket's Content +-------------------------- + +This gets a list of hashes with the contents of each object +This also prints out each object's name, the file size, and last +modified date. + +.. code-block:: ruby + + new_bucket = AWS::S3::Bucket.find('my-new-bucket') + new_bucket.each do |object| + puts "#{object.key}\t#{object.about['content-length']}\t#{object.about['last-modified']}" + end + +The output will look something like this if the bucket has some files:: + + myphoto1.jpg 251262 2011-08-08T21:35:48.000Z + myphoto2.jpg 262518 2011-08-08T21:38:01.000Z + + +Deleting a Bucket +----------------- +.. note:: + The Bucket must be empty! Otherwise it won't work! + +.. code-block:: ruby + + AWS::S3::Bucket.delete('my-new-bucket') + + +Forced Delete for Non-empty Buckets +----------------------------------- + +.. code-block:: ruby + + AWS::S3::Bucket.delete('my-new-bucket', :force => true) + + +Creating an Object +------------------ + +This creates a file ``hello.txt`` with the string ``"Hello World!"`` + +.. code-block:: ruby + + AWS::S3::S3Object.store( + 'hello.txt', + 'Hello World!', + 'my-new-bucket', + :content_type => 'text/plain' + ) + + +Change an Object's ACL +---------------------- + +This makes the object ``hello.txt`` to be publicly readable, and ``secret_plans.txt`` +to be private. + +.. code-block:: ruby + + policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket') + policy.grants = [ AWS::S3::ACL::Grant.grant(:public_read) ] + AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy) + + policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket') + policy.grants = [] + AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy) + + +Download an Object (to a file) +------------------------------ + +This downloads the object ``poetry.pdf`` and saves it in +``/home/larry/documents/`` + +.. code-block:: ruby + + open('/home/larry/documents/poetry.pdf', 'w') do |file| + AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk| + file.write(chunk) + end + end + + +Delete an Object +---------------- + +This deletes the object ``goodbye.txt`` + +.. code-block:: ruby + + AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket') + + +Generate Object Download URLs (signed and unsigned) +--------------------------------------------------- + +This generates an unsigned download URL for ``hello.txt``. This works +because we made ``hello.txt`` public by setting the ACL above. +This then generates a signed download URL for ``secret_plans.txt`` that +will work for 1 hour. Signed download URLs will work for the time +period even if the object is private (when the time period is up, the +URL will stop working). + +.. code-block:: ruby + + puts AWS::S3::S3Object.url_for( + 'hello.txt', + 'my-new-bucket', + :authenticated => false + ) + + puts AWS::S3::S3Object.url_for( + 'secret_plans.txt', + 'my-new-bucket', + :expires_in => 60 * 60 + ) + +The output of this will look something like:: + + http://objects.dreamhost.com/my-bucket-name/hello.txt + http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX + +.. _`AWS::S3`: http://amazon.rubyforge.org/ +.. _`AWS::S3::Bucket`: http://amazon.rubyforge.org/doc/ + diff --git a/doc/radosgw/s3/serviceops.rst b/doc/radosgw/s3/serviceops.rst new file mode 100644 index 00000000..c55ce988 --- /dev/null +++ b/doc/radosgw/s3/serviceops.rst @@ -0,0 +1,39 @@ +Service Operations +================== + +List Buckets +------------ +``GET /`` returns a list of buckets created by the user making the request. ``GET /`` only +returns buckets created by an authenticated user. You cannot make an anonymous request. + +Syntax +~~~~~~ +:: + + GET / HTTP/1.1 + Host: cname.domain.com + + Authorization: AWS {access-key}:{hash-of-header-and-secret} + +Response Entities +~~~~~~~~~~~~~~~~~ + ++----------------------------+-------------+-----------------------------------------------------------------+ +| Name | Type | Description | ++============================+=============+=================================================================+ +| ``Buckets`` | Container | Container for list of buckets. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``Bucket`` | Container | Container for bucket information. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``Name`` | String | Bucket name. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``CreationDate`` | Date | UTC time when the bucket was created. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``ListAllMyBucketsResult`` | Container | A container for the result. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``Owner`` | Container | A container for the bucket owner's ``ID`` and ``DisplayName``. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``ID`` | String | The bucket owner's ID. | ++----------------------------+-------------+-----------------------------------------------------------------+ +| ``DisplayName`` | String | The bucket owner's display name. | ++----------------------------+-------------+-----------------------------------------------------------------+ |