swiftclient¶
OpenStack Swift Python client binding.
swiftclient.client¶
OpenStack Swift client library used internally
-
class
swiftclient.client.
Connection
(authurl=None, user=None, key=None, retries=5, preauthurl=None, preauthtoken=None, snet=False, starting_backoff=1, max_backoff=64, tenant_name=None, os_options=None, auth_version='1', cacert=None, insecure=False, ssl_compression=True, retry_on_ratelimit=False, timeout=None)¶ Bases:
object
Convenience class to make requests that will also retry the request
Requests will have an X-Auth-Token header whose value is either the preauthtoken or a token obtained from the auth service using the user credentials provided as args to the constructor. If os_options includes a service_username then requests will also have an X-Service-Token header whose value is a token obtained from the auth service using the service credentials. In this case the request url will be set to the storage_url obtained from the auth service for the service user, unless this is overridden by a preauthurl.
Parameters: - authurl – authentication URL
- user – user name to authenticate as
- key – key/password to authenticate with
- retries – Number of times to retry the request before failing
- preauthurl – storage URL (if you have already authenticated)
- preauthtoken – authentication token (if you have already authenticated) note authurl/user/key/tenant_name are not required when specifying preauthtoken
- snet – use SERVICENET internal network default is False
- starting_backoff – initial delay between retries (seconds)
- max_backoff – maximum delay between retries (seconds)
- auth_version – OpenStack auth version, default is 1.0
- tenant_name – The tenant/account name, required when connecting to an auth 2.0 system.
- os_options – The OpenStack options which can have tenant_id, auth_token, service_type, endpoint_type, tenant_name, object_storage_url, region_name, service_username, service_project_name, service_key
- insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.
- ssl_compression – Whether to enable compression at the SSL layer. If set to ‘False’ and the pyOpenSSL library is present an attempt to disable SSL compression will be made. This may provide a performance increase for https upload/download operations.
- retry_on_ratelimit – by default, a ratelimited connection will raise an exception to the caller. Setting this parameter to True will cause a retry after a backoff.
- timeout – The connect timeout for the HTTP connection.
-
close
()¶
-
delete_container
(container, response_dict=None)¶ Wrapper for
delete_container()
-
delete_object
(container, obj, query_string=None, response_dict=None)¶ Wrapper for
delete_object()
-
get_account
(marker=None, limit=None, prefix=None, end_marker=None, full_listing=False)¶ Wrapper for
get_account()
-
get_auth
()¶
-
get_capabilities
(url=None)¶
-
get_container
(container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, full_listing=False)¶ Wrapper for
get_container()
-
get_object
(container, obj, resp_chunk_size=None, query_string=None, response_dict=None, headers=None)¶ Wrapper for
get_object()
-
get_service_auth
()¶
-
head_account
()¶ Wrapper for
head_account()
-
head_container
(container)¶ Wrapper for
head_container()
-
head_object
(container, obj)¶ Wrapper for
head_object()
-
http_connection
(url=None)¶
-
post_account
(headers, response_dict=None)¶ Wrapper for
post_account()
-
post_container
(container, headers, response_dict=None)¶ Wrapper for
post_container()
-
post_object
(container, obj, headers, response_dict=None)¶ Wrapper for
post_object()
-
put_container
(container, headers=None, response_dict=None)¶ Wrapper for
put_container()
-
put_object
(container, obj, contents, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, query_string=None, response_dict=None)¶ Wrapper for
put_object()
-
class
swiftclient.client.
HTTPConnection
(url, proxy=None, cacert=None, insecure=False, ssl_compression=False, default_user_agent=None, timeout=None)¶ Bases:
object
Make an HTTPConnection or HTTPSConnection
Parameters: - url – url to connect to
- proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888‘ to set one
- cacert – A CA bundle file to use in verifying a TLS server certificate.
- insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.
- ssl_compression – SSL compression should be disabled by default and this setting is not usable as of now. The parameter is kept for backward compatibility.
- default_user_agent – Set the User-Agent header on every request. If set to None (default), the user agent will be “python-swiftclient-<version>”. This may be overridden on a per-request basis by explicitly setting the user-agent header on a call to request().
- timeout – socket read timeout value, passed directly to the requests library.
Raises ClientException: Unable to handle protocol scheme
-
getresponse
()¶ Adapt requests response to httplib interface
-
putrequest
(full_path, data=None, headers=None, files=None)¶ Use python-requests files upload
Parameters: - data – Use data generator for chunked-transfer
- files – Use files for default transfer
-
request
(method, full_path, data=None, headers=None, files=None)¶ Encode url and header, then call requests.request
-
swiftclient.client.
delete_container
(url, token, container, http_conn=None, response_dict=None, service_token=None)¶ Delete a container
Parameters: - url – storage URL
- token – auth token
- container – container name to delete
- http_conn – HTTP connection object (If None, it will create the conn object)
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Raises ClientException: HTTP DELETE request failed
-
swiftclient.client.
delete_object
(url, token=None, container=None, name=None, http_conn=None, headers=None, proxy=None, query_string=None, response_dict=None, service_token=None)¶ Delete object
Parameters: - url – storage URL
- token – auth token; if None, no token will be sent
- container – container name that the object is in; if None, the container name is expected to be part of the url
- name – object name to delete; if None, the object name is expected to be part of the url
- http_conn – HTTP connection object (If None, it will create the conn object)
- headers – additional headers to include in the request
- proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888‘ to set one
- query_string – if set will be appended with ‘?’ to generated path
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Raises ClientException: HTTP DELETE request failed
-
swiftclient.client.
encode_meta_headers
(headers)¶ Only encode metadata headers keys
-
swiftclient.client.
encode_utf8
(value)¶
-
swiftclient.client.
get_account
(url, token, marker=None, limit=None, prefix=None, end_marker=None, http_conn=None, full_listing=False, service_token=None)¶ Get a listing of containers for the account.
Parameters: - url – storage URL
- token – auth token
- marker – marker query
- limit – limit query
- prefix – prefix query
- end_marker – end_marker query
- http_conn – HTTP connection object (If None, it will create the conn object)
- full_listing – if True, return a full listing, else returns a max of 10000 listings
- service_token – service auth token
Returns: a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase.
Raises ClientException: HTTP GET request failed
-
swiftclient.client.
get_auth
(auth_url, user, key, **kwargs)¶ Get authentication/authorization credentials.
Parameters: - auth_version – the api version of the supplied auth params
- os_options – a dict, the openstack idenity service options
Returns: a tuple, (storage_url, token)
N.B. if the optional os_options paramater includes an non-empty ‘object_storage_url’ key it will override the the default storage url returned by the auth service.
The snet parameter is used for Rackspace’s ServiceNet internal network implementation. In this function, it simply adds snet- to the beginning of the host name for the returned storage URL. With Rackspace Cloud Files, use of this network path causes no bandwidth charges but requires the client to be running on Rackspace’s ServiceNet network.
-
swiftclient.client.
get_auth_1_0
(url, user, key, snet, **kwargs)¶
-
swiftclient.client.
get_auth_keystone
(auth_url, user, key, os_options, **kwargs)¶ Authenticate against a keystone server.
We are using the keystoneclient library for authentication.
-
swiftclient.client.
get_capabilities
(http_conn)¶ Get cluster capability infos.
Parameters: http_conn – HTTP connection Returns: a dict containing the cluster capabilities Raises ClientException: HTTP Capabilities GET failed
-
swiftclient.client.
get_container
(url, token, container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, http_conn=None, full_listing=False, service_token=None)¶ Get a listing of objects for the container.
Parameters: - url – storage URL
- token – auth token
- container – container name to get a listing for
- marker – marker query
- limit – limit query
- prefix – prefix query
- delimiter – string to delimit the queries on
- end_marker – marker query
- path – path query (equivalent: “delimiter=/” and “prefix=path/”)
- http_conn – HTTP connection object (If None, it will create the conn object)
- full_listing – if True, return a full listing, else returns a max of 10000 listings
- service_token – service auth token
Returns: a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase.
Raises ClientException: HTTP GET request failed
-
swiftclient.client.
get_keystoneclient_2_0
(auth_url, user, key, os_options, **kwargs)¶
-
swiftclient.client.
get_object
(url, token, container, name, http_conn=None, resp_chunk_size=None, query_string=None, response_dict=None, headers=None, service_token=None)¶ Get an object
Parameters: - url – storage URL
- token – auth token
- container – container name that the object is in
- name – object name to get
- http_conn – HTTP connection object (If None, it will create the conn object)
- resp_chunk_size – if defined, chunk size of data to read. NOTE: If you specify a resp_chunk_size you must fully read the object’s contents before making another request.
- query_string – if set will be appended with ‘?’ to generated path
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- headers – an optional dictionary with additional headers to include in the request
- service_token – service auth token
Returns: a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase.
Raises ClientException: HTTP GET request failed
-
swiftclient.client.
head_account
(url, token, http_conn=None, service_token=None)¶ Get account stats.
Parameters: - url – storage URL
- token – auth token
- http_conn – HTTP connection object (If None, it will create the conn object)
- service_token – service auth token
Returns: a dict containing the response’s headers (all header names will be lowercase)
Raises ClientException: HTTP HEAD request failed
-
swiftclient.client.
head_container
(url, token, container, http_conn=None, headers=None, service_token=None)¶ Get container stats.
Parameters: - url – storage URL
- token – auth token
- container – container name to get stats for
- http_conn – HTTP connection object (If None, it will create the conn object)
- service_token – service auth token
Returns: a dict containing the response’s headers (all header names will be lowercase)
Raises ClientException: HTTP HEAD request failed
-
swiftclient.client.
head_object
(url, token, container, name, http_conn=None, service_token=None)¶ Get object info
Parameters: - url – storage URL
- token – auth token
- container – container name that the object is in
- name – object name to get info for
- http_conn – HTTP connection object (If None, it will create the conn object)
- service_token – service auth token
Returns: a dict containing the response’s headers (all header names will be lowercase)
Raises ClientException: HTTP HEAD request failed
-
swiftclient.client.
http_connection
(*arg, **kwarg)¶ Returns: tuple of (parsed url, connection object)
-
swiftclient.client.
http_log
(args, kwargs, resp, body)¶
-
swiftclient.client.
post_account
(url, token, headers, http_conn=None, response_dict=None, service_token=None)¶ Update an account’s metadata.
Parameters: - url – storage URL
- token – auth token
- headers – additional headers to include in the request
- http_conn – HTTP connection object (If None, it will create the conn object)
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Raises ClientException: HTTP POST request failed
-
swiftclient.client.
post_container
(url, token, container, headers, http_conn=None, response_dict=None, service_token=None)¶ Update a container’s metadata.
Parameters: - url – storage URL
- token – auth token
- container – container name to update
- headers – additional headers to include in the request
- http_conn – HTTP connection object (If None, it will create the conn object)
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Raises ClientException: HTTP POST request failed
-
swiftclient.client.
post_object
(url, token, container, name, headers, http_conn=None, response_dict=None, service_token=None)¶ Update object metadata
Parameters: - url – storage URL
- token – auth token
- container – container name that the object is in
- name – name of the object to update
- headers – additional headers to include in the request
- http_conn – HTTP connection object (If None, it will create the conn object)
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Raises ClientException: HTTP POST request failed
-
swiftclient.client.
put_container
(url, token, container, headers=None, http_conn=None, response_dict=None, service_token=None)¶ Create a container
Parameters: - url – storage URL
- token – auth token
- container – container name to create
- headers – additional headers to include in the request
- http_conn – HTTP connection object (If None, it will create the conn object)
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Raises ClientException: HTTP PUT request failed
-
swiftclient.client.
put_object
(url, token=None, container=None, name=None, contents=None, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, http_conn=None, proxy=None, query_string=None, response_dict=None, service_token=None)¶ Put an object
Parameters: - url – storage URL
- token – auth token; if None, no token will be sent
- container – container name that the object is in; if None, the container name is expected to be part of the url
- name – object name to put; if None, the object name is expected to be part of the url
- contents – a string, a file like object or an iterable to read object data from; if None, a zero-byte put will be done
- content_length – value to send as content-length header; also limits the amount read from contents; if None, it will be computed via the contents or chunked transfer encoding will be used
- etag – etag of contents; if None, no etag will be sent
- chunk_size – chunk size of data to write; it defaults to 65536; used only if the contents object has a ‘read’ method, e.g. file-like objects, ignored otherwise
- content_type – value to send as content-type header; if None, an empty string value will be sent
- headers – additional headers to include in the request, if any
- http_conn – HTTP connection object (If None, it will create the conn object)
- proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888‘ to set one
- query_string – if set will be appended with ‘?’ to generated path
- response_dict – an optional dictionary into which to place the response - status, reason and headers
- service_token – service auth token
Returns: etag
Raises ClientException: HTTP PUT request failed
-
swiftclient.client.
quote
(value, safe='/')¶ Patched version of urllib.quote that encodes utf8 strings before quoting. On Python 3, call directly urllib.parse.quote().
-
swiftclient.client.
store_response
(resp, response_dict)¶ store information about an operation into a dict
Parameters: - resp – an http response object containing the response headers
- response_dict – a dict into which are placed the status, reason and a dict of lower-cased headers
swiftclient.service¶
-
class
swiftclient.service.
ResultsIterator
(futures)¶ Bases:
six.Iterator
-
exception
swiftclient.service.
SwiftError
(value, container=None, obj=None, segment=None, exc=None)¶ Bases:
exceptions.Exception
-
class
swiftclient.service.
SwiftPostObject
(object_name, options=None)¶ Bases:
object
Class for specifying an object post, allowing the headers/metadata to be specified separately for each individual object.
-
class
swiftclient.service.
SwiftService
(options=None)¶ Bases:
object
Service for performing swift operations
-
capabilities
(url=None)¶ List the cluster capabilities.
Parameters: url – Proxy URL of the cluster to retrieve capabilities. Returns: A dictionary containing the capabilities of the cluster. Raises: ClientException Raises: SwiftError
-
delete
(container=None, objects=None, options=None)¶ Delete operations on an account, optional container and optional list of objects.
Parameters: - container – The container to delete or delete from.
- objects – The list of objects to delete.
- options –
A dictionary containing options to override the global options specified during the service object creation:
{ 'yes_all': False, 'leave_segments': False, }
Returns: A generator for returning the results of the delete operations. Each result yielded from the generator is either a ‘delete_container’, ‘delete_object’ or ‘delete_segment’ dictionary containing the results of an individual delete operation.
Raises: ClientException
Raises: SwiftError
-
download
(container=None, objects=None, options=None)¶ Download operations on an account, optional container and optional list of objects.
Parameters: - container – The container to download from.
- objects – A list of object names to download (a list of strings).
- options –
A dictionary containing options to override the global options specified during the service object creation:
{ 'yes_all': False, 'marker': '', 'prefix': None, 'no_download': False, 'header': [], 'skip_identical': False, 'out_directory': None, 'out_file': None, 'remove_prefix': False, 'shuffle' : False }
Returns: A generator for returning the results of the download operations. Each result yielded from the generator is a ‘download_object’ dictionary containing the results of an individual file download.
Raises: ClientException
Raises: SwiftError
-
list
(container=None, options=None)¶ List operations on an account, container.
Parameters: - container – The container to make the list operation against.
- options –
A dictionary containing options to override the global options specified during the service object creation:
{ 'long': False, 'prefix': None, 'delimiter': None, }
Returns: A generator for returning the results of the list operation on an account or container. Each result yielded from the generator is either a ‘list_account_part’ or ‘list_container_part’, containing part of the listing.
-
post
(container=None, objects=None, options=None)¶ Post operations on an account, container or list of objects
Parameters: - container – The container to make the post operation against.
- objects –
A list of object names (strings) or SwiftPostObject instances containing an object name, and an options dict (can be None) to override the options for that individual post operation:
[ 'object_name', SwiftPostObject('object_name', options={...}), ... ]
The options dict is described below.
- options –
A dictionary containing options to override the global options specified during the service object creation. These options are applied to all post operations performed by this call, unless overridden on a per object basis. Possible options are given below:
{ 'meta': [], 'headers': [], 'read_acl': None, # For containers only 'write_acl': None, # For containers only 'sync_to': None, # For containers only 'sync_key': None # For containers only }
Returns: Either a single result dictionary in the case of a post to a container/account, or an iterator for returning the results of posts to a list of objects.
Raises: SwiftError
-
stat
(container=None, objects=None, options=None)¶ Get account stats, container stats or information about a list of objects in a container.
Parameters: - container – The container to query.
- objects – A list of object paths about which to return information (a list of strings).
- options –
A dictionary containing options to override the global options specified during the service object creation. These options are applied to all stat operations performed by this call:
{ 'human': False }
Returns: Either a single dictionary containing stats about an account or container, or an iterator for returning the results of the stat operations on a list of objects.
Raises: SwiftError
-
upload
(container, objects, options=None)¶ Upload a list of objects to a given container.
Parameters: - container – The container to put the uploads into.
- objects –
A list of file/directory names (strings) or SwiftUploadObject instances containing a source for the created object, an object name, and an options dict (can be None) to override the options for that individual upload operation:
[ '/path/to/file', SwiftUploadObject('/path', object_name='obj1'), ... ]
The options dict is as described below.
The SwiftUploadObject source may be one of:
file - A file like object (with a read method) path - A string containing the path to a local fileor directoryNone - Indicates that we want an empty object
- options –
A dictionary containing options to override the global options specified during the service object creation. These options are applied to all upload operations performed by this call, unless overridden on a per object basis. Possible options are given below:
{ 'meta': [], 'header': [], 'segment_size': None, 'use_slo': False, 'segment_container': None, 'leave_segments': False, 'changed': None, 'skip_identical': False, 'fail_fast': False, 'dir_marker': False # Only for None sources }
Returns: A generator for returning the results of the uploads.
Raises: SwiftError
Raises: ClientException
-
-
class
swiftclient.service.
SwiftUploadObject
(source, object_name=None, options=None)¶ Bases:
object
Class for specifying an object upload, allowing the object source, name and options to be specified separately for each individual object.
-
swiftclient.service.
get_conn
(options)¶ Return a connection building it from the options.
-
swiftclient.service.
get_from_queue
(q, timeout=864000)¶
-
swiftclient.service.
get_future_result
(f, timeout=86400)¶
-
swiftclient.service.
interruptable_as_completed
(fs, timeout=86400)¶
-
swiftclient.service.
mkdirs
(path)¶
-
swiftclient.service.
process_options
(options)¶
-
swiftclient.service.
split_headers
(options, prefix='')¶ Splits ‘Key: Value’ strings and returns them as a dictionary.
Parameters: - options – An array of ‘Key: Value’ strings
- prefix – String to prepend to all of the keys in the dictionary. reporting.
swiftclient.exceptions¶
-
exception
swiftclient.exceptions.
ClientException
(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=0, http_reason='', http_device='', http_response_content='')¶ Bases:
exceptions.Exception
swiftclient.multithreading¶
-
class
swiftclient.multithreading.
ConnectionThreadPoolExecutor
(create_connection, max_workers)¶ Bases:
concurrent.futures.thread.ThreadPoolExecutor
A wrapper class to maintain a pool of connections alongside the thread pool. We start by creating a priority queue of connections, and each job submitted takes one of those connections (initialising if necessary) and passes it as the first arg to the executed function.
At the end of execution that connection is returned to the queue.
By using a PriorityQueue we avoid creating more connections than required. We will only create as many connections as are required concurrently.
-
submit
(fn, *args, **kwargs)¶
-
-
class
swiftclient.multithreading.
MultiThreadingManager
(create_connection, segment_threads=10, object_dd_threads=10, object_uu_threads=10, container_threads=10)¶ Bases:
object
One object to manage context for multi-threading. This should make bin/swift less error-prone and allow us to test this code.
Parameters: - segment_threads – The number of threads allocated to segment uploads
- object_dd_threads – The number of threads allocated to object download/delete jobs
- object_uu_threads – The number of threads allocated to object upload/update based jobs
- container_threads – The number of threads allocated to container/account level jobs
-
class
swiftclient.multithreading.
OutputManager
(print_stream=None, error_stream=None)¶ Bases:
object
One object to manage and provide helper functions for output.
This object is a context manager and returns itself into the context. When entering the context, two printing threads are created (see below) and they are waited on and cleaned up when exiting the context.
Also, thread-safe printing to two streams is provided. The
print_msg()
method will print to the suppliedprint_stream
(defaults tosys.stdout
) and theerror()
method will print to the suppliederror_stream
(defaults tosys.stderr
). Both of these printing methods will format the given string with any supplied*args
(a la printf). On Python 2, Unicode messages are encoded to utf8.The attribute
self.error_count
is incremented once per error message printed, so an application can tell if any worker threads encountered exceptions or otherwise callederror()
on this instance. The swift command-line tool uses this to exit non-zero if any error strings were printed.Parameters: - print_stream – The stream to which
print_msg()
sends formatted messages. - error_stream – The stream to which
error()
sends formatted messages.
On Python 2, Unicode messages are encoded to utf8.
-
DEFAULT_OFFSET
= 14¶
-
error
(msg, *fmt_args)¶
-
get_error_count
()¶
-
print_items
(items, offset=14, skip_missing=False)¶
-
print_msg
(msg, *fmt_args)¶
-
print_raw
(data)¶
-
warning
(msg, *fmt_args)¶
- print_stream – The stream to which