API reference¶
All the objects below are importable from copernicusmarine_delivery.
Operations¶
- pydantic model copernicusmarine_delivery.Upload¶
Upload
filesto the given dataset and product.- Parameters:
files – Relative or absolute path to local files. See Files paths and anchors in the documentation.
max_concurrent_uploads – The maximum number of parallel threads that will be used to upload files defined in the files attribute. Defaults to 5.
chunk_size_mb – The chunk size (in MB) in which the files will be split into for multipart uploads. Defaults to 16 MB.
chunk_concurrency – The number of chunks per file that will be uploaded in parallel in multipart uploads. Defaults to 6.
dry_run – Validate the upload without performing any actual operation in S3.
- add(file: str) None¶
Add a file to the operation.
- Parameters:
file – Path of the file. See File paths and anchors in the documentation.
- submit(product_id: str, dataset_id: str, anchor: str | None = None, raise_on_upload_error: bool = False, max_concurrent_uploads: int = 5, chunk_size_mb: int = 16, chunk_concurrency: int = 6, dry_run: bool = False, disable_progress_bar: bool = False, staging: bool = False, **kwargs) ResponseUpload¶
Perform the upload.
- Parameters:
product_id – The ID of the product.
dataset_id – The ID of the dataset.
anchor – Optional anchor to convert the local file path to an S3 suffix. See File paths and anchors in the documentation.
raise_on_upload_error – If True, raise an exception and stop the upload if any file fails to upload. By default, the upload will continue and skip any files that fail to upload.
max_concurrent_uploads – The maximum number of parallel threads that will be used to upload files defined in the operations attribute. Defaults to 5.
chunk_size_mb – The chunk size (in MB) in which the files will be split into for multipart uploads. Defaults to 16 MB.
chunk_concurrency – The number of chunks per file that will be uploaded in parallel in multipart uploads. Defaults to 6.
staging – Use the staging environment instead of production.
- Returns:
ResponseUploadobject containing the result of the upload.
- field anchor: str | None = None¶
- field files: list[str] [Required]¶
- field operation: Literal[OperationNames.upload] = OperationNames.upload¶
- pydantic model copernicusmarine_delivery.Delete¶
Delete
filesfrom the given dataset and product.- Parameters:
files – Files’ Key suffix in S3 without product_id/dataset_id in the folder structure. See File paths and anchors in the documentation for more details.
dry_run – Validate the delete without performing any actual operation in S3.
- add(file: str) None¶
Add a file to the operation.
- Parameters:
file – Path of the file. See File paths and anchors in the documentation.
- submit(product_id: str, dataset_id: str, dry_run: bool = False, staging: bool = False, **kwargs) ResponseDelete¶
Perform the delete.
- Parameters:
product_id – The ID of the product.
dataset_id – The ID of the dataset.
staging – Use the staging environment instead of production.
- Returns:
ResponseDeleteobject containing the result of the delete.
- field files: list[str] [Required]¶
- field operation: Literal[OperationNames.delete] = OperationNames.delete¶
- pydantic model copernicusmarine_delivery.Delivery¶
Creates a delivery. A delivery is a set of operations (upload or delete) to be performed on a dataset.
Each operation will be performed sequentially in the order they are given. Each operation can have multiple sources (files) to be processed.
- Parameters:
operations – A list of operations to be performed. Each operation is a tuple with the operation name as the first element and a list of sources as the second element. Available operations are ‘upload’ and ‘delete’.
max_concurrent_uploads – The maximum number of parallel threads that will be used to upload files defined in the operations attribute. Defaults to 5.
chunk_size_mb – The chunk size (in MB) in which the files will be split into for multipart uploads. Defaults to 16 MB.
chunk_concurrency – The number of chunks per file that will be uploaded in parallel in multipart uploads. Defaults to 6.
dry_run – Validate the delivery without performing any actual operation in S3.
- add(operation: Upload | Delete) None¶
Add an operation to the delivery. Operations are performed in the order they are added.
- submit(product_id: str, dataset_id: str, anchor: str | None = None, raise_on_upload_error: bool = False, max_concurrent_uploads: int = 5, chunk_size_mb: int = 16, chunk_concurrency: int = 6, dry_run: bool = False, disable_progress_bar: bool = False, staging: bool = False, **kwargs) ResponseDelivery¶
Perform the delivery by executing all added operations sequentially.
- Parameters:
product_id – The ID of the product.
dataset_id – The ID of the dataset.
anchor – Optional anchor to convert the local file path to an S3 suffix. See File paths and anchors in the documentation for more details.
raise_on_upload_error – If True, raise an exception and stop the delivery if any file fails to upload. By default, the delivery will continue and skip any files that fail to upload.
max_concurrent_uploads – The maximum number of parallel threads that will be used to upload files defined in the operations attribute. Defaults to 5.
chunk_size_mb – The chunk size (in MB) in which the files will be split into for multipart uploads. Defaults to 16 MB.
chunk_concurrency – The number of chunks per file that will be uploaded in parallel in multipart uploads. Defaults to 6.
staging – Use the staging environment instead of production.
- Returns:
ResponseDeliveryobject containing the result of the delivery.
Delivery status¶
- copernicusmarine_delivery.delivery_status(delivery_id: str, on_behalf_of: str | None = None, staging: bool = False) Delivery¶
Get the status of a delivery.
- Parameters:
delivery_id – The ID of the delivery to check.
on_behalf_of – For internal usage only.
staging – Use the staging environment instead of production.
- Returns:
DeliveryModel object representing the status of the delivery.
- copernicusmarine_delivery.list_deliveries(timeframe: Literal['s', 'm', 'h', 'd', 'w', 'mo', 'y', 'all'] = 'mo', on_behalf_of: str | None = None, staging: bool = False) list[Delivery]¶
List all deliveries for a given pushing entity.
- Parameters:
timeframe – The timeframe to filter deliveries by. Defaults to “mo” (last month).
staging – Use the staging environment instead of production.
- Returns:
A list of DeliveryModel objects sorted by delivery_id in descending order (most recent first).
Models¶
- pydantic model copernicusmarine_delivery.ResponseUpload¶
Metadata returned when using
upload()- field dataset_id: str | None = None¶
Dataset ID.
- field delivery_id: str | None = None¶
Delivery ID.
- field dry_run: bool | None = None¶
Dry run flag, if True, no actual operation was performed in S3.
- field fatal_error: str | None = None¶
Any error that may prematurely stop the delivery.
- field files_failed: list[ErrorFile] [Optional]¶
Potential I/O errors, might be on the user side, not necessarily user fault
- field files_uploaded: list[tuple[Path, S3KeySuffix]] [Optional]¶
Successful uploaded file names
- field product_id: str | None = None¶
Product ID.
- field pushing_entity_id: str | None = None¶
Pushing entity ID.
- pydantic model copernicusmarine_delivery.ResponseDelete¶
Metadata returned when using
delete()- field dataset_id: str | None = None¶
Dataset ID.
- field delivery_id: str | None = None¶
Delivery ID.
- field dry_run: bool | None = None¶
Dry run flag, if True, no actual operation was performed in S3.
- field fatal_error: str | None = None¶
Any error that may prematurely stop the delivery.
- field files_to_delete: list[DeleteFile] [Optional]¶
- field product_id: str | None = None¶
Product ID.
- field pushing_entity_id: str | None = None¶
Pushing entity ID.
- pydantic model copernicusmarine_delivery.ResponseDelivery¶
Metadata returned when using
delivery()- field dataset_id: str | None = None¶
Dataset ID.
- field delivery_id: str | None = None¶
Delivery ID.
- field dry_run: bool | None = None¶
Dry run flag, if True, no actual operation was performed in S3.
- field fatal_error: str | None = None¶
Any error that may prematurely stop the delivery.
- field operations_responses: list[ResponseUpload | ResponseDelete] [Optional]¶
List of responses for each operation in the delivery.
- field product_id: str | None = None¶
Product ID.
- field pushing_entity_id: str | None = None¶
Pushing entity ID.
- copernicusmarine_delivery.DeliveryModel¶
alias of
Delivery
Exceptions¶
- exception copernicusmarine_delivery.AuthenticationError¶
Authentication Error is raised if the token response from the Keycloak service returns >= 400
- exception copernicusmarine_delivery.InvalidDeliveryIdsError(message: str)¶
Raised when one or more IDs in the delivery are not expected.
- exception copernicusmarine_delivery.InvalidDeliverySizeError(limit_raised: str, details: str)¶
Raised when either the number of files, or a file size or the total delivery size exceed the limits.
- exception copernicusmarine_delivery.InvalidFilesError(invalid_files: list[InvalidFile])¶
Raised when an error occurs during the validation of files. Will list the files and their errors that failed validation.
- exception copernicusmarine_delivery.NoSuccessfulUploadsError(error_files: list[ErrorFile])¶
Raised when an error occurs during the upload process. Will list the files and their errors that failed to upload.