Quick Start

Quick guide to set up and test the Copernicus Marine Delivery tool, using mostly the command line interface (CLI).

See the other pages for more detailed documentation.

Get An Account

You need a Copernicus Marine account configured as your Pushing Entity.

Please, contact the MDS Service Desk to arrange it.

Once ready, load the account’s credentials into environment variables:

export COPERNICUSMARINE_SERVICE_USERNAME="my-pe-account"
export COPERNICUSMARINE_SERVICE_PASSWORD="*************"

For more details, see the page about credentials.

Install

Install via pip and check that the installation was successful:

pip install copernicusmarine-delivery
copernicusmarine-delivery --help

For installation alternatives, see the installation guide.

Staging environment

To prevent impacting the official MDS catalogue with the tests of the Copernicus Marine Delivery tool, a dedicated staging environment has been setup. Use the --staging flag to deliver the data to this staging environment:

copernicusmarine-delivery --staging <OTHER-OPTIONS>

Note

All commands below use the --staging flag (staging=True in Python) to run against this staging environment. Remove it to run against production.

Similarly, the copernicusmarine toolbox also has the --staging flag. Hence, the data you upload tot he MDS staging environment can be listed with copernicusmarine get --staging --dry-run <OTHER-OPTIONS>

Test Access

You might want to check first that you can access the ingestion service and that you have the right credentials.

Run this command (that lists previous deliveries) to check that the account and the connection work:

copernicusmarine-delivery list-deliveries --staging

If there is an error, please contact the MDS Service Desk.

If the output is No deliveries found. or a list of deliveries, it means that the account is recognised as a Pushing Entity.

Simple Upload

Create or select a file you have locally and upload it:

echo "hello" > test.nc
copernicusmarine-delivery upload --product-id <MY_PRODUCT> --dataset-id <MY_DATASET_WITH_TAG> --source test.nc --staging

Use a real Copernicus Marine product and dataset that belong to your Pushing Entity.

[!NOTE] Refer to the paragraph about file paths in the concepts page to make sure you can map your local folder structure to the S3 MDS structure for your datasets.

[!WARNING]
At this moment, delivered data is not published in the operational MDS, to facilitate testing. Hence, the data you upload will not be visible through the copernicusmarine toolbox.

[!WARNING]
By using the --staging flag, delivered data is not published in the operational MDS.

Check the delivery status

Check the status of the previous delivery:

copernicusmarine-delivery status --delivery-id <DELIVERY-ID> --staging

Where DELIVERY-ID is the one from the output of the upload/delete/delivery command. You can also list all deliveries to find the relevant delivery ID:

copernicusmarine-delivery list-deliveries --staging

More complex delivery

Create a delivery YAML file (delivery-test.yml) with the following structure:

# delivery-test.yml
delivery:
  - operation: delete
    files:
      - test.nc
  - operation: upload
    files:
      - my/local/files/file_01.nc
      - my/local/files/file_02.nc

This delivery will delete the file uploaded in the previous step and add 2 new files, preserving their subfolder structure.

The file has one entry per operation, run in the order written: delete takes the S3 MDS path of files already in MDS, upload takes local paths. See the delivery file format for all fields.

Make sure the new files exist. Otherwise, create them as follows:

mkdir -p my/local/files/
echo "hello 1" > my/local/files/file_01.nc
echo "hello 2" > my/local/files/file_02.nc

Submit the delivery

copernicusmarine-delivery delivery --product-id <MY_PRODUCT> --dataset-id <MY_DATASET_WITH_TAG> --file delivery-test.yml --staging

Finally, check the status of this delivery:

copernicusmarine-delivery status --delivery-id <DELIVERY-ID> --staging

Using an anchor

If your local folders do not match the MDS structure, add an anchor to the upload: the folder name after which the structure is kept. For instance, with local files in /data/exports/my_dataset/2024/file_01.nc:

  - operation: upload
    anchor: my_dataset
    files:
      - /data/exports/my_dataset/2024/file_01.nc

The file is published as <MY_PRODUCT>/<MY_DATASET_WITH_TAG>/2024/file_01.nc: everything up to and including my_dataset is dropped. See using the anchor for the rules.

Python snippet

The same delivery done in Python would look like this:

from copernicusmarine_delivery import Delivery, Upload, Delete, delivery_status

delivery = Delivery(
    operations=[
        Delete(files=["test.nc"]),
        Upload(files=["my/local/files/file_01.nc", "my/local/files/file_02.nc"]),
    ]
)

response = delivery.submit(product_id="<MY_PRODUCT>", dataset_id="<MY_DATASET_WITH_TAG>", staging=True)

if not response.fatal_error:
    print(f"Delivery successful with delivery ID: {response.delivery_id}")
else:
    print(f"Delivery failed: {response.fatal_error}")

status = delivery_status(response.delivery_id, staging=True)

print(f"Delivery status: {status.model_dump_json(indent=2)}")

Further reading

Now that the initial setup and test are done, please follow the rest of the documentation to understand how the Copernicus Marine Delivery tool can be used for your use case.