Best practices¶
Delivery limits¶
The ingestion system enforces the following limits per delivery:
Maximum number of files to upload: 10.000. It counts the files of all the upload operations of the delivery. Deletes are not counted.
Maximum size of the delivery: 200 GB. It is the sum of the size of all the files to upload.
Maximum size of a single file: 50 GB. Larger files cannot be processed.
Note
These are the current values. They might change, so if you need to deliver more, please contact the MDS Service Desk.
Note
Sizes are counted in 1024-based units: 1 GB = 1,073,741,824 bytes. Some tools, such as the macOS Finder, use 1000-based units and show bigger numbers for the same file. For example, a file of 45 GB here is shown as 48.3 GB in the Finder.
What happens if a limit is exceeded¶
The limits are checked as part of the validation, before anything is uploaded. If one is exceeded, nothing is uploaded and no delivery is submitted. It is also checked with --dry-run. The error looks like:
Invalid delivery size error: Invalid delivery limit exceeded: Maximum number of files exceeded. Reason: The number of files in delivery (200) exceeded the limit (10.000)
Invalid delivery size error: Invalid delivery limit exceeded: Maximum delivery size exceeded. Reason: The delivery size (210.0 GB) exceeded the limit (200.0 GB)
Invalid delivery size error: Invalid delivery limit exceeded: Maximum file size exceeded. Reason: One or more files exceeded the maximum file size limit (40.0 GB): {'data/big_file.nc': 46080}
For the file size, each file is listed with its size in MB. In Python it is raised as InvalidDeliverySizeError.
The ingestion system checks the limits again when the delivery is submitted. In that case the delivery is rejected with a message like:
HTTP error: Manifest validation failed: The total number of files to upload (200) exceeds the limit: 10.000
HTTP error: Manifest validation failed: Delivery size 210.0 GB is bigger than the server limits: 50.0 GB
If a file is bigger than the maximum size, the upload operation is not processed and all its files fail with Cannot process upload operation: some files exceed the disk allowance. You can see it with the status command and --show-all.
Recommendations¶
Do not go over the following values. They leave some margin below the limits:
Limit |
Recommended maximum |
|
|---|---|---|
Files to upload per delivery |
10.000 |
10.000 |
Size of a delivery |
200 GB |
190 GB (204,010,946,560 bytes) |
Size of a single file |
50 GB |
40 GB (42,949,672,960 bytes) |
Other recommendations:
Split large deliveries into several deliveries, each within the recommended maximums. To know how many you need, calculate it twice and keep the larger number:
deliveries needed by number of files = total number of files / maximum number of files deliveries needed by size = total size / maximum size of a delivery
For example, for a given release with 400 files of 2 GB each (800 GB in total), the above formulas result:
By number of files: 400 / 135 = 2.96, so 3 deliveries.
By size: 800 / 190 = 4.21, so 5 deliveries.
You need 5 deliveries, the larger number. Each one has at most 80 files (160 GB).
Keep operations that depend on each other in the same delivery. The operations of a delivery run in order, but separate deliveries are processed in parallel. For example, a delete followed by an upload of the same file must be in the same delivery. Remember that a delivery is for a single dataset.
Join small files into bigger ones, if your data format allows it. Files smaller than about 1.4 GB (190 GB / 135 files) reach the maximum number of files before the maximum size, so you can send less data per delivery. Bigger files reach the maximum size first.
Adjust the upload concurrency to the shape of your delivery. By default, 5 files are uploaded in parallel. If your delivery has many small files (for example, more than 5.000), the upload time is dominated by the per-file overhead, not by the transfer. In that case, you can increase
--max-concurrent-uploads(max_concurrent_uploadsin Python) to something like 10 or 15 to speed it up, as long as your environment allows it. Very high values can exhaust the network resources of your machine (for example, with errors likeCan't assign requested address).Large files take longer. To protect the disk of the ingestion system, the bigger the files of an upload operation, the fewer of them are processed at the same time. It depends on the median size of the files of the operation (not the biggest one):
files processed at the same time = maximum size of a single file / median file size
Round it down, with a minimum of 1 and a maximum of 10. Some examples:
Median file size of the operation
Files processed at the same time
4 GB or less
10
7 GB
5
10 GB
4
20 GB
2
40 GB
1
Deliveries made of very large files can therefore take a long time to complete.