Amazon Driver: Performance Tuning with the AWS CLI

Note

This section applies only to the Amazon driver of the S3/Amazon Cloud Plugin. The deprecated S3 driver uses bacula-sd-cloud-s3-driver and does not read the files described here.

The Amazon driver (bacula-sd-cloud-aws-driver) performs its transfers by invoking the AWS CLI. The Cloud resource directives control what Bacula asks the CLI to do – which endpoint, which bucket, how many parts at a time – while the CLI itself decides how each individual transfer is executed: how the object is split, how many HTTP requests run in parallel, which timeouts and retries apply.

Those lower-level settings have no equivalent Bacula directive. They are read from the AWS CLI configuration file, and they can have a significant effect on throughput because the CLI defaults are tuned for transfers across the internet to AWS rather than for an on-premises object store reached over a local network.

Configuration File Location

The Storage Daemon runs as the Bacula user, normally bacula, whose home directory is /opt/bacula/working. Every AWS CLI process started by the Amazon driver inherits that environment, so the CLI looks for its configuration at:

/opt/bacula/working/.aws/config

Create the directory if it does not exist, and make sure it is owned by the Bacula user:

mkdir -p /opt/bacula/working/.aws
touch /opt/bacula/working/.aws/config
chown -R bacula:bacula /opt/bacula/working/.aws
chmod 700 /opt/bacula/working/.aws
chmod 600 /opt/bacula/working/.aws/config

Important

If the Bacula user’s home directory is not /opt/bacula/working, the file must be placed under whatever that home directory actually is. Confirm it with:

getent passwd bacula | cut -d: -f6

A configuration file placed in /root/.aws/ or in the home directory of the administrator performing the configuration will not be read by the Storage Daemon.

Warning

Do not place aws_access_key_id or aws_secret_access_key in this file. Credentials for the Amazon driver come from the Access Key and Secret Key directives of the Cloud resource. Use this file for transfer tuning only.

File Format

The settings that control transfer behavior live in a nested s3 block inside a profile. Indentation is significant: each sub-key must be indented beneath the s3 = line. Settings that control retries and timeouts are top-level keys of the profile and are placed outside the s3 block.

[default]
max_attempts = 10
retry_mode = adaptive
s3 =
  multipart_threshold = 64MB
  multipart_chunksize = 64MB
  max_concurrent_requests = 16
  max_queue_size = 10000

Note

The keys inside the s3 block can only be set in the configuration file. Unlike most AWS CLI options they have no environment-variable equivalent, which is why this file is the only way to apply them to the Amazon driver.

AWS CLI Settings and Cloud Directives

Bacula writes a Volume as a sequence of part files in the local cache, then uploads those part files as S3 objects. Two layers of splitting are therefore involved, and they are configured in different places:

Layer	Configured by	Effect
Volume split into parts	`MaximumPartSize` (Device)	Size of each object uploaded to the bucket
Number of parts in flight	`MaximumConcurrentUploads` (Cloud)	How many AWS CLI transfers run at the same time
Part split into chunks	`multipart_chunksize` (AWS CLI)	Size of each HTTP `UploadPart` request
Chunks in flight per part	`max_concurrent_requests` (AWS CLI)	Parallel requests within one part upload

The total number of parallel HTTP requests reaching the object store is therefore approximately:

MaximumConcurrentUploads  x  max_concurrent_requests

Both layers must be considered together. Raising one while ignoring the other commonly produces either an idle network or an overloaded storage target.

Relevant AWS CLI Settings

Note

The settings below are defined and maintained by AWS as part of the AWS CLI. This section only describes how they interact with the Cloud Plugin. For the full definition, accepted values and current defaults of each setting, refer to the AWS CLI documentation:

Default values and available options may change between AWS CLI versions. Check the version installed on the Storage Daemon host with aws --version.

The suggested starting points are intended for an on-premises or S3-compatible object store reached over a local network. They are a basis for measurement, not final values; see Tuning Method.

Throughput Settings

Note

Place these settings inside the s3 block.

Setting	Why it matters with Bacula	Suggested starting point
`multipart_chunksize`	Bacula part files are usually much larger than the CLI default chunk, so each part is sent as many `UploadPart` requests. On a local network the per-request overhead (signing, round trip) is often the dominant cost. Larger chunks increase memory use; see Memory Considerations.	`64MB` to `128MB`
`multipart_threshold`	Bacula part files are normally larger than the threshold, so effectively every part upload is a multipart upload. Keep it aligned with `multipart_chunksize`.	Same value as `multipart_chunksize`
`max_concurrent_requests`	Multiplies with `MaximumConcurrentUploads` to give the total number of parallel requests. Each request uses a thread, a buffer and a TCP connection. Because the AWS CLI is a Python program, raising this value has diminishing returns within one process; raising `MaximumConcurrentUploads` usually scales better.	`16`
`max_queue_size`	Raise it in proportion to `max_concurrent_requests` so worker threads are not starved.	`10000`
`addressing_style`	Must match the Cloud resource: use `path` when the Cloud resource specifies `UriStyle = Path`, which is the usual case for on-premises and S3-compatible object stores addressed by IP or by a hostname without wildcard DNS.	`path` when `UriStyle = Path`
`max_bandwidth`	Limits a single CLI process only. Prefer the Cloud resource directive Maximum Upload Bandwidth, which limits the aggregate across all devices using that Cloud resource.	Leave unset
`payload_signing_enabled`	Only relevant with `Protocol = HTTP`. With HTTPS, upload payloads are not signed by default and this setting has no practical effect. With HTTP, disabling it saves CPU but leaves the request body without any integrity protection (see warning below).	Leave unset
`preferred_transfer_client`	The `crt` client is a native library not limited by the Python Global Interpreter Lock, but it does not honor all of the settings above and has known compatibility constraints with some S3-compatible object stores, particularly with path-style addressing (see note below).	Leave unset

Warning

Setting payload_signing_enabled = false together with Protocol = HTTP means that neither TLS nor request signing protects the integrity of the uploaded data. Consider it only on a trusted, isolated storage network, and treat it as a deliberate security decision.

Note

If you evaluate preferred_transfer_client = crt, test it thoroughly against your object store, including restores, before adopting it in production. Check the AWS CLI documentation for which s3 settings the CRT client ignores and which ones it uses instead.

Memory Considerations

The AWS CLI holds approximately multipart_chunksize x max_concurrent_requests of data in memory per process, and the Amazon driver runs up to MaximumConcurrentUploads processes concurrently. The worst-case footprint is therefore:

multipart_chunksize  x  max_concurrent_requests  x  MaximumConcurrentUploads

With multipart_chunksize = 64MB, max_concurrent_requests = 16 and MaximumConcurrentUploads = 4, this is approximately 4 GB of RAM used by transfers alone, in addition to the memory required by the Storage Daemon itself.

Always verify this calculation against the memory available on the Storage Daemon host before adopting a tuned configuration. A configuration that performs well in a short test can cause memory pressure when several jobs run concurrently during the backup window.

Complete Example

A Storage Daemon writing to an on-premises S3-compatible object store over a local network using HTTPS, with the corresponding Bacula resources shown for context:

# /opt/bacula/working/.aws/config
# Owned by the Bacula user. No credentials here: they come from the
# Access Key and Secret Key directives of the Cloud resource.

[default]
max_attempts = 10
retry_mode = adaptive
connect_timeout = 60
read_timeout = 120
tcp_keepalive = true
s3 =
  addressing_style = path
  multipart_threshold = 64MB
  multipart_chunksize = 64MB
  max_concurrent_requests = 16
  max_queue_size = 10000

Cloud {
  Name = MyCloud
  Driver = "Amazon"
  HostName = "objectstore.example.com"
  BlobEndpoint = "https://objectstore.example.com"
  BucketName = "bacula-backup"
  AccessKey = "..."
  SecretKey = "..."
  Protocol = HTTPS
  UriStyle = Path
  Upload = EachPart
  TruncateCache = AfterUpload
  MaximumConcurrentUploads = 4
  MaximumConcurrentDownloads = 4
}

Device {
  Name = CloudDevice0
  DeviceType = Cloud
  Cloud = MyCloud
  ArchiveDevice = /mnt/bacula-cache
  MaximumPartSize = 5GB
  MaximumFileSize = 5GB
  MaximumConcurrentJobs = 1
  MediaType = CloudType
  LabelMedia = yes
  RandomAccess = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = no
}

Note

With MaximumPartSize = 5GB, make sure the local cache filesystem can hold at least MaximumConcurrentUploads part files in addition to the data being written. Larger part files also mean that a restore downloads more data from the object store, since part files are retrieved whole.

Configuration Verification

The AWS CLI reports which settings it has loaded and where each value came from. Run it as the Bacula user so that the same home directory is used:

su - bacula -s /bin/bash -c "aws configure list"

To confirm that the file is syntactically valid and being read, check a single value:

su - bacula -s /bin/bash -c "aws configure get s3.multipart_chunksize"

If this returns nothing, the file is not being read. The usual causes are a home directory other than /opt/bacula/working, incorrect ownership, or sub-keys of the s3 block that are not indented.

After changing the file, restart the Storage Daemon so that new transfers pick up the settings:

systemctl restart bacula-sd

Tuning Method

Optimal values depend on the object store, the network and the cache filesystem, so they should be measured rather than assumed. A practical sequence is:

Establish a baseline with the current configuration, using status storage and the Cloud statistics described in Cloud Plugin: Status Storage and Cloud Statistics Explained.
Measure the read throughput of the cache filesystem. If the cache cannot be read faster than the object store can accept data, no amount of transfer tuning will help.
Vary one layer at a time. Test MaximumPartSize and MaximumConcurrentUploads first, since these govern how Bacula presents work to the CLI, then tune multipart_chunksize and max_concurrent_requests.
Watch for throughput that decreases as concurrency rises. This indicates contention inside the storage target rather than a client limitation, and in that situation lowering MaximumConcurrentUploads can increase total throughput.
Validate the chosen settings with a real backup job, and confirm that a restore from the resulting Volume completes correctly.

Tip

If you need assistance with fine-tuning backup performance using the Amazon driver of the S3/Amazon Cloud Plugin, please contact Bacula Systems Support. Support can also supply a collection script that performs this measurement sequence automatically and produces a report with suggested values.