How to set up log storage for a long term in Plesk?

By default, this script collects all known logs of the server, Plesk, and websites. You can view the list of the collected logs right in the script and modify it in a configuration file if required.

To get a list of available options, run the script with the --help option:

# ./archive-logs.sh --help
Usage: archive-logs [OPTIONS]
Collect and transfer logs for long-term storage.

-c, --config CONFIG
Use this config (default: 'config.sh' in the directory
of the script). Config defines where to upload logs,
how to do it, which additional logs to include, and
some other settings. Use '--show-config-sample' to
obtain a sample. Do not make the config world-writable.

-l, --log-level LEVEL
Set logging level, one of: 'ERROR', 'WARNING', 'INFO',
'DEBUG' (default: 'WARNING'). Logs are written to
stdout, additional error output from children and
the script errors may appear on stderr.

-n, --dry-run
Simulate operation, don't create or transfer files

-s, --show-config-sample
Print config.sh sample and exit

-h, --help
Display this help and exit

With default config this will simply copy all logs to '/tmp/cold-storage-example'.

To customize the script, create a configuration file using the sample:

# ./archive-logs.sh -s > config.sh

First, the script will collect all logs that match LOG_GLOBS (except for the logs that match LOG_IGNORE_PATTERNS) to STAGING_DIR. Then it will call TRANSPORT_CMD STAGING_DIRTRANSPORT_DEST to transfer the collected logs from the staging directory to the configured storage.

When collecting logs that are to be copied, COPY_STRATEGY will be used (which should be accounted for by TRANSPORT_CMD).

When collecting logs that are to be compressed, COMPRESS_CMD and COMPRESS_EXT will be used. COMPRESSED_EXTENSIONS will also be used to recognize already compressed logs.

You can adjust the script's behavior by overriding the transport(), compress(), and log_filter() functions.

Rotated logs are compressed and suffixed with the UTC timestamp of their modification time (mtime) to ensure their names are reasonably unique. Non-rotated logs are typically left uncompressed.

The script does not change any actual logs on the server itself, but only copies them or creates new compressed versions in the staging directory.

To run the script with the minimal configuration, specify TRANSPORT_DEST ( the storage where logs will be stored).

You can also choose the desired utility that will be used to copy the logs; f.e. cp, rsync, scp, etc.

CONFIG_TEXT: ## Command to use as transport from staging to storage.
## Typical examples: cp, rsync, scp.
## The command should accept two arguments: staging directory and destination (see below).
## The command should account for COPY_STRATEGY (see below).
#TRANSPORT_CMD=(cp -arT)

## Transport destination (the storage). Should be understandable by the TRANSPORT_CMD above.
#TRANSPORT_DEST="/tmp/cold-storage-example"

Run the script:

# ./archive-logs.sh -c config.sh -l debug

CONFIG_TEXT: # copy to mounted storage (all logs on one partition)
TRANSPORT_CMD=(cp -arT)
TRANSPORT_DEST="/mnt/storage"
COPY_STRATEGY="hardlink"

CONFIG_TEXT: # copy to mounted storage (/var/www/vhosts is on a separate partition)
TRANSPORT_CMD=(cp -arTL)
TRANSPORT_DEST="/mnt/storage"
COPY_STRATEGY="link"

CONFIG_TEXT: # sync to remote storage
TRANSPORT_CMD=(rsync -avi)
TRANSPORT_DEST="root@storage.example.net:/path/to/storage"

CONFIG_TEXT: # sync to remote storage and don't create /path/to/storage/archive-logs sub-directory at destination
TRANSPORT_CMD=(rsync -avi)
TRANSPORT_DEST="root@storage.example.net:/path/to/storage"
transport()
{
for src in "$@"; do
"${TRANSPORT_CMD[@]}" "$src/" "$TRANSPORT_DEST"
done
}

1. Make sure your TRANSPORT_CMD does not remove files missing in STORAGE_DIR at TRANSPORT_DEST, but only adds or updates existing ones.
2. Using rsync or a similar tool as transport provides optimal updates to the storage and minimizes the traffic and transfer time.
3. If your transport tool fails on copying logs that are being written to, use COPY_STRATEGY="copy" instead.