High CPU usage for "sa-learn" and "spamtrain" processes in a server with Plesk

Stefan Yakubov

Updated August 02, 2021 06:53

Plesk for Linux kb: technical ABT: Group A

Applicable to:

Plesk for Linux

Symptoms

High CPU usage for sa-learn and spamtrain processes in the Plesk server.

Cause

Since the sa-learn script is used by spamassassin to learn the specific properties of spam from a folder that supposedly contains only spam emails.
CPU usage of the sa-learn process depends on the number of emails in mailboxes and their size. In turn, if the number of emails is huge - CPU over usage is expected.

Resolution

To work around the issue, use one of the following solutions:

Create a wrapper script that will launch sa-learn with low priority:

Connect to the server using SSH.
Use nice or ionice utilities to decrease the priority of sa-update from the cron tasks.
Create a wrapper file which will execute sa-learn with the lowest CPU priority:

# cp -a /usr/bin/sa-learn{,.orig}
# echo "/bin/nice -19 /usr/bin/perl -T -w /usr/bin/sa-learn.orig \$@" > /usr/bin/sa-learn
# chmod +x /usr/bin/sa-learn

Note: In Debian based OS the correct path is /usr/bin/nice

Lower the priority of the currently running processes:

Set the lowest priority to the currently running sa-learn processes execute the following:

# ps auxwf | grep -v grep | grep sa-learn | awk {'print $2'} | xargs -i renice 19 {}

Note: Process priority values range from -20 to 19. A process with the nice value of -20 is considered to be on top of the priority. And a process with nice value of 19 is considered to be low on the priority list.

Disable the sa-update functionality:

Disable the sa-update execution by moving 60sa-update and spamassassin files from /etc/cron.daily to a different location:

# mv /etc/cron.daily/60sa-update /root

Comments

10 comments

Alexander Bien October 24, 2018 11:56

Are you sure that we need to modify 60sa-update in order to change high cpu from sa-learn?

This pstreelog output (C7, P17.8.x) would suggest otherwise:

|-anacron---run-parts-+-50plesk-daily---sw-engine---sw-engine---spamtrain---su---sa-learn

I couldnt find any mentioning of sa-learn in sa-update either. Please advise.

0

Comment actions Permalink
Robert Asilbekov November 28, 2018 06:47

@Alexander Bien

Thank you for the feedback. The article has been updated.

0

Comment actions Permalink
Peter Debik November 29, 2019 11:57

On CentOS

echo "/bin/nice -19 /usr/bin/perl -T -w /usr/bin/sa-learn.orig $@" > /usr/bin/sa-learn

must be executed as

echo "/bin/nice -19 /usr/bin/perl -T -w /usr/bin/sa-learn.orig \$@" > /usr/bin/sa-learn

because else, the "$@" will not be exported into /usr/bin/sa-learn.

0

Comment actions Permalink
Anton Maslov November 29, 2019 20:15

Peter, article updated! Thank you for report!

0

Comment actions Permalink
Bbennett January 20, 2020 10:09

Trying to disable and get the error:

mv /etc/cron.daily/60sa-update /root
mv: cannot stat ‘/etc/cron.daily/60sa-update’: No such file or directory

0

Comment actions Permalink
Ivan Postnikov January 27, 2020 14:49

Hello Bbennett

Did you execute this command as a root user?

The 60sa-update exists on my test server and the issue wasn't reproduced:

# stat /etc/cron.daily/60sa-update
File: '/etc/cron.daily/60sa-update'
Size: 448 Blocks: 8 IO Block: 4096 regular file
Device: 40f0b681h/1089517185d Inode: 409504 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-01-23 13:44:39.000000000 +0700
Modify: 2020-01-23 13:44:39.000000000 +0700
Change: 2020-01-27 20:52:40.452451535 +0700
Birth: -

# plesk -v
Product version: Plesk Obsidian 18.0.23.1
OS version: CentOS 7.5.1804 x86_64

0

Comment actions Permalink
Jair Cueva Junior June 04, 2020 12:50

In Ubuntu 18.04 this wrapper scripts was failing due to a slightly difference in the path of "nice" command, it should be: /usr/bin/nice

Thanks for exposing this approach, it has opened my mind for possibilities!

0

Comment actions Permalink
Nelson Leiva June 16, 2020 23:30

Hi Jair Cueva Junior,

The article has been updated. Thank you for noticing that!

0

Comment actions Permalink

Michael Fryd July 02, 2020 03:12

When importing large email accounts for new clients, my server was bogged down, even with sa-learn running with low priority.

We don't see single instances of sa-learn using a lot of resources, we see many instances of sa-learn running sequentially.

My solution was to write a script that checks the system load average. If the load is getting high, it pauses before running sa-learn. This helps keep the load from getting too high. If the load doesn't come down, or the load is too high, it doesn't bother running sa-learn. This seems to be OK. I suspect the message that were skipped will be examined on a subsequent run.

Here's my replacement for /usr/bin/sa-learn:

#!/bin/bash
#
# replacement for /usr/bin/sa-learn
# hack to keep sa-learn process from using too many resources
# created 6/26/2020
#
# PLESK periodically calls /usr/bin/sa-learn to train spam assassin on recent mail.
# if large amounts of new mail are present (perhaps a large external email
# account was recently imported) the i/o burden can bog down the server. We have
# seen load averages above 300.
#
# our solution is multi-pronged.
# first, we simply don't run sa-learn if the load avergage is too high.
# secondly, if the load average is starting to get high, we pause a little bit before
# starting sa-learn
# finally, if we do start sa-learn, we use nice to run it at a low priority.
#
# We have moved the original sa-learn script to /usr/bin/sa-learn.orig
#
# Note: we truncate the load average and look at only the integer portion.
#
# -Michael Fryd (michael@fryd.com) 6/26/2020
#


PUNT_LOAD_LIMIT=4 # If the load is higher than this we exit and don't run sa-learn.orig
WAIT_LOAD_LIMIT=3 # if the load is higher than this, but below PUNT_LOAD_LIMIT, we delay before running
SLEEP_TIME=0.4    # how long to delay (in seconds)
MAX_ATTEMPTS=2    # max number of times to wait for load to come down before punting


count=0
while [ "$count" -lt "$MAX_ATTEMPTS" ]; do 
    # get the integer portion of the current 1 minute load average
    load=$(uptime | awk -F: '{print $5}' | awk -F"," '{print $1}' | tr -d '[:space:]' | cut -c1) # Poll Load

    # if the load is greater than the punt limit, give up. Don't wait, and no
    # additional passes
    if [[ "$load" -gt "$PUNT_LOAD_LIMIT" ]]; then
        exit 0; # give up, don't run sa-learn.orig at this time
    fi

    # if the load is low, then run sa-learn.orig at low priority, otherwise pause, then loop again
    if [[ "$load" -lt "$WAIT_LOAD_LIMIT" ]]; then
        /bin/nice -n 19 /usr/bin/perl -T -w /usr/bin/sa-learn.orig $@
        exit 0; # If successful
    else
        sleep $SLEEP_TIME;
    fi;
    (( count++ ))
done;

Francisco Roman Garcia Rodriguez July 16, 2020 12:06

Hi Michael Fryd,

Thanks for such useful script whenever a mail importing is happening, it may be useful for other users.

0

Comment actions Permalink

Please sign in to leave a comment.

Have more questions? Submit a request

Articles in this section

Applicable to:

Symptoms

Cause

Resolution

Related articles