Articles in this section

See more

[FIXED BUG] [BUG ]Let's Encrypt challenge files are not removed from .well-known/acme-challenge/ directories

Avatar
Marc López
Updated
Follow
Plesk for Linux kb: fixed kb: bug ext: le

Applicable to:

  • Plesk for Linux

Applicable to:

  • Plesk for Linux

Symptoms

  • Folders /var/www/vhosts/default/htdocs/.well-known/acme-challenge/ and/or /usr/share/psa-horde/.well-known/acme-challenge/, as well as websites' document root folders (like /var/www/vhosts/example.com/httpdocs/.well-known/acme-challenge) are becoming populated with an indefinite number of files.

  • Cleanup is not explicitly disabled in /usr/local/psa/var/modules/letsencrypt/cli.ini Let's Encrypt configuration file; the following command returns empty output:

    # cat /usr/local/psa/var/modules/letsencrypt/cli.ini | grep 'disable-cleanup'

Cause

Product issue:

  • #EXTLETSENC-676 "Challenge token files (which are created after certificates failed to be renewed or issued) are now deleted after 3 months. The extension’s folders are no longer cluttered."
    Fixed in:

Resolution

Please consider updating your server:

Workaround

If update is not possible for some reason you may try the following

workaround

These files can be safely removed manually:

  1. Connect to the server via SSH.

  2. Remove the files from the default website and webmail directories:

    # rm -rf /var/www/vhosts/default/htdocs/.well-known/acme-challenge/*
    # rm -rf /usr/share/psa-horde/.well-known/acme-challenge/*

    Note: In case there are thousands of files, the following commands can be used to empty the folders faster:

    # mkdir /empty_dir
    # rsync -a --delete /empty_dir/ /var/www/vhosts/default/htdocs/.well-known/acme-challenge/
    # rsync -a --delete /empty_dir/ /usr/share/psa-horde/.well-known/acme-challenge/

  3. Get a list of the document root directories for all domains and put it into domains_root_dir.list file using the following command:

    # plesk db -NBe "select www_root from hosting" > domains_root_dir.list

  4. Remove files from these directories using the following command:

    # cat domains_root_dir.list | while read i; do echo $i; rm -rf $i/.well-known/acme-challenge/*; done

    Note: In case there command above fails due to the amount of files, the following command can be used instead:

    # cat domains_root_dir.list | while read i; do echo $i; rsync -a --delete /empty_dir/ $i/.well-known/acme-challenge/; done 

Additionally, these commands can be set up as daily Scheduled Tasks. See the following documentation for instructions on setting up tasks: Scheduling Tasks.

Comments

10 comments

Sort by Date Votes
  • Avatar
    Websavers Inc

    Does the fix included in version 2.8.0 of the extension clean up existing acme-challenge files, or only ones that were created *after* the update?

    0
    Comment actions Permalink
  • Avatar
    Ekaterina Babenko

    Websavers Inc, hi! Thanks for your question!

    After fix applied, it will clean up existing files, which are not older than 3 months.

    Only the oldest one need to be removed manually.

    0
    Comment actions Permalink
  • Avatar
    Jonas Frey

    It seems the cleanup (for old files from prior the upgrade to 2.8.0) does not work. I have tested this on CentOS 6 and 7 (with Plesk 17.8) and it did not clean up the existing old files (even not after 1 week).

     

    0
    Comment actions Permalink
  • Avatar
    Denis Bykov

    @Jonas Frey

    I can explain it if permissions or ownership of those files was changed. In that case, I can suggest removing them manually.
    If there are some further issues, consider contacting regular Plesk support.

    0
    Comment actions Permalink
  • Avatar
    Websavers Inc (Edited )

    We didn't find many cases where the argument list wasn't too long for rm to work, so the following (only slightly modified) ones did the trick:

    mkdir /root/empty_dir
    rsync -a --delete /root/empty_dir/ /var/www/vhosts/default/htdocs/.well-known/acme-challenge/
    rsync -a --delete /root/empty_dir/ /usr/share/psa-horde/.well-known/acme-challenge/
    plesk db -NBe "select www_root from hosting" > /root/domains_root_dir.list
    cat /root/domains_root_dir.list | while read i; do [[ -d "$i/.well-known/acme-challenge/" ]] && echo $i && rsync -a --delete /root/empty_dir/ $i/.well-known/acme-challenge/; done
    rm -rf /root/empty_dir/ /root/domains_root_dir.list

     

    0
    Comment actions Permalink
  • Avatar
    burnleyvic

    Just stumbled on this article after a major clean up on our Linux plesk servers, littered with millions of such stale files. Now running Plesk Obsidian
    Version 18.0.23 Update #3 with Let's Encrypt extension Version 2.8.6-571 on all of them since 5th of February and we're still seeing thousands of challenge files created and left scattered around. Both in /var/www/vhosts/<vhost>/httpdocs/.well-known/acme-challenge/ as well as /var/www/vhosts/default/htdocs/.well-known/acme-challenge/

    How should we configure Plesk to have this issue fixed once and for all? Reading https://docs.plesk.com/en-US/obsidian/administrator-guide/78586/ I haven't found anything that tackles the removal of old files. Are we supposed to set up our own cron jobs for this?

     

    0
    Comment actions Permalink
  • Avatar
    Websavers Inc (Edited )

    I can confirm burnlyvic's results. We manually cleared all of our servers when I posted my last comment in July 2019. Since then there are now thousands more of these files.

    Ekaterina Babenko I think you guys need to re-open whatever internal bug tracker you have on this as the issue is definitely not fixed.

    On the servers I've checked, we were previously running the latest Onyx release and then as of a few days ago updated to the very latest Obsidian release and the issue continues.

    There are multiple files dated today and a total number of files of around 50k on just one of our servers.

    Given how long this has gone on for, I think you also need to make it part of your fix to clean up all existing files.

    0
    Comment actions Permalink
  • Avatar
    Ivan Postnikov

    Hello Websavers Inc

    The support request was created to check this issue, please, expect an update.

    0
    Comment actions Permalink
  • Avatar
    Marcel Aulenbacher

    Hi!

    I have challenge files in the vhost directories from may 29 2020 which have not been cleaned. Please provide a solution yet integrity checks on installed software fails because of unexpected files in root directories. Thank you very much. 

    0
    Comment actions Permalink
  • Avatar
    Anton Maslov

    Hi Marcel,


    Please check solution under "Workaround" section it has commands on how to clean up and remove challenge files.

    0
    Comment actions Permalink

Please sign in to leave a comment.

Have more questions? Submit a request

Related articles