Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-38350

PHP Warning when purging all caches: race condition?

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.5, 2.6.3, 2.7
    • Fix Version/s: STABLE backlog
    • Component/s: Caching
    • Labels:
    • Affected Branches:
      MOODLE_25_STABLE, MOODLE_26_STABLE, MOODLE_27_STABLE

      Description

      Never hit before now, I was using 3 tabs each with a Moodle page using them quite at the same time and one of them used to trigger the Purge all caches (http://localhost/moodle-master/admin/purgecaches.php) action.

      PHP Warning:  rmdir(/path/to/moodle-master-data/cache/cachestore_file/default_application) [<a href='function.rmdir'>function.rmdir</a>]: Directory not empty in /path/to/moodle-master/lib/moodlelib.php on line 10750
      

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            matteo Matteo Scaramuccia added a comment -

            Now that we have MUC what about reducing the scope of the remove_dir call in purge_all_caches?

            diff --git a/lib/moodlelib.php b/lib/moodlelib.php
            index 5e89439..8e0a07b 100644
            --- a/lib/moodlelib.php
            +++ b/lib/moodlelib.php
            @@ -1557,7 +1557,7 @@ function purge_all_caches() {
                 cache_helper::purge_all();
             
                 // purge all other caches: rss, simplepie, etc.
            -    remove_dir($CFG->cachedir.'', true);
            +    remove_dir($CFG->cachedir.'');
             
                 // make sure cache dir is writable, throws exception if not
                 make_cache_directory('');

            Show
            matteo Matteo Scaramuccia added a comment - Now that we have MUC what about reducing the scope of the remove_dir call in purge_all_caches ? diff --git a/lib/moodlelib.php b/lib/moodlelib.php index 5e89439..8e0a07b 100644 --- a/lib/moodlelib.php +++ b/lib/moodlelib.php @@ -1557,7 +1557,7 @@ function purge_all_caches() { cache_helper::purge_all();   // purge all other caches: rss, simplepie, etc. - remove_dir($CFG->cachedir.'', true); + remove_dir($CFG->cachedir.'');   // make sure cache dir is writable, throws exception if not make_cache_directory('');
            Hide
            samhemelryk Sam Hemelryk added a comment -

            Hi Matteo,

            Thanks for creating this report.

            I've had a quick try to reproduce the issue without luck. Not to much of a surprise as its definitely sounds like a race condition and they can be hard to reproduce.
            You're right the now that we have MUC we should probably look at how we are cleaning the cache direction.
            I've been looking at something similiar within the unit tests as they do the same kind of aggressive cleaning of the moodledata dir including the cache dir.
            I'll keep this in mind and expand my research of what can be done to the whole Moodle scope.

            Many thanks
            Sam

            Show
            samhemelryk Sam Hemelryk added a comment - Hi Matteo, Thanks for creating this report. I've had a quick try to reproduce the issue without luck. Not to much of a surprise as its definitely sounds like a race condition and they can be hard to reproduce. You're right the now that we have MUC we should probably look at how we are cleaning the cache direction. I've been looking at something similiar within the unit tests as they do the same kind of aggressive cleaning of the moodledata dir including the cache dir. I'll keep this in mind and expand my research of what can be done to the whole Moodle scope. Many thanks Sam
            Hide
            salvetore Michael de Raadt added a comment -

            A similar error was reported in MDL-38273 and I experienced the errors while testing MDL-38387.

            Show
            salvetore Michael de Raadt added a comment - A similar error was reported in MDL-38273 and I experienced the errors while testing MDL-38387 .
            Hide
            leblangi Gilles-Philippe Leblanc added a comment - - edited

            We just experiment the same behaviour when purging all the cache:

            Warning: rmdir(/path/to/moodle-master-data/cache/cachestore_file/default_application/core_string): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 Warning: rmdir(/path/to/moodle-master-data/cachestore_file/default_application/core_databasemeta): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 Warning: rmdir(/path/to/moodle-master-data/cache/cachestore_file/default_application): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 Warning: rmdir(/path/to/moodle-master-data/cachestore_file): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 
            

            Show
            leblangi Gilles-Philippe Leblanc added a comment - - edited We just experiment the same behaviour when purging all the cache: Warning: rmdir(/path/to/moodle-master-data/cache/cachestore_file/default_application/core_string): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 Warning: rmdir(/path/to/moodle-master-data/cachestore_file/default_application/core_databasemeta): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 Warning: rmdir(/path/to/moodle-master-data/cache/cachestore_file/default_application): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728 Warning: rmdir(/path/to/moodle-master-data/cachestore_file): Directory not empty in /path/to/moodle/lib/moodlelib.php on line 10728
            Hide
            marina Marina Glancy added a comment -

            hopefully patch will resolve MDL-41414 as well

            Show
            marina Marina Glancy added a comment - hopefully patch will resolve MDL-41414 as well
            Hide
            marina Marina Glancy added a comment -

            I just got it again. I changed the cache definition (from session to application), bumped the version, run upgrade process and here it is.

             
            ( ! ) Warning: rmdir(/home/marina/repositories/master/moodledata/cache/cachestore_file/default_application): Directory not empty in /home/marina/repositories/master/moodle/lib/moodlelib.php on line 8953
            Call Stack
            #	Time	Memory	Function	Location
            1	0.0019	369696	{main}( )	../index.php:0
            2	0.9706	25831680	upgrade_noncore( )	../index.php:422
            3	3.0105	34044544	purge_all_caches( )	../upgradelib.php:1589
            4	3.0741	34054144	remove_dir( )	../moodlelib.php:1613
            5	3.0742	34054824	remove_dir( )	../moodlelib.php:8942
            6	3.0742	34055584	remove_dir( )	../moodlelib.php:8942
            7	3.0830	34055720	rmdir ( )	../moodlelib.php:8953
             
            ( ! ) Warning: rmdir(/home/marina/repositories/master/moodledata/cache/cachestore_file): Directory not empty in /home/marina/repositories/master/moodle/lib/moodlelib.php on line 8953
            Call Stack
            #	Time	Memory	Function	Location
            1	0.0019	369696	{main}( )	../index.php:0
            2	0.9706	25831680	upgrade_noncore( )	../index.php:422
            3	3.0105	34044544	purge_all_caches( )	../upgradelib.php:1589
            4	3.0741	34054144	remove_dir( )	../moodlelib.php:1613
            5	3.0742	34054824	remove_dir( )	../moodlelib.php:8942
            6	3.0857	34054976	rmdir ( )	../moodlelib.php:8953

            Show
            marina Marina Glancy added a comment - I just got it again. I changed the cache definition (from session to application), bumped the version, run upgrade process and here it is.   ( ! ) Warning: rmdir(/home/marina/repositories/master/moodledata/cache/cachestore_file/default_application): Directory not empty in /home/marina/repositories/master/moodle/lib/moodlelib.php on line 8953 Call Stack # Time Memory Function Location 1 0.0019 369696 {main}( ) ../index.php:0 2 0.9706 25831680 upgrade_noncore( ) ../index.php:422 3 3.0105 34044544 purge_all_caches( ) ../upgradelib.php:1589 4 3.0741 34054144 remove_dir( ) ../moodlelib.php:1613 5 3.0742 34054824 remove_dir( ) ../moodlelib.php:8942 6 3.0742 34055584 remove_dir( ) ../moodlelib.php:8942 7 3.0830 34055720 rmdir ( ) ../moodlelib.php:8953   ( ! ) Warning: rmdir(/home/marina/repositories/master/moodledata/cache/cachestore_file): Directory not empty in /home/marina/repositories/master/moodle/lib/moodlelib.php on line 8953 Call Stack # Time Memory Function Location 1 0.0019 369696 {main}( ) ../index.php:0 2 0.9706 25831680 upgrade_noncore( ) ../index.php:422 3 3.0105 34044544 purge_all_caches( ) ../upgradelib.php:1589 4 3.0741 34054144 remove_dir( ) ../moodlelib.php:1613 5 3.0742 34054824 remove_dir( ) ../moodlelib.php:8942 6 3.0857 34054976 rmdir ( ) ../moodlelib.php:8953
            Hide
            kimberw Kimber Warden added a comment -

            I get the same error when installing any new module or block in a fresh local installation of 2.7.

            Show
            kimberw Kimber Warden added a comment - I get the same error when installing any new module or block in a fresh local installation of 2.7.
            Hide
            poltawski Dan Poltawski added a comment -

            I've seen this many times too - but have never eliminating my 'development environment' from the cause

            Show
            poltawski Dan Poltawski added a comment - I've seen this many times too - but have never eliminating my 'development environment' from the cause
            Hide
            mudrd8mz David Mudrak added a comment -

            I just hit this on very fresh 2.7 installation just after installing a new plugin.

            Show
            mudrd8mz David Mudrak added a comment - I just hit this on very fresh 2.7 installation just after installing a new plugin.
            Hide
            mr-russ Russell Smith added a comment -

            Is looking at a a strategy of renaming the cache directory first a possible way to reduce the race condition. If you move the directory, all existing requests will still finish reading and/or writing to the renames space. And all new requests will attempt to recreate the cache folder and write files to the new location.

            This ensure that's other threads will not try and write new cache files to the version of the cache we are trying to remove. I'm not convinced that rename is an atomic operation on all filesystems but it has to reduce the window from seconds to microseconds.

            However the reports of this happening during an upgrade appears strange. That's a single threaded process that should always finish clearing the cache before it move on. The only thought is that there is a file it can't remove and it just continues to attempt to remove the folder. If that is true, renaming the folder should remove that issue as well.

            Maybe the folder couple be moved into the trash bin and the trash code could clean it up. But that might be taking it too far.

            Show
            mr-russ Russell Smith added a comment - Is looking at a a strategy of renaming the cache directory first a possible way to reduce the race condition. If you move the directory, all existing requests will still finish reading and/or writing to the renames space. And all new requests will attempt to recreate the cache folder and write files to the new location. This ensure that's other threads will not try and write new cache files to the version of the cache we are trying to remove. I'm not convinced that rename is an atomic operation on all filesystems but it has to reduce the window from seconds to microseconds. However the reports of this happening during an upgrade appears strange. That's a single threaded process that should always finish clearing the cache before it move on. The only thought is that there is a file it can't remove and it just continues to attempt to remove the folder. If that is true, renaming the folder should remove that issue as well. Maybe the folder couple be moved into the trash bin and the trash code could clean it up. But that might be taking it too far.
            Hide
            mudrd8mz David Mudrak added a comment -

            Continuously experiencing this - just hit it on 2.8 site when installing a plugin.

            Show
            mudrd8mz David Mudrak added a comment - Continuously experiencing this - just hit it on 2.8 site when installing a plugin.

              People

              • Votes:
                5 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated: