Moodle
  1. Moodle
  2. MDL-28505

Asynchronous course backup / restore

    Details

    • Affected Branches:
      MOODLE_20_STABLE, MOODLE_23_STABLE
    • Rank:
      18163

      Description

      It would be great if user-performed course backups could be asynchronous, as in, the request returns instantly with a message that the backup is "queued". The user can then return to the backups page later to check if the backup is complete. There could be an AJAX interface to prevent them from needing to hit refresh.

      The back-end would be implemented in a process-queue. The advantage being:

      • Long-running backup processes do not hold up apache threads and use excessive memory/cpu on page-serving nodes which can effect other page requests.
      • The user's session is not locked during the time that the backup is running
      • It prevents all issues relating to max execution time, load balancer idle timeouts, partially completed backups, that can make course backups/restores fail.
      • It would allow the process queue to be scaled by adding more servers that pull from the queue.
      • Most importantly, it prevents multiple concurrent backups from slowing down the system. We can limit and control the concurrency and scale the worker pool as needed.

      This is obviously not a trivial thing to implement, and for low-end users who download a WAMP package it needs to be just as easy to get running. However, for enterprise deployments this would be an excellent architectural change.

      Note that it is pretty common amongst large-scale web applications to not 'block' on long-running operations - but rather to return instantly and send a job to a back-end process queue.

        Activity

        Hide
        Michael Blake added a comment -

        This feature request has been made by a MP. Please give it priority.

        Show
        Michael Blake added a comment - This feature request has been made by a MP. Please give it priority.
        Hide
        Michael de Raadt added a comment -

        Thanks for suggesting this, Ash.

        Feel free to help us work on this issue.

        Show
        Michael de Raadt added a comment - Thanks for suggesting this, Ash. Feel free to help us work on this issue.
        Hide
        Michael de Raadt added a comment -

        Eloy: This seems like a useful change, but a significant piece of work. Can you please add a comment?

        Show
        Michael de Raadt added a comment - Eloy: This seems like a useful change, but a significant piece of work. Can you please add a comment?
        Hide
        Eloy Lafuente (stronk7) added a comment -

        Ho,

        first of all, let me state that, due to the nature of PHP, running within the webserver process, there is no effective way to detach/fork any job (the backup) from the webserver itself (surely there are some extension providing that, but surely it is not cross-OS nor part of PHP "core", so let's assume it's impossible).

        That implies that, whatever the solution we use for "asynchronous" it won't be really asynchronous but executed in a different way, out from the webserver completely.

        And afaik, there are two ways to achieve it:

        1) The simplest: Create one CLI script for executing backups is "trivial". I've attached above the very basic/hacky one I use here for my repeated tests (instead of doing them using the UI). Of course, a proper CLI will accept params, defaults perform validations and so on. The attachment just shows how to interact with the backup API. Of course with this solution everything happens in the CLI, aka, no UI in Moodle at all.

        2) The complex: since day 0 backup was planned to support immediate/delayed executions (backup::EXECUTION_DELAYED) but it has not been implemented yet. The idea beyond this mode is that the user performs the request of the course backup and decides not to execute it immediately (backup::EXECUTION_INMEDIATE). So the configured backup is stored in DB (backup_controllers table). And later some script goes over all delayed backups (could be Moodle's cron but I really see it being done by another script, very much like the 1) above.

        So perhaps, the best way to achieve this is to do it in a progressive way, by implementing a complete 1) first, able to, also execute backups from the backup_controller table and later, implement 2) by allowing the UI to end if the backup is delayed (we'll need cap checks here for sure) and making a new cron-able script to fetch them and call 1)

        Hope this helps, ciao

        Show
        Eloy Lafuente (stronk7) added a comment - Ho, first of all, let me state that, due to the nature of PHP, running within the webserver process, there is no effective way to detach/fork any job (the backup) from the webserver itself (surely there are some extension providing that, but surely it is not cross-OS nor part of PHP "core", so let's assume it's impossible). That implies that, whatever the solution we use for "asynchronous" it won't be really asynchronous but executed in a different way, out from the webserver completely. And afaik, there are two ways to achieve it: 1) The simplest: Create one CLI script for executing backups is "trivial". I've attached above the very basic/hacky one I use here for my repeated tests (instead of doing them using the UI). Of course, a proper CLI will accept params, defaults perform validations and so on. The attachment just shows how to interact with the backup API. Of course with this solution everything happens in the CLI, aka, no UI in Moodle at all. 2) The complex: since day 0 backup was planned to support immediate/delayed executions (backup::EXECUTION_DELAYED) but it has not been implemented yet. The idea beyond this mode is that the user performs the request of the course backup and decides not to execute it immediately (backup::EXECUTION_INMEDIATE). So the configured backup is stored in DB (backup_controllers table). And later some script goes over all delayed backups (could be Moodle's cron but I really see it being done by another script, very much like the 1) above. So perhaps, the best way to achieve this is to do it in a progressive way, by implementing a complete 1) first, able to, also execute backups from the backup_controller table and later, implement 2) by allowing the UI to end if the backup is delayed (we'll need cap checks here for sure) and making a new cron-able script to fetch them and call 1) Hope this helps, ciao
        Hide
        Tony Levi added a comment -

        Hi Eloy,

        I think #2 performed by a CLI/daemon script like #1 is exactly what we had in mind, and 'asyncronous' just means away from the httpd process as you noted.

        It seems clear to me the moodle cron isn't the best or only place to do this work; it is already quite busy enough and we can divide work up better with an option to do this work seperately. There is also a significant delay to doing this in the cron; ideally we can have a backup/restore 'daemon' that will sleep most of the time and check for new work every few seconds, run on as many processors as the user likes etc.

        I guess a big chunk (maybe even the 'larger' portion) is in integrating this into the UI and security model.

        Show
        Tony Levi added a comment - Hi Eloy, I think #2 performed by a CLI/daemon script like #1 is exactly what we had in mind, and 'asyncronous' just means away from the httpd process as you noted. It seems clear to me the moodle cron isn't the best or only place to do this work; it is already quite busy enough and we can divide work up better with an option to do this work seperately. There is also a significant delay to doing this in the cron; ideally we can have a backup/restore 'daemon' that will sleep most of the time and check for new work every few seconds, run on as many processors as the user likes etc. I guess a big chunk (maybe even the 'larger' portion) is in integrating this into the UI and security model.
        Hide
        Ashley Holman added a comment -

        Thanks for looking into this Eloy. The main purpose of this feature request is to prevent the problems that occur due to the "blocking" approach. For example, a user may start creating a backup of a 4GB course, and then after 2 minutes of waiting they think that there is a problem, so they hit back and then retry the backup again. The first request will either stop and leave a partially complete backup temp dir, or will continue running in the background (usually what we see happen). When there are multiple backup/restore executions running at once, it can effect the performance of the whole server. We can't queue user-driven backups so several users could all execute them at the same time and hold up apache threads & place too much load on the system. Also it locks the user's session so if they try to load up more tabs, those requests will also block and hence clog up more webserver connections.

        So the point of this is to keep the web interface fast and lean, and non-blocking (aka asynchronous). Option #1 (CLI script) comes in handy for system administrators, and we have a script like that which we use already, but that does not address the problem with the web interface. Option #2 sounds like it would address the problem, and it's good that delayed execution has already been planned. I wouldn't want to do any forking of the PHP process as this still puts load on the webserver and doesn't limit concurrency. By having a proper queue & consumer process, we can limit the number of concurrent backup executions, and also put it on its own server to isolate it from effecting user requests.

        Show
        Ashley Holman added a comment - Thanks for looking into this Eloy. The main purpose of this feature request is to prevent the problems that occur due to the "blocking" approach. For example, a user may start creating a backup of a 4GB course, and then after 2 minutes of waiting they think that there is a problem, so they hit back and then retry the backup again. The first request will either stop and leave a partially complete backup temp dir, or will continue running in the background (usually what we see happen). When there are multiple backup/restore executions running at once, it can effect the performance of the whole server. We can't queue user-driven backups so several users could all execute them at the same time and hold up apache threads & place too much load on the system. Also it locks the user's session so if they try to load up more tabs, those requests will also block and hence clog up more webserver connections. So the point of this is to keep the web interface fast and lean, and non-blocking (aka asynchronous). Option #1 (CLI script) comes in handy for system administrators, and we have a script like that which we use already, but that does not address the problem with the web interface. Option #2 sounds like it would address the problem, and it's good that delayed execution has already been planned. I wouldn't want to do any forking of the PHP process as this still puts load on the webserver and doesn't limit concurrency. By having a proper queue & consumer process, we can limit the number of concurrent backup executions, and also put it on its own server to isolate it from effecting user requests.
        Hide
        Tai Farmer added a comment -

        When Moodle is placed behind a load balancer, long requests will eventually time out. Also, with Amazon's ELB, there is no setting to change this timeout value as there are with other load balancers, so, the solutions are very limited. And, the course restore process does not continue in the background, it appears, when it does timeout.

        The CLI script is already available for creating course backups. However, a similar script does not yet exist for course restores. Any idea if a course restore CLI script is currently being worked on?

        Show
        Tai Farmer added a comment - When Moodle is placed behind a load balancer, long requests will eventually time out. Also, with Amazon's ELB, there is no setting to change this timeout value as there are with other load balancers, so, the solutions are very limited. And, the course restore process does not continue in the background, it appears, when it does timeout. The CLI script is already available for creating course backups. However, a similar script does not yet exist for course restores. Any idea if a course restore CLI script is currently being worked on?
        Hide
        Damyon Wiese added a comment -

        This was discussed at the hackfest and we have started a specification.

        http://docs.moodle.org/dev/ASynchronous_Backup_and_Restore_Specification

        Show
        Damyon Wiese added a comment - This was discussed at the hackfest and we have started a specification. http://docs.moodle.org/dev/ASynchronous_Backup_and_Restore_Specification
        Hide
        Eloy Lafuente (stronk7) added a comment -

        This issue was assigned to me automatically, however I will not be able to work on this issue in the immediate future. In order to create a truer sense of the state of this issue and to allow other developers to have chance to become involved, I am removing myself as the assignee of this issue.

        For more information, see http://docs.moodle.org/dev/Changes_to_issue_assignment

        Show
        Eloy Lafuente (stronk7) added a comment - This issue was assigned to me automatically, however I will not be able to work on this issue in the immediate future. In order to create a truer sense of the state of this issue and to allow other developers to have chance to become involved, I am removing myself as the assignee of this issue. For more information, see http://docs.moodle.org/dev/Changes_to_issue_assignment

          People

          • Votes:
            12 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated: