Moodle
  1. Moodle
  2. MDL-29262

Moodle 2 backup_controllers table is needlessly massive

    Details

    • Database:
      MySQL
    • Testing Instructions:
      Hide

      NOTE: Testing this requires DB access.
      NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master

      1) With the patch applied, go to admin -> courses -> backup -> general

      2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days.

      3) Change it to 7 days and save changes.

      4) TEST: Verify the setting has been saved ok (7 days).

      5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case).

      6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok.

      7) TEST: The backup ends ok.

      8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string).

      9) Perform one import operation (from one course to another, pick some activity).

      10) TEST: The process ends ok.

      11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

      12) Perform one duplicate activity operation (x2 icon under edit mode).

      13) TEST: The process ends ok.

      14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

      15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual".

      16) CLI execute: admin/cli/automated_backups.php

      17) TEST: The process ends ok.

      18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string).

      19) Count the number of records in the backup_controllers table.

      20) Update 2 of those records to have timecreated = 1. Annotate their ids.

      21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time).

      22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly.

      23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table.

      24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated.

      25) TEST: Execute controller unit tests:

      • For 21 and 22 (simpletest), in admin -> development -> unit tests, run tests in "backup/controller". Results: 6 passed, 0 exceptions and 0 fails (and no PHP Notice/Warning/Error/Strict problem in output/logs).
      • For master (phpunit), install and configure phpunit and run: "phpunit backup/controller/tests/controller_test.php". Results: OK (1 test, 6 assertions) (and no PHP Notice/Warning/Error/Strict problem in output/logs).

      26) Repeat 1-25 for all fixed branches

      Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.

      Show
      NOTE: Testing this requires DB access. NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master 1) With the patch applied, go to admin -> courses -> backup -> general 2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days. 3) Change it to 7 days and save changes. 4) TEST: Verify the setting has been saved ok (7 days). 5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case). 6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok. 7) TEST: The backup ends ok. 8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string). 9) Perform one import operation (from one course to another, pick some activity). 10) TEST: The process ends ok. 11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string). 12) Perform one duplicate activity operation (x2 icon under edit mode). 13) TEST: The process ends ok. 14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string). 15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual". 16) CLI execute: admin/cli/automated_backups.php 17) TEST: The process ends ok. 18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string). 19) Count the number of records in the backup_controllers table. 20) Update 2 of those records to have timecreated = 1. Annotate their ids. 21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time). 22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly. 23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table. 24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated. 25) TEST: Execute controller unit tests: For 21 and 22 (simpletest), in admin -> development -> unit tests, run tests in "backup/controller". Results: 6 passed, 0 exceptions and 0 fails (and no PHP Notice/Warning/Error/Strict problem in output/logs). For master (phpunit), install and configure phpunit and run: "phpunit backup/controller/tests/controller_test.php". Results: OK (1 test, 6 assertions) (and no PHP Notice/Warning/Error/Strict problem in output/logs). 26) Repeat 1-25 for all fixed branches Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.
    • Workaround:
      Hide

      Having searched the forums and tracker it seems I can probably truncate this table (when a backup isn't running) and cause no problems, but it's likely that this table will grow to excessive size again.

      Show
      Having searched the forums and tracker it seems I can probably truncate this table (when a backup isn't running) and cause no problems, but it's likely that this table will grow to excessive size again.
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_22_STABLE
    • Fixed Branches:
      MOODLE_21_STABLE, MOODLE_22_STABLE
    • Pull from Repository:
    • Pull Master Branch:
    • Rank:
      18805

      Description

      Most relevant forum post: http://moodle.org/mod/forum/discuss.php?d=172859
      Most likely related tracker entry: http://tracker.moodle.org/browse/MDL-29138

      Having done some final checks of our new Moodle 2 install prior to teaching commencing in a few days' time, I noticed that the database was 3.4Gb in size. This was a surprise as our previous Moodle 1.9 install had three times as many courses and well over 20,000 users and had been in place for over 5 years, and it's database ended up at 1.1Gb.

      Closer inspection revealed that the backup_controllers table was 3.0Gb in size and that the actual database size was more like 400Mb.

        Issue Links

          Activity

          Paul Vaughan created issue -
          Hide
          Emma Richardson added a comment - - edited

          Also affects Moodle 2.0.3. Table seems to grow about 5Mb a day after each run of automated backups.

          Show
          Emma Richardson added a comment - - edited Also affects Moodle 2.0.3. Table seems to grow about 5Mb a day after each run of automated backups.
          Hide
          Michael de Raadt added a comment -

          Thanks for reporting this.

          If you find out anything more, please add information here.

          Show
          Michael de Raadt added a comment - Thanks for reporting this. If you find out anything more, please add information here.
          Michael de Raadt made changes -
          Field Original Value New Value
          Fix Version/s STABLE backlog [ 10463 ]
          Labels triaged
          Michael de Raadt made changes -
          Link This issue will help resolve MDL-29138 [ MDL-29138 ]
          Hide
          Paul Vaughan added a comment - - edited

          I have just duplicated the backup_controllers table (just in case) and truncated the original. Obviously this won't save space but I'll see what happens after the next set of backups, which I have moved forward to 9pm (BST) this evening.

          I'm not sure what other info or details I can provide, but if you suggest them, I will provide them.

          Closer inspection shows that that the base64-encoded serialised array held in backup_controllers.controller is the main culprit, easily reaching 1Mb of text in places. I'm not sure if that's relevant or not though. Can supply one if requested (maybe privately, not sure if there's sensitive data in there or not).

          Edited to add: the worst offender is 3.1Mb of data in the controllers field, and we have lots over 2Mb too.

          Show
          Paul Vaughan added a comment - - edited I have just duplicated the backup_controllers table (just in case) and truncated the original. Obviously this won't save space but I'll see what happens after the next set of backups, which I have moved forward to 9pm (BST) this evening. I'm not sure what other info or details I can provide, but if you suggest them, I will provide them. Closer inspection shows that that the base64-encoded serialised array held in backup_controllers.controller is the main culprit, easily reaching 1Mb of text in places. I'm not sure if that's relevant or not though. Can supply one if requested (maybe privately, not sure if there's sensitive data in there or not). Edited to add: the worst offender is 3.1Mb of data in the controllers field, and we have lots over 2Mb too.
          Hide
          Paul Vaughan added a comment -

          I can't actually run the backups at the moment due to MDL-28180. As soon as I have either succeeded or given up on trying to fix the data, I'll let you know.

          Show
          Paul Vaughan added a comment - I can't actually run the backups at the moment due to MDL-28180 . As soon as I have either succeeded or given up on trying to fix the data, I'll let you know.
          Hide
          Paul Vaughan added a comment - - edited

          Since truncating the backup_controllers table last week, we now have 271 entries in it, mostly with the operation set to 'backup', and the table itself has already reached 90Mb. In one row, the data in the controller column was just 169Kb, much lower then the previously noted 2-3Mb, but I am still having trouble base64-decoding and unserialising it, just to see what it is.

          What I can see is a lot of "RECURSION" (without the quotes, but wish asterisks) in the resultant unserialised array. I'm not sure if this is a fault with how I'm working with it, or if this is what the data contains, but it seems a bit odd.

          Show
          Paul Vaughan added a comment - - edited Since truncating the backup_controllers table last week, we now have 271 entries in it, mostly with the operation set to 'backup', and the table itself has already reached 90Mb. In one row, the data in the controller column was just 169Kb, much lower then the previously noted 2-3Mb, but I am still having trouble base64-decoding and unserialising it, just to see what it is. What I can see is a lot of " RECURSION " (without the quotes, but wish asterisks) in the resultant unserialised array. I'm not sure if this is a fault with how I'm working with it, or if this is what the data contains, but it seems a bit odd.
          Hide
          Paul Vaughan added a comment -

          Our backups are now working again. The backup_controllers table now has 662 entries in it and is now 224, making up around one third of our whole database's storage requirement.

          I don't think I can add any more detail to this report unless there is something specific you want me to locate and report on.

          Show
          Paul Vaughan added a comment - Our backups are now working again. The backup_controllers table now has 662 entries in it and is now 224, making up around one third of our whole database's storage requirement. I don't think I can add any more detail to this report unless there is something specific you want me to locate and report on.
          Hide
          Mark Ward added a comment -

          +1ed, this is seems to be very odd behaviour. Truncating the table seems to have no adverse effects.

          Show
          Mark Ward added a comment - +1ed, this is seems to be very odd behaviour. Truncating the table seems to have no adverse effects.
          Hide
          Paul Vaughan added a comment -

          Today's stats: 1,241 rows, 420Mb in size which is roughly half of the Moodle database.

          Show
          Paul Vaughan added a comment - Today's stats: 1,241 rows, 420Mb in size which is roughly half of the Moodle database.
          Hide
          Robert Northcutt added a comment -

          Running Moodle 2.1.1+ (Build: 20110916). backup_controller table is 273 MB out of 350 MB total (78%). Run backups x3 per week and only retain most recent version.

          This needs to be resolved (more than a minor issue).

          Show
          Robert Northcutt added a comment - Running Moodle 2.1.1+ (Build: 20110916). backup_controller table is 273 MB out of 350 MB total (78%). Run backups x3 per week and only retain most recent version. This needs to be resolved (more than a minor issue).
          Hide
          Robert Northcutt added a comment -

          Also, could someone please confirm that truncating the BACKUP_CONTROLLER table will not cause any adverse effects. I see that a couple of people have done this but would like someone to officially confirm that this workaround is OK before I do it.

          Show
          Robert Northcutt added a comment - Also, could someone please confirm that truncating the BACKUP_CONTROLLER table will not cause any adverse effects. I see that a couple of people have done this but would like someone to officially confirm that this workaround is OK before I do it.
          Hide
          Paul Vaughan added a comment - - edited

          Hi Robert.

          I have truncated the backup_controllers table once, and backups have continued after without any apparent problems. If you are concerned that this might cause problems somehow or you just want to play it safe, use a program like PHPMyAdmin to copy the table (to, for example, backup_controllers_backup) and then truncate the original. This way you can restore it pretty quick if you get problems (I haven't).

          I hope this is official enough.

          Edited to add: todays stats are 2,260 rows, 800Mb, or around two thirds of the whole database.

          Show
          Paul Vaughan added a comment - - edited Hi Robert. I have truncated the backup_controllers table once, and backups have continued after without any apparent problems. If you are concerned that this might cause problems somehow or you just want to play it safe, use a program like PHPMyAdmin to copy the table (to, for example, backup_controllers_backup) and then truncate the original. This way you can restore it pretty quick if you get problems (I haven't). I hope this is official enough. Edited to add: todays stats are 2,260 rows, 800Mb, or around two thirds of the whole database.
          Hide
          Robert Northcutt added a comment -

          Performed truncate of backup_controller table. All appears to be working with daily backups after two days...so I would say that workaround is successful...but wish it was not necessary.

          Show
          Robert Northcutt added a comment - Performed truncate of backup_controller table. All appears to be working with daily backups after two days...so I would say that workaround is successful...but wish it was not necessary.
          Hide
          Paul Vaughan added a comment -

          Our backup_controllers table just topped 1Gb so have done yet another truncate. If removing the data from this table is causing no problems, why are we keeping it in the first place? Can't the automatic course backups perform a truncate on the successful completion of the backups?

          Show
          Paul Vaughan added a comment - Our backup_controllers table just topped 1Gb so have done yet another truncate. If removing the data from this table is causing no problems, why are we keeping it in the first place? Can't the automatic course backups perform a truncate on the successful completion of the backups?
          Hide
          Claus A. Us. added a comment -

          Hi,

          Since my backup_controller table also uses half of the space used for the whole db, I had a look at the table data. As far as I could see there are some interesting points:
          Most of the entities are automated course backups with the status 700 (STATUS_AWAITING), execution time 0 (immediately), and the purpose 50 (MODE_AUTOMATED). If one orders the table entries by itemid and id, you will notice due to creation time and modification time of entities with the same itemid (which is the course id), that every week two entries are generated with almost one day difference between the first and the second entry.
          As the backups could be found in the backup folder, I assume that the controllers are saved in the DB and won't be deleted after they are finished. Compare backup_cron_helper.class.php: function launch_automated_backup. The status is set to STATUS_AWAITING which results in saving the controller to DB (/backup/controller/backup_controller.class.php: function set_status). To my understanding, the problem is that a controller is saved when setting the status to ‘awaiting’, but it won’t be deleted when changing the status to finished.

          Hopes that helps while solving this issue.

          PS: deleting entries that are older than a few days can’t have any negative effect on the platform.

          Show
          Claus A. Us. added a comment - Hi, Since my backup_controller table also uses half of the space used for the whole db, I had a look at the table data. As far as I could see there are some interesting points: Most of the entities are automated course backups with the status 700 (STATUS_AWAITING), execution time 0 (immediately), and the purpose 50 (MODE_AUTOMATED). If one orders the table entries by itemid and id, you will notice due to creation time and modification time of entities with the same itemid (which is the course id), that every week two entries are generated with almost one day difference between the first and the second entry. As the backups could be found in the backup folder, I assume that the controllers are saved in the DB and won't be deleted after they are finished. Compare backup_cron_helper.class.php: function launch_automated_backup. The status is set to STATUS_AWAITING which results in saving the controller to DB (/backup/controller/backup_controller.class.php: function set_status). To my understanding, the problem is that a controller is saved when setting the status to ‘awaiting’, but it won’t be deleted when changing the status to finished. Hopes that helps while solving this issue. PS: deleting entries that are older than a few days can’t have any negative effect on the platform.
          Ray Lawrence made changes -
          Labels triaged partner triaged
          Hide
          Ray Lawrence added a comment -

          +1 for a fix on this.

          Show
          Ray Lawrence added a comment - +1 for a fix on this.
          Hide
          Bart Kamphuis added a comment -

          This is rediculous !!
          This should be considered a MAJOR issue - on our systems this is causing havoc.
          Please fix this asap.

          Show
          Bart Kamphuis added a comment - This is rediculous !! This should be considered a MAJOR issue - on our systems this is causing havoc. Please fix this asap.
          Paul Vaughan made changes -
          Affects Version/s 2.1.2 [ 10851 ]
          Hide
          Paul Vaughan added a comment -

          Hi Bart. As the OP I can change the initial report (for example, to major instead of minor) however I'd like to know what havoc this is causing on your Moodle installation. I know it's a problem, but I've not considered it major (just annoying) as we have a large capacity disk on our database and would take quite some time to fill.

          If you can add detail to this report, showing in what ways this error is inconveniencing you, it is more likely to receive attention.

          Paul.

          Show
          Paul Vaughan added a comment - Hi Bart. As the OP I can change the initial report (for example, to major instead of minor) however I'd like to know what havoc this is causing on your Moodle installation. I know it's a problem, but I've not considered it major (just annoying) as we have a large capacity disk on our database and would take quite some time to fill. If you can add detail to this report, showing in what ways this error is inconveniencing you, it is more likely to receive attention. Paul.
          Hide
          Ray Lawrence added a comment -

          It's an issue when migrating databases.

          Show
          Ray Lawrence added a comment - It's an issue when migrating databases.
          Hide
          Paul Vaughan added a comment -

          Hi Ray.

          Is the issue only one of size? If so, it can easily be remedied by truncating the table immediately before migration. If there's something else, please detail here, where someone may be able to assist, and the devs will become aware of it.

          Paul.

          Show
          Paul Vaughan added a comment - Hi Ray. Is the issue only one of size? If so, it can easily be remedied by truncating the table immediately before migration. If there's something else, please detail here, where someone may be able to assist, and the devs will become aware of it. Paul.
          Paul Vaughan made changes -
          Component/s Backup [ 10068 ]
          Hide
          Ray Lawrence added a comment -

          Yes.

          We can empty the table beforehand but that shouldn't really be necessary.

          Show
          Ray Lawrence added a comment - Yes. We can empty the table beforehand but that shouldn't really be necessary.
          Hide
          Brent Lee added a comment -

          I have run into this again with an update. Can you increase the priority?

          Show
          Brent Lee added a comment - I have run into this again with an update. Can you increase the priority?
          Paul Vaughan made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Hide
          Paul Vaughan added a comment -

          As per requests, priority has been increased to major.

          We're not running 2.2 in production yet so I cannot tell if this is still an issue. If anyone reading this is using 2.2 (or any version of 2.1 not listed above) please note it in the comments: the more people have the issue, the more traction this issue will gain and the quicker a solution will happen.

          Show
          Paul Vaughan added a comment - As per requests, priority has been increased to major. We're not running 2.2 in production yet so I cannot tell if this is still an issue. If anyone reading this is using 2.2 (or any version of 2.1 not listed above) please note it in the comments: the more people have the issue, the more traction this issue will gain and the quicker a solution will happen.
          Hide
          Wiktor Wandachowicz added a comment -

          Recently we've upgraded our Moodle installations to "Moodle 2.2.1+ (Build: 20120119)" and we see the issue. On our university we have several Moodle installations, one per department. Some of them are actively used, some are in the works.

          On one of the servers there are over 1500 empty courses (reflecting organizational structure), ready to be filled with content. And "backup_controllers" table already takes 13GB of hard drive space. On another server there are 253 courses, and "backup_controllers" is near 10GB. This is a big deal as our servers have disks of 80GB size, and we are nearing a couple of GB's of free space on some of them.

          My quick search resulted in following:

          • backup_controllers and backup_logs tables are related in DB schema, see: [$MOODLE]/lib/db/upgrade.php
            Important lines: "$table = new xmldb_table('backup_logs');" and "$table->add_key('backupid', XMLDB_KEY_FOREIGN, array('backupid'), 'backup_controllers', array('backupid'));"
          • Records from both log and backup_controllers tables are removed in cron, see: [$MOODLE]/lib/cronlib.php
            Important lines: "// Delete old logs to save space (this might need a timer to slow it down...)" and "// Delete old backup_controllers and logs"
          • The above removal is controlled by CFG->loglifetime setting. The default value of "loglifetime" setting is "Never remove logs".
          • In case of our university this "loglifetime" setting is left default on our Moodle servers, and automatic course backup is turned on.

          For us "never remove logs" is required in our situation, because students may repeat courses they've failed on later semesters/years of their studies, if their ECTS points allow them. It's useful to have long log of their history on courses, to see all their previous actions, not only grades.

          I'm looking forward to help resolve this issue.

          Wiktor Wandachowicz

          Show
          Wiktor Wandachowicz added a comment - Recently we've upgraded our Moodle installations to "Moodle 2.2.1+ (Build: 20120119)" and we see the issue. On our university we have several Moodle installations, one per department. Some of them are actively used, some are in the works. On one of the servers there are over 1500 empty courses (reflecting organizational structure), ready to be filled with content. And "backup_controllers" table already takes 13GB of hard drive space. On another server there are 253 courses, and "backup_controllers" is near 10GB. This is a big deal as our servers have disks of 80GB size, and we are nearing a couple of GB's of free space on some of them. My quick search resulted in following: backup_controllers and backup_logs tables are related in DB schema, see: [$MOODLE] /lib/db/upgrade.php Important lines: "$table = new xmldb_table('backup_logs');" and "$table->add_key('backupid', XMLDB_KEY_FOREIGN, array('backupid'), 'backup_controllers', array('backupid'));" Records from both log and backup_controllers tables are removed in cron, see: [$MOODLE] /lib/cronlib.php Important lines: "// Delete old logs to save space (this might need a timer to slow it down...)" and "// Delete old backup_controllers and logs" The above removal is controlled by CFG->loglifetime setting. The default value of "loglifetime" setting is "Never remove logs". In case of our university this "loglifetime" setting is left default on our Moodle servers, and automatic course backup is turned on. For us "never remove logs" is required in our situation, because students may repeat courses they've failed on later semesters/years of their studies, if their ECTS points allow them. It's useful to have long log of their history on courses, to see all their previous actions, not only grades. I'm looking forward to help resolve this issue. Wiktor Wandachowicz
          Hide
          Yvonne Hamilton added a comment -

          Paul, we have Moodle 2.1.3 and have the same issue

          Show
          Yvonne Hamilton added a comment - Paul, we have Moodle 2.1.3 and have the same issue
          Chris Follin made changes -
          Labels partner triaged moodlerooms partner triaged
          Hide
          Chris Follin added a comment -

          We're on 2.1.3 and experiencing this issue. We are also a Moodle Partner and while this could be a problem for anybody, you can imagine how much more massive this problem becomes for us when we're hosting millions of courses across a large number of sites. Like other people have already said, this one table is over 75% of the entire database size in many cases. Databases are growing faster than anticipated, which could compromise site performance and reliability if left unchecked. We are truncating tables to mitigate this but doing so is only acceptable for the very short term. This issue needs a permanent resolution.

          Show
          Chris Follin added a comment - We're on 2.1.3 and experiencing this issue. We are also a Moodle Partner and while this could be a problem for anybody, you can imagine how much more massive this problem becomes for us when we're hosting millions of courses across a large number of sites. Like other people have already said, this one table is over 75% of the entire database size in many cases. Databases are growing faster than anticipated, which could compromise site performance and reliability if left unchecked. We are truncating tables to mitigate this but doing so is only acceptable for the very short term. This issue needs a permanent resolution.
          Hide
          Ray Lawrence added a comment -

          +1

          Show
          Ray Lawrence added a comment - +1
          Hide
          Michael de Raadt added a comment -

          Bumping this issue.

          Show
          Michael de Raadt added a comment - Bumping this issue.
          Michael de Raadt made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Simon Cole added a comment -

          We're on 2.2, and we've definitely got this issue!

          Show
          Simon Cole added a comment - We're on 2.2, and we've definitely got this issue!
          Paul Vaughan made changes -
          Labels moodlerooms partner triaged backup moodlerooms partner triaged
          Hide
          Paul Vaughan added a comment -

          Just truncated 10.4Gb, which was 86% of the whole database.

          Show
          Paul Vaughan added a comment - Just truncated 10.4Gb, which was 86% of the whole database.
          Eloy Lafuente (stronk7) made changes -
          Status Open [ 1 ] Development in progress [ 3 ]
          Chris Follin made changes -
          Affects Version/s 2.2 [ 10656 ]
          Ray Lawrence made changes -
          Labels backup moodlerooms partner triaged backup howtomoodle moodlerooms partner triaged
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Update: this is in progress and I wanted to have it ready to integration for today. But right now I've decided to target it for next week, to complete it properly. It's being fixed by:

          1) the creation of one new setting, 100% separated from $CFG->loglifetime (that seems that a lot of sites have it set for unlimited/big periods), to control the deletion of backup_controllers and backup_logs. Will default to 30 days and it will explain clearly that it's not recommended to use BIG periods at all.

          2) avoid saving the controllers fullinfo (serialized object) once the configuration phase is done. That's already implemented but have found some cases where it may be bypassed leading to BIG objects being inserted.

          3) after each successful backup, delete the serialized object from DB completely. It won't be needed anymore. It implies that, for failed backups, the serialized info will be there (can be useful to reproduce a bug) but, will be small (because of 2) and only will persist for 30 days (because of 1).

          And that is it. Once again, sorry for the delay and thanks for your patience. Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Update: this is in progress and I wanted to have it ready to integration for today. But right now I've decided to target it for next week, to complete it properly. It's being fixed by: 1) the creation of one new setting, 100% separated from $CFG->loglifetime (that seems that a lot of sites have it set for unlimited/big periods), to control the deletion of backup_controllers and backup_logs. Will default to 30 days and it will explain clearly that it's not recommended to use BIG periods at all. 2) avoid saving the controllers fullinfo (serialized object) once the configuration phase is done. That's already implemented but have found some cases where it may be bypassed leading to BIG objects being inserted. 3) after each successful backup, delete the serialized object from DB completely. It won't be needed anymore. It implies that, for failed backups, the serialized info will be there (can be useful to reproduce a bug) but, will be small (because of 2) and only will persist for 30 days (because of 1). And that is it. Once again, sorry for the delay and thanks for your patience. Ciao
          Hide
          Ray Lawrence added a comment -

          No problem.

          Small delay is nothing compared to the prospect of a big solution.

          Show
          Ray Lawrence added a comment - No problem. Small delay is nothing compared to the prospect of a big solution.
          Adam Olley made changes -
          Labels backup howtomoodle moodlerooms partner triaged backup howtomoodle moodlerooms netspot partner triaged
          Hide
          Tony Levi added a comment -

          Adding our +1, this is making some very big DBs for us too.

          Show
          Tony Levi added a comment - Adding our +1, this is making some very big DBs for us too.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Sending to integration (21, 22 and master).

          It includes 3 commits to separate the changes:

          1) The new setting to decide the ttl of backup logs (separating them from the main $CFG->loglifetime).

          2) Changes in backup controller to clean the "controller" column after finishing execution without errors.

          3) Changes in restore controller (100% copy of previous) to clean the column after finishing execution without errors.

          It applies to all operations (normal backups, import, duplicate, automated...).

          Also, on purpose, it's not allowed to configure the site to keep backup logs for > 1 year.

          And the only drawback I can imagine is that the 1st execution of the cleaning in cron can consume a lot of time due to the number of records to delete, specially in sites running mysql and having the general $CFG->loglifetime set to 0 (never delete logs).

          That's all. Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Sending to integration (21, 22 and master). It includes 3 commits to separate the changes: 1) The new setting to decide the ttl of backup logs (separating them from the main $CFG->loglifetime). 2) Changes in backup controller to clean the "controller" column after finishing execution without errors. 3) Changes in restore controller (100% copy of previous) to clean the column after finishing execution without errors. It applies to all operations (normal backups, import, duplicate, automated...). Also, on purpose, it's not allowed to configure the site to keep backup logs for > 1 year. And the only drawback I can imagine is that the 1st execution of the cleaning in cron can consume a lot of time due to the number of records to delete, specially in sites running mysql and having the general $CFG->loglifetime set to 0 (never delete logs). That's all. Ciao
          Eloy Lafuente (stronk7) made changes -
          Status Development in progress [ 3 ] Waiting for integration review [ 10010 ]
          Pull Master Diff URL https://github.com/stronk7/moodle/compare/master...MDL-29262
          Pull Master Branch MDL-29262
          Pull from Repository git://github.com/stronk7/moodle.git
          Fix Version/s 2.1.6 [ 12052 ]
          Fix Version/s 2.2.3 [ 12053 ]
          Fix Version/s STABLE backlog [ 10463 ]
          Testing Instructions Unsure, use Moodle normally, restore a lot of courses, or back up a lot of courses, regularly.

          I'm not sure if this is relevant but we restored close to 500 courses from other Moodle 1.9 and 2.0 installations, and are currently backing up 560 courses twice per week.
          NOTE: Testing this requires DB access.
          NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master

          1) With the patch applied, go to admin -> courses -> backup -> general

          2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days.

          3) Change it to 7 days and save changes.

          4) TEST: Verify the setting has been saved ok (7 days).

          5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case).

          6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok.

          7) TEST: The backup ends ok.

          8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string).

          9) Perform one import operation (from one course to another, pick some activity).

          10) TEST: The process ends ok.

          11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          12) Perform one duplicate activity operation (x2 icon under edit mode).

          13) TEST: The process ends ok.

          14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual".

          16) CLI execute: admin/cli/automated_backups.php

          17) TEST: The process ends ok.

          18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string).

          19) Count the number of records in the backup_controllers table.

          20) Update 2 of those records to have timecreated = 1. Annotate their ids.

          21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time).

          22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly.

          23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table.

          24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated.

          25) Repeat 1-24 for all fixed branches

          Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.

          Pull 2.1 Branch MDL-29262_21
          Pull 2.2 Diff URL https://github.com/stronk7/moodle/compare/MOODLE_22_STABLE...MDL-29262_22
          Pull 2.1 Diff URL https://github.com/stronk7/moodle/compare/MOODLE_21_STABLE...MDL-29262_21
          Pull 2.2 Branch MDL-29262_22
          Eloy Lafuente (stronk7) made changes -
          Labels backup howtomoodle moodlerooms netspot partner triaged backup docs_required howtomoodle moodlerooms netspot partner triaged ui_change
          Eloy Lafuente (stronk7) made changes -
          Currently in integration Yes [ 10041 ]
          Dan Poltawski made changes -
          Status Waiting for integration review [ 10010 ] Integration review in progress [ 10004 ]
          Integrator poltawski
          Hide
          Dan Poltawski added a comment -

          Thanks, this has been integrated now. Lets have some great testing for this issue!

          Show
          Dan Poltawski added a comment - Thanks, this has been integrated now. Lets have some great testing for this issue!
          Dan Poltawski made changes -
          Status Integration review in progress [ 10004 ] Waiting for testing [ 10005 ]
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Note: One extra commit has been added (and integrated) to each branch in order to fix some E_STRICT messages in tests were some "mock" controller classes were left behind the changes in core. Tidying testing instructions to run unit tests.

          Show
          Eloy Lafuente (stronk7) added a comment - Note: One extra commit has been added (and integrated) to each branch in order to fix some E_STRICT messages in tests were some "mock" controller classes were left behind the changes in core. Tidying testing instructions to run unit tests.
          Eloy Lafuente (stronk7) made changes -
          Testing Instructions NOTE: Testing this requires DB access.
          NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master

          1) With the patch applied, go to admin -> courses -> backup -> general

          2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days.

          3) Change it to 7 days and save changes.

          4) TEST: Verify the setting has been saved ok (7 days).

          5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case).

          6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok.

          7) TEST: The backup ends ok.

          8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string).

          9) Perform one import operation (from one course to another, pick some activity).

          10) TEST: The process ends ok.

          11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          12) Perform one duplicate activity operation (x2 icon under edit mode).

          13) TEST: The process ends ok.

          14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual".

          16) CLI execute: admin/cli/automated_backups.php

          17) TEST: The process ends ok.

          18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string).

          19) Count the number of records in the backup_controllers table.

          20) Update 2 of those records to have timecreated = 1. Annotate their ids.

          21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time).

          22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly.

          23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table.

          24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated.

          25) Repeat 1-24 for all fixed branches

          Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.

          NOTE: Testing this requires DB access.
          NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master

          1) With the patch applied, go to admin -> courses -> backup -> general

          2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days.

          3) Change it to 7 days and save changes.

          4) TEST: Verify the setting has been saved ok (7 days).

          5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case).

          6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok.

          7) TEST: The backup ends ok.

          8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string).

          9) Perform one import operation (from one course to another, pick some activity).

          10) TEST: The process ends ok.

          11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          12) Perform one duplicate activity operation (x2 icon under edit mode).

          13) TEST: The process ends ok.

          14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual".

          16) CLI execute: admin/cli/automated_backups.php

          17) TEST: The process ends ok.

          18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string).

          19) Count the number of records in the backup_controllers table.

          20) Update 2 of those records to have timecreated = 1. Annotate their ids.

          21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time).

          22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly.

          23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table.

          24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated.

          25) TEST: Execute controller unit tests:
              - For 21 and 22 (simpletest), in admin -> development -> unit tests, run tests in "backup/controller". Results: 6 passed, 0 exceptions and 0 fails (and no PHP Notice/Warning/Error/Strict problem in output/logs).
              - For master (phpunit), install and configure phpunit and run: "phpunit backup/controller/tests/controller_test.php". Results: OK (1 test, 6 assertions)

          26) Repeat 1-25 for all fixed branches

          Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.

          Eloy Lafuente (stronk7) made changes -
          Testing Instructions NOTE: Testing this requires DB access.
          NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master

          1) With the patch applied, go to admin -> courses -> backup -> general

          2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days.

          3) Change it to 7 days and save changes.

          4) TEST: Verify the setting has been saved ok (7 days).

          5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case).

          6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok.

          7) TEST: The backup ends ok.

          8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string).

          9) Perform one import operation (from one course to another, pick some activity).

          10) TEST: The process ends ok.

          11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          12) Perform one duplicate activity operation (x2 icon under edit mode).

          13) TEST: The process ends ok.

          14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual".

          16) CLI execute: admin/cli/automated_backups.php

          17) TEST: The process ends ok.

          18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string).

          19) Count the number of records in the backup_controllers table.

          20) Update 2 of those records to have timecreated = 1. Annotate their ids.

          21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time).

          22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly.

          23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table.

          24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated.

          25) TEST: Execute controller unit tests:
              - For 21 and 22 (simpletest), in admin -> development -> unit tests, run tests in "backup/controller". Results: 6 passed, 0 exceptions and 0 fails (and no PHP Notice/Warning/Error/Strict problem in output/logs).
              - For master (phpunit), install and configure phpunit and run: "phpunit backup/controller/tests/controller_test.php". Results: OK (1 test, 6 assertions)

          26) Repeat 1-25 for all fixed branches

          Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.

          NOTE: Testing this requires DB access.
          NOTE: This needs to be tested under 21_STABLE, 22_STABLE and master

          1) With the patch applied, go to admin -> courses -> backup -> general

          2) TEST: Verify that a new setting (backup | loglifetime) is shown and, if upgrade has happened, it shows, by default 30 days.

          3) Change it to 7 days and save changes.

          4) TEST: Verify the setting has been saved ok (7 days).

          5) Look to the contents of the backup_controllers table and annotate the last id there (the table may be empty, annotate 0 in that case).

          6) Perform one backup operation of some course. It doesn't matter if the course is big or no, any course is ok.

          7) TEST: The backup ends ok.

          8) TEST: Look to the contents of the backup_controllers table. One new record must be present there (id +1) and the size of its "controller" column must be 0 (empty string).

          9) Perform one import operation (from one course to another, pick some activity).

          10) TEST: The process ends ok.

          11) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          12) Perform one duplicate activity operation (x2 icon under edit mode).

          13) TEST: The process ends ok.

          14) TEST: Two new records must be in the backup_controllers table (backup & restore). Both with the "controller" column with size 0 (empty string).

          15) Go to admin -> courses -> backup -> automated backups and set the "active" setting to "manual".

          16) CLI execute: admin/cli/automated_backups.php

          17) TEST: The process ends ok.

          18) TEST: A bunch (as many as courses in the site) records have been created in the backup_controllers table. All them with the "controller" column with size 0 (empty string).

          19) Count the number of records in the backup_controllers table.

          20) Update 2 of those records to have timecreated = 1. Annotate their ids.

          21) CLI execute: admin/cli/cron.php (you will need to repeat this execution until getting "Deleted old backup records" on output because the cleanup is performed only 20% - randomly - of the time).

          22) TEST: Look to the backup_controllers table and verify that there are at least 2 records less than then counted @ point 19. Note that the difference may be bigger if you already had records in the table before starting this testing round. Next point will tell us more exactly.

          23) TEST: Verify that the records annotated @ 20 have been deleted from the backup_controllers table.

          24) TEST: Verify that there are no records in the backup_logs table with backupid being the ids annotated.

          25) TEST: Execute controller unit tests:
              - For 21 and 22 (simpletest), in admin -> development -> unit tests, run tests in "backup/controller". Results: 6 passed, 0 exceptions and 0 fails (and no PHP Notice/Warning/Error/Strict problem in output/logs).
              - For master (phpunit), install and configure phpunit and run: "phpunit backup/controller/tests/controller_test.php". Results: OK (1 test, 6 assertions) (and no PHP Notice/Warning/Error/Strict problem in output/logs).

          26) Repeat 1-25 for all fixed branches

          Final note: If testing this under Oracle, note that "empty string" is represented as " " (1-space char), hence the size of the column of some of the tests above will be 1 instead of 0. Only if testing under Oracle.

          Michael de Raadt made changes -
          Tester salvetore
          Hide
          Michael de Raadt added a comment -

          I'm taking myself off testing this issue as I'm not going to get to it today.

          Show
          Michael de Raadt added a comment - I'm taking myself off testing this issue as I'm not going to get to it today.
          Michael de Raadt made changes -
          Tester salvetore
          Michael de Raadt made changes -
          Tester ankit_frenz
          Hide
          Ankit Agarwal added a comment - - edited

          Hi Eloy,
          Results for master
          All works as expected except step 18
          In step 18, I had 9 new entries, where as I have only 8 courses.
          PHP unit tests ran as expected

          Results under 21
          All works as expected except step 18
          In step 18, I had 5 new entries, where as I have only 4 courses.
          Output of simpletest

          1/1 test cases complete: 6 passes, 0 fails and 0 exceptions.
          

          Please suggest if the issue @step18 is worth failing this test?
          Thanks

          Show
          Ankit Agarwal added a comment - - edited Hi Eloy, Results for master All works as expected except step 18 In step 18, I had 9 new entries, where as I have only 8 courses. PHP unit tests ran as expected Results under 21 All works as expected except step 18 In step 18, I had 5 new entries, where as I have only 4 courses. Output of simpletest 1/1 test cases complete: 6 passes, 0 fails and 0 exceptions. Please suggest if the issue @step18 is worth failing this test? Thanks
          Hide
          Ankit Agarwal added a comment -

          The extra entry was because of Front page which is treated as course.
          Working in all versions as expected (21 22 and master)
          Passing
          Thanks

          Show
          Ankit Agarwal added a comment - The extra entry was because of Front page which is treated as course. Working in all versions as expected (21 22 and master) Passing Thanks
          Ankit Agarwal made changes -
          Status Waiting for testing [ 10005 ] Testing in progress [ 10011 ]
          Ankit Agarwal made changes -
          Status Testing in progress [ 10011 ] Tested [ 10006 ]
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Thanks Ankit!

          Show
          Eloy Lafuente (stronk7) added a comment - Thanks Ankit!
          Hide
          Eloy Lafuente (stronk7) added a comment -

          This has been near becoming rejected, because it's not the best code you are able to produce.

          But, luckily, at the end, it has landed and has been spread to all repos out there.

          Many thanks and, don't forget it, keep improving your skills, you can!

          Closing, ciao

          Show
          Eloy Lafuente (stronk7) added a comment - This has been near becoming rejected, because it's not the best code you are able to produce. But, luckily, at the end, it has landed and has been spread to all repos out there. Many thanks and, don't forget it, keep improving your skills, you can! Closing, ciao
          Eloy Lafuente (stronk7) made changes -
          Status Tested [ 10006 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Currently in integration Yes [ 10041 ]
          Integration date 27/Apr/12
          Hide
          josé oliveira added a comment -

          Hi,

          I upgraded to 2.2.3, but the database keeps its size with backup_controllers table needlessly massive.

          How can i solve this?

          Thanks,

          JOsé Oliveira

          Show
          josé oliveira added a comment - Hi, I upgraded to 2.2.3, but the database keeps its size with backup_controllers table needlessly massive. How can i solve this? Thanks, JOsé Oliveira
          Hide
          Emanuel Delgado added a comment -

          Hi,
          The bug is fixed but it does not clean the table.
          Probably you won't need this information, so make a backup and empty the table with a statement like this:
          TRUNCATE mdl_backup_controllers

          Show
          Emanuel Delgado added a comment - Hi, The bug is fixed but it does not clean the table. Probably you won't need this information, so make a backup and empty the table with a statement like this: TRUNCATE mdl_backup_controllers
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Hi José & Emanuel.

          The fix for this bug includes code to delete all the old backup_controllers and logs by the cron execution.

          Just note the deletion of old data is part of the "random" cron execution, so it is not executed always, but only 20% of the times. So you will need to wait some time (say, 5-10 executions) before those records are effectively deleted.

          Just in case you're capturing cron executions output, you can look for "Deleted old backup records". When that is shown, the deletion has happened.

          So it's just a matter of waiting some time (normally within the 24h period).

          Finally, be careful about deleting records manually from the mdl_backup_controllers table. It has one child table (backup_logs) which records you also should be deleting of will become orphaned.

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Hi José & Emanuel. The fix for this bug includes code to delete all the old backup_controllers and logs by the cron execution. Just note the deletion of old data is part of the "random" cron execution, so it is not executed always, but only 20% of the times. So you will need to wait some time (say, 5-10 executions) before those records are effectively deleted. Just in case you're capturing cron executions output, you can look for "Deleted old backup records". When that is shown, the deletion has happened. So it's just a matter of waiting some time (normally within the 24h period). Finally, be careful about deleting records manually from the mdl_backup_controllers table. It has one child table (backup_logs) which records you also should be deleting of will become orphaned. Ciao
          Hide
          Emanuel Delgado added a comment -

          Hi Eloy,

          Thank you for this information, I was somehow misguided by my previous experience concerning this bug.
          Could you tell me where is this clean up code in the source so that I can take a look at it?
          Professional curiosity...

          Best Regards,
          ED

          Show
          Emanuel Delgado added a comment - Hi Eloy, Thank you for this information, I was somehow misguided by my previous experience concerning this bug. Could you tell me where is this clean up code in the source so that I can take a look at it? Professional curiosity... Best Regards, ED
          Hide
          Dan Marsden added a comment -

          HI Emanuel - if you hit the "source" tab above these comments on this tracker issue it will hopefully show you a list of the code changes related to this bug - you should be able to find the info you need there.

          Show
          Dan Marsden added a comment - HI Emanuel - if you hit the "source" tab above these comments on this tracker issue it will hopefully show you a list of the code changes related to this bug - you should be able to find the info you need there.
          Hide
          Emanuel Delgado added a comment -

          Hi Dan, that is exactly what I need, thanks for your answer.

          Show
          Emanuel Delgado added a comment - Hi Dan, that is exactly what I need, thanks for your answer.
          Hide
          Paul Vaughan added a comment -

          Thanks Eloy, this is great work, but also thanks for the note about the backup_logs table: as it was small I hadn't even looked into it, but just yesterday I truncated the backup_controllers table again as it reached 6Gb and didn't consider the _logs table. I'll truncate that now too so both tables have a blank slate to start from and can stay in sync.

          Cheers,

          Paul.

          Show
          Paul Vaughan added a comment - Thanks Eloy, this is great work, but also thanks for the note about the backup_logs table: as it was small I hadn't even looked into it, but just yesterday I truncated the backup_controllers table again as it reached 6Gb and didn't consider the _logs table. I'll truncate that now too so both tables have a blank slate to start from and can stay in sync. Cheers, Paul.
          Hide
          Mary Cooch added a comment - - edited

          I'm just going round looking at issues with docs_required. I wonder if anyone could either confirm this should have the label dev_docs_required or if not, if they could suggest appropriate documentation to go in the user docs and I will place it there? Thanks

          Show
          Mary Cooch added a comment - - edited I'm just going round looking at issues with docs_required. I wonder if anyone could either confirm this should have the label dev_docs_required or if not, if they could suggest appropriate documentation to go in the user docs and I will place it there? Thanks
          Hide
          Dan Poltawski added a comment -

          Hi Mary,

          I think the docs needed here are for the new settings in backup settings - loglifetime, which allows the 'expiry' of backup logs (which cane be large) to be controlled.

          Show
          Dan Poltawski added a comment - Hi Mary, I think the docs needed here are for the new settings in backup settings - loglifetime, which allows the 'expiry' of backup logs (which cane be large) to be controlled.
          Hide
          Mary Cooch added a comment -

          Thanks Dan. Documented in 2.3 and 2.2 now in General backup defaults.

          Show
          Mary Cooch added a comment - Thanks Dan. Documented in 2.3 and 2.2 now in General backup defaults.
          Mary Cooch made changes -
          Documentation link http://docs.moodle.org/23/en/Course_backup
          Labels backup docs_required howtomoodle moodlerooms netspot partner triaged ui_change backup howtomoodle moodlerooms netspot partner triaged ui_change
          Eloy Lafuente (stronk7) made changes -
          Link This issue is duplicated by MDL-29505 [ MDL-29505 ]

            People

            • Votes:
              35 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: