Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-73808

Moodle 4.0 question bank upgrade uses a lot of memory and is slow

    XMLWordPrintable

Details

    • MOODLE_400_STABLE
    • MOODLE_400_STABLE
    • master_MDL-73808-question-bank-upgrade-bkp
    • Hide

      Environments required:
      1. PgSQL
      2. MySQL
      3. Oracle
      Please test in all these three environments. PHP version can be set accordingly without any requirement.

      Migrating from 3.9 (basic test as it changes the migration code and performance test)

      1. Standup a new 3.9 site
      2. Log in as an admin
      3. Create a course and 50K questions with tags using the test_73808_solo.php script
      4. Confirm:
        1. That the new course and quiz is created by the script and questions are available in the course context
      5. Add some random questions using the same question category and add some tags for each one of them
      6. Add some regular questions from the question bank
      7. Logout and login as a student
      8. Attempt the quiz
      9. Logout from the account
      10. Change the code using this tracker
      11. Upgrade the site
      12. Log in as a site admin
      13. Confirm:
        1. That the created questions are available in the question bank
        2. That the created questions have only one version and the version is 1
        3. That the added random questions are available in the quiz
        4. That the attempts are available in the quiz
        5. The added tags for the random questions are available in the quiz
      14. Logout and login as a student
      15. Attempt the quiz
      16. Confirm:
        1. The quiz attempt works as expected and all the questions are available.
      17. Confirm:
        1. That the upgrade happens as expected without timeout or asking for extra memory.

       

      Migrating from 3.11 (basic test as it changes the migration code and performance test)

      1. Standup a new 3.11 site
      2. Log in as an admin
      3. Create a course and 50K questions with tags using the test_73808_solo.php script
      4. Confirm:
        1. That the new course and quiz is created by the script and questions are available in the course context
      5. Add some random questions using the same question category and add some tags for each one of them
      6. Add some regular questions from the question bank
      7. Logout and login as a student
      8. Attempt the quiz
      9. Logout from the account
      10. Change the code using this tracker
      11. Upgrade the site
      12. Log in as a site admin
      13. Confirm:
        1. That the created questions are available in the question bank
        2. That the created questions have only one version and the version is 1
        3. That the added random questions are available in the quiz
        4. That the attempts are available in the quiz
        5. The added tags for the random questions are available in the quiz
      14. Logout and login as a student
      15. Attempt the quiz
      16. Confirm:
        1. The quiz attempt works as expected and all the questions are available.
      17. Confirm:
        1. That the upgrade happens as expected without timeout or asking for extra memory.

       

      Show
      Environments required: 1. PgSQL 2. MySQL 3. Oracle Please test in all these three environments. PHP version can be set accordingly without any requirement. Migrating from 3.9 (basic test as it changes the migration code and performance test ) Standup a new 3.9 site Log in as an admin Create a course and 50K questions with tags using the test_73808_solo.php script Confirm: That the new course and quiz is created by the script and questions are available in the course context Add some random questions using the same question category and add some tags for each one of them Add some regular questions from the question bank Logout and login as a student Attempt the quiz Logout from the account Change the code using this tracker Upgrade the site Log in as a site admin Confirm: That the created questions are available in the question bank That the created questions have only one version and the version is 1 That the added random questions are available in the quiz That the attempts are available in the quiz The added tags for the random questions are available in the quiz Logout and login as a student Attempt the quiz Confirm: The quiz attempt works as expected and all the questions are available. Confirm: That the upgrade happens as expected without timeout or asking for extra memory.   Migrating from 3.11 (basic test as it changes the migration code and performance test) Standup a new 3.11 site Log in as an admin Create a course and 50K questions with tags using the test_73808_solo.php script Confirm: That the new course and quiz is created by the script and questions are available in the course context Add some random questions using the same question category and add some tags for each one of them Add some regular questions from the question bank Logout and login as a student Attempt the quiz Logout from the account Change the code using this tracker Upgrade the site Log in as a site admin Confirm: That the created questions are available in the question bank That the created questions have only one version and the version is 1 That the added random questions are available in the quiz That the attempts are available in the quiz The added tags for the random questions are available in the quiz Logout and login as a student Attempt the quiz Confirm: The quiz attempt works as expected and all the questions are available. Confirm: That the upgrade happens as expected without timeout or asking for extra memory.  

    Description

      When I upgrade my Moodle environment from Moodle 3.9.12  (Build: 20220117) or from Moodle 4.0dev+ (Build: 20220129), the upgrade fails.

      When I look in the upgrade logfile I see these errors:

      [#######################################################

      [Exception ignored in shutdown function upgrade_finished_handler: Fout bij het schrijven van de databank

      Potential coding error - active database transaction detected during request shutdown:

      • line 263 of /lib/dml/moodle_read_slave_trait.php: call to moodle_database->start_delegated_transaction()
      • line 1308 of /lib/db/upgradelib.php: call to mysqli_native_moodle_database->start_delegated_transaction()
      • line 3912 of /lib/db/upgrade.php: call to upgrade_migrate_question_table()
      • line 1875 of /lib/upgradelib.php: call to xmldb_main_upgrade()
      • line 196 of /admin/cli/upgrade.php: call to upgrade_core()

       I checked the code in /lib/db/upgrade.php where my upgrade fails:

      if ($oldversion < 2022020200.02)

      { // Next, split question records into the new tables. upgrade_migrate_question_table(); // Main savepoint reached. upgrade_main_savepoint(true, 2022020200.02); }

      And this function where it goes wrong in /lib/db/upgradelib.php:

      /**

      • Split question table in 2 new tables:
        *
      • question_bank_entries
      • question_versions
        *
      • Move the random questions records to the following table:
      • question_set_reference
        *
      • Move the question related records from quiz_slots table to:
      • question_reference
        *
      • Move the tag related data from quiz_slot_tags to:
      • question_references
        *
      • For more information: https://moodle.org/mod/forum/discuss.php?d=417599#p1688163
        */
        function upgrade_migrate_question_table(): void {
        global $DB;

      // Maximum size of array.
      $maxlength = 30000;

      // Array of question_versions objects.
      $questionversions = [];

      // Array of question_set_references objects.
      $questionsetreferences = [];

      // The actual update/insert done with multiple DB access, so we do it in a transaction.
      $transaction = $DB->start_delegated_transaction(); <= this is the line where the error happens <=

      Attachments

        1. 0001-MDL-73808-question-Alternative-question-upgrade-appr.patch
          10 kB
        2. image-2022-03-29-20-58-11-372.png
          image-2022-03-29-20-58-11-372.png
          108 kB
        3. MDL-73808_postgres_v311_1.png
          MDL-73808_postgres_v311_1.png
          55 kB
        4. MDL-73808_postgres_v311_2.png
          MDL-73808_postgres_v311_2.png
          129 kB
        5. MDL-73808_postgres_v311_3.png
          MDL-73808_postgres_v311_3.png
          105 kB
        6. MDL-73808_postgres_v311_4.png
          MDL-73808_postgres_v311_4.png
          132 kB
        7. MDL-73808_postgres_v39_1.png
          MDL-73808_postgres_v39_1.png
          56 kB
        8. MDL-73808_postgres_v39_2.png
          MDL-73808_postgres_v39_2.png
          123 kB
        9. MDL-73808_postgres_v39_3.png
          MDL-73808_postgres_v39_3.png
          102 kB
        10. MDL-73808_postgres_v39_4.png
          MDL-73808_postgres_v39_4.png
          130 kB
        11. moodle-upgrade.20220209082533-1.log
          5 kB
        12. moodle-upgrade.20220209133943.log
          50 kB
        13. moodle-upgrade.20220214130702.log
          49 kB
        14. moodle-upgrade.20220307113635.log
          51 kB
        15. mysql-questionbank.png
          mysql-questionbank.png
          138 kB
        16. mysql-question-version.png
          mysql-question-version.png
          123 kB
        17. mysql-upgrade.png
          mysql-upgrade.png
          63 kB
        18. oracle_cli_upgrade_39.png
          oracle_cli_upgrade_39.png
          59 kB
        19. random question tag filter.png
          random question tag filter.png
          75 kB
        20. sqlsrv-39,311-to-master.png
          sqlsrv-39,311-to-master.png
          24 kB
        21. sqlsrv-questionbank.png
          sqlsrv-questionbank.png
          91 kB
        22. sqlsrv-question-version-1.png
          sqlsrv-question-version-1.png
          90 kB
        23. test_73808_solo.php
          3 kB
        24. test_73808.php
          3 kB

        Issue Links

          Activity

            People

              safatshahin Safat Shahin
              alainraap Alain Raap
              Tim Hunt Tim Hunt
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Angelia Dela Cruz Angelia Dela Cruz
              Matteo Scaramuccia, David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo, Safat Shahin, Tim Hunt, Amaia Anabitarte, Bas Brands, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Kevin Percy, Laurent David, Mathew May, Mihail Geshoski, Sabina Abellan, Sara Arjona (@sarjona), Shamim Rezaie
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                19/Apr/22

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 4 hours, 7 minutes
                  2d 4h 7m