Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-64652

Personal data export dies after exhausting all the available memory

    XMLWordPrintable

    Details

      Description

      We are experiencing this on moodle.org running on latest 3.6.2 version. I've requested my personal data to be exported. Once the request is approved, the adhoc task is created and executed as expected. The output of that processing looks like this

      [PID 5857] Execute adhoc task: tool_dataprivacy\task\process_data_request_task
      [PID 5857] ... started 04:21:05. Current memory use 5.7MB.
      [PID 5857] Pre-processing request...
      [PID 5857]   Fetching data from 486 components (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing antivirus_clamav (1/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing availability_completion (2/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing availability_date (3/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing availability_grade (4/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing availability_group (5/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing availability_grouping (6/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing availability_profile (7/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_calculated (8/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_calculatedmulti (9/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_calculatedsimple (10/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_ddimageortext (11/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_ddmarker (12/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_ddwtos (13/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_description (14/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_essay (15/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_gapselect (16/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_match (17/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_missingtype (18/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_multianswer (19/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_multichoice (20/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_numerical (21/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_random (22/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_randomsamatch (23/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_shortanswer (24/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing qtype_truefalse (25/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_assign (26/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_assignment (27/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_book (28/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_chat (29/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_choice (30/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_data (31/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_feedback (32/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_folder (33/486) (Monday, 21 January 2019, 9:21 PM)
      [PID 5857]     Processing mod_forum (34/486) (Monday, 21 January 2019, 9:21 PM)
      

      where is gets stuck for a while (seems like 3 hours), and then eventually the process dies with

      PHP Fatal error:  Allowed memory size of 8589934592 bytes exhausted (tried to allocate 4096 bytes) in .../lib/dml/mysqli_native_moodle_recordset.php on line 55
      

      There are 4309 records in the forum_posts with userid = 1601 (my user id), to give an estimation about the amount of exported data.

      Those 8GB are roughly a half of the whole RAM available on that machine.

      We will appreciate an advise from HQ experts in the area, pinging Andrew Nicols and Eloy Lafuente (stronk7). Also letting know Carlo Polizzi as this has impact on our legal obligations, and Paul Greidanus, the site's sysadmin.

        Attachments

        1. adhoc_task.log
          27 kB
        2. adhoc_task2.log.gz
          2.06 MB
        3. adhoc_task3.log.gz
          2.01 MB

          Activity

            People

            • Assignee:
              dobedobedoh Andrew Nicols
              Reporter:
              mudrd8mz David Mudrák (@mudrd8mz)
              Peer reviewer:
              David Mudrák (@mudrd8mz)
              Integrator:
              Jun Pataleta
              Tester:
              CiBoT
              Participants:
              Component watchers:
              Andrew Nicols, Mathew May, Michael Hawkins, Shamim Rezaie, Simey Lameze, Matteo Scaramuccia, Jake Dallimore, Jun Pataleta, Ryan Wyllie, Andrew Nicols, Mathew May, Michael Hawkins, Shamim Rezaie, Simey Lameze
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Fix Release Date:
                11/Mar/19

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 hours
                2h