Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69207

Add library file caching to h5p

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Before installing the patch

      We emulate remote file storage, adding a delay when retrieving the content of the files:
      1. Download the attached file fakeremotefilestorage.diff in your Moodle root folder.
      2. Apply the patch:

      patch -p1 < fakeremotefilestorage.diff

      3. Log in as admin.
      4. Go to 'Content bank' (in the drawer for Boost theme, in Navigation block > Site pages for Classic theme).
      5. Open DevTools by pressing Control+Shift+J or Command+Option+J (Mac).
      6. Click on the Network tab.
      7. Type embed.php? into the Filter text box.
      8. Click on 'Upload' button.
      9. Upload attached interactivevideo.h5p file and 'Save changes'.
      10. Wait until the page displays the H5P content.
      11. Write down the load time of the embed.php request.
      12. Click on the Close button.
      13. Repeat steps 8-11: the timing should be similar.

      After installing the patch and upgrading the site

      14. Go to 'Content bank' (in the drawer for Boost theme, in Navigation block > Site pages for Classic theme).
      15. Repeat steps 8-11: the cache is filled with the H5P library files.
      16. Repeat steps 8-11: H5P content display should be faster.
      17. Confirm that the load time in step 16 is significantly less than the previous ones.

      Show
      Before installing the patch We emulate remote file storage, adding a delay when retrieving the content of the files: 1. Download the attached file fakeremotefilestorage.diff in your Moodle root folder. 2. Apply the patch: patch -p1 < fakeremotefilestorage. diff 3. Log in as admin. 4. Go to 'Content bank' (in the drawer for Boost theme, in Navigation block > Site pages for Classic theme). 5. Open DevTools by pressing Control+Shift+J or Command+Option+J (Mac). 6. Click on the Network tab. 7. Type embed.php? into the Filter text box. 8. Click on 'Upload' button. 9. Upload attached interactivevideo.h5p file and 'Save changes'. 10. Wait until the page displays the H5P content. 11. Write down the load time of the embed.php request. 12. Click on the Close button. 13. Repeat steps 8-11: the timing should be similar. After installing the patch and upgrading the site 14. Go to 'Content bank' (in the drawer for Boost theme, in Navigation block > Site pages for Classic theme). 15. Repeat steps 8-11: the cache is filled with the H5P library files. 16. Repeat steps 8-11: H5P content display should be faster. 17. Confirm that the load time in step 16 is significantly less than the previous ones.
    • Affected Branches:
      MOODLE_38_STABLE, MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_310_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-69207-master
    • Pull Master Diff URL:

      Description

      When adding an h5p activity, Moodle downloads around 1500 files from the h5p library filearea in order to generate a zip file which then gets pushed back into the file store.

      With an alternate file storage provider like Catalyst's ObjectFS backed by a remote store like AWS S3, downloading these library files one by one can be an extremely slow task - around 2 minutes to fetch 1500 files with the additional s3 api overhead.

      It would be nice to have these files available in a muc cache.

      Storing all library files currently available comes out to around 32mb (deduplicated by contenthash) on disk or just over 2mb in an igbinary+gzip redis store and I was able to bring down the activity creation time to 2 seconds.

      We already have this in production and is working very well.

        Attachments

        1. after.png
          after.png
          84 kB
        2. before.png
          before.png
          88 kB
        3. fakeremotefilestorage.diff
          0.7 kB
        4. interactivevideo.h5p
          1.99 MB
        5. Test_AfterPatch_FirstUpload.png
          Test_AfterPatch_FirstUpload.png
          238 kB
        6. Test_AfterPatch_NextUpload.png
          Test_AfterPatch_NextUpload.png
          200 kB
        7. Test_BeforePatch_FirstUpload.png
          Test_BeforePatch_FirstUpload.png
          156 kB
        8. Test_BeforePatch_NextUpload.png
          Test_BeforePatch_NextUpload.png
          181 kB

          Activity

            People

            Assignee:
            jtomkinson Jordan Tomkinson
            Reporter:
            jtomkinson Jordan Tomkinson
            Peer reviewer:
            cameron1729
            Integrator:
            Víctor Déniz Falcón
            Tester:
            Carlos Escobedo
            Participants:
            Component watchers:
            Matteo Scaramuccia, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Sara Arjona (@sarjona), Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Sara Arjona (@sarjona)
            Votes:
            7 Vote for this issue
            Watchers:
            10 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Fix Release Date:
              9/Nov/20

                Time Tracking

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 days, 1 hour, 10 minutes
                2d 1h 10m