Moodle
  1. Moodle
  2. MDL-16271

Slow loading time of SCORM package content

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: 1.9.1, 1.9.2
    • Fix Version/s: 2.0
    • Component/s: Performance, SCORM
    • Labels:
      None
    • Environment:
      Any
    • Database:
      Any
    • Affected Branches:
      MOODLE_19_STABLE
    • Fixed Branches:
      MOODLE_20_STABLE
    • Rank:
      34889

      Description

      It takes minimum 1 second to load a file from the content of a SCORM package. Like this, if a SCO contains a html file with 60 small images, it would take at least 60 seconds to load the page. 60 seconds is generaly unacceptable for loading a single web page.

      The reason why a SCORM package content file takes at least a second of processing at 100% server cpu load: The file.php script used for getting files from the Moodle data directory to the user is a script including a collection of scripts with together thousands of lines of code.

      Possible solution: A SCORM package file parser script faster than file.php, added to the SCORM module especially for loading SCORM files. VDAB ( http://vdab.be ) is working on this solution.

      1. 20080910_fastFileParser.php
        1 kB
        Gerd Goetschalckx
      2. 20080910_loadSCO.patch
        1 kB
        Gerd Goetschalckx
      3. 20080926_fastFileParser.php
        2 kB
        Gerd Goetschalckx
      4. 20080926_loadSCO.php
        5 kB
        Gerd Goetschalckx
      5. patch_MDL-16271.txt
        2 kB
        Gerd Goetschalckx
      1. MDL_16271.jpeg
        145 kB

        Issue Links

          Activity

          Hide
          Dan Marsden added a comment -

          Hi Gerd,

          as mentioned in response to your discussion posting here:
          http://moodle.org/mod/forum/discuss.php?d=102356

          I haven't seen a package behave in the manner you mention - 1sec per file. - this doesn't seem right - in fact this sounds like a badly configured browser or server - all files should be available to download concurrently instead of 1 at a time.

          The File API for Moodle 2.0 is changing a lot, and the use of file.php will also be quite different - see here for more details:
          http://docs.moodle.org/en/Development:File_API

          We won't be looking at putting in a new file handler just for the SCORM objects at this stage. However this may be something we look into with the new File API in Moodle 2.0

          thanks,

          Dan

          Show
          Dan Marsden added a comment - Hi Gerd, as mentioned in response to your discussion posting here: http://moodle.org/mod/forum/discuss.php?d=102356 I haven't seen a package behave in the manner you mention - 1sec per file. - this doesn't seem right - in fact this sounds like a badly configured browser or server - all files should be available to download concurrently instead of 1 at a time. The File API for Moodle 2.0 is changing a lot, and the use of file.php will also be quite different - see here for more details: http://docs.moodle.org/en/Development:File_API We won't be looking at putting in a new file handler just for the SCORM objects at this stage. However this may be something we look into with the new File API in Moodle 2.0 thanks, Dan
          Hide
          Gerd Goetschalckx added a comment -

          A patch file demonstrating not to use file.php for SCORM content transfer, to speed up SCORM page loading.

          While loading a course, a series of scripts are executed on the server. We zoomed in on these scripts that load the SCOs and removed redundant code for a system running courses locally from the server on a php 5 and apache 2.

          We replaced the original file file.php with a Fastfileparser.php and redid the tests.
          Result: the client-side loading time drastically improved. A SCORM package preloading about 80 small image files took 57 seconds to load. On the same server using the patch it took only 9 seconds for the same screen.

          Maybe you can also test SCORM loading with a SCORM page containing a lot of small image files.

          Show
          Gerd Goetschalckx added a comment - A patch file demonstrating not to use file.php for SCORM content transfer, to speed up SCORM page loading. While loading a course, a series of scripts are executed on the server. We zoomed in on these scripts that load the SCOs and removed redundant code for a system running courses locally from the server on a php 5 and apache 2. We replaced the original file file.php with a Fastfileparser.php and redid the tests. Result: the client-side loading time drastically improved. A SCORM package preloading about 80 small image files took 57 seconds to load. On the same server using the patch it took only 9 seconds for the same screen. Maybe you can also test SCORM loading with a SCORM page containing a lot of small image files.
          Hide
          Dan Marsden added a comment -

          Hi Gerd, can you plese provide the fastfileparser file as well? - thanks.

          Show
          Dan Marsden added a comment - Hi Gerd, can you plese provide the fastfileparser file as well? - thanks.
          Hide
          Dan Marsden added a comment -

          also - it looks like you're using an old version of LoadSCO.php - we no longer use get_file_url() as slashargs don't work with SCORM so there's no point in checking for it. - see MDL-16060 for more details.

          Show
          Dan Marsden added a comment - also - it looks like you're using an old version of LoadSCO.php - we no longer use get_file_url() as slashargs don't work with SCORM so there's no point in checking for it. - see MDL-16060 for more details.
          Hide
          Gerd Goetschalckx added a comment -

          In the diagram you can see the concept of our solution. Please have a look at it. We would appreciate your feedback.

          Show
          Gerd Goetschalckx added a comment - In the diagram you can see the concept of our solution. Please have a look at it. We would appreciate your feedback.
          Hide
          Gerd Goetschalckx added a comment -

          Ok Dan,

          We usualy do all our test on Moodle version 1.9.2+, since that's the environment that we are preparing for production. We've also tested with the head version of CVS, with simular results. Please check out the diagram for a possible solution.

          Show
          Gerd Goetschalckx added a comment - Ok Dan, We usualy do all our test on Moodle version 1.9.2+, since that's the environment that we are preparing for production. We've also tested with the head version of CVS, with simular results. Please check out the diagram for a possible solution.
          Hide
          Dan Marsden added a comment -

          Hi Gerd,

          I'm still not sold on this. - You haven't provided an example of your fastfileparser.php. Also one of the critical functions of file.php is the security checks - to check if the user is allowed access to the file they have requested. How will your fastfileparser.php file address this?

          Show
          Dan Marsden added a comment - Hi Gerd, I'm still not sold on this. - You haven't provided an example of your fastfileparser.php. Also one of the critical functions of file.php is the security checks - to check if the user is allowed access to the file they have requested. How will your fastfileparser.php file address this?
          Hide
          Dan Marsden added a comment -

          Also - you still have not addressed my query regarding your claims on load time. 1 sec per file does not seem correct.

          Show
          Dan Marsden added a comment - Also - you still have not addressed my query regarding your claims on load time. 1 sec per file does not seem correct.
          Hide
          Petr Škoda added a comment -

          hello,

          the MDL-16088 contains major changes in scorm file handling, the files are now server by pluginfile.php, it is now possible to use much longer cache lifetime. I has more security checks, but I hope the perf will be better.

          I did not read your patch yet, going to do that later today

          thanks for the report anyway

          Show
          Petr Škoda added a comment - hello, the MDL-16088 contains major changes in scorm file handling, the files are now server by pluginfile.php, it is now possible to use much longer cache lifetime. I has more security checks, but I hope the perf will be better. I did not read your patch yet, going to do that later today thanks for the report anyway
          Hide
          Gerd Goetschalckx added a comment -

          Dan,
          Following your questions you can find:

          • A content package for testing the loading time. We will use this package for testing the issue from now on.
          • The fastfileparser.php
          • The loadsco.php that we use initiate the fastfileparser.php

          We are aware that this does not cover all security issues yet, but we think that we can look at this part of the code for performance improvement and then again at the whole (security issues included).

          Show
          Gerd Goetschalckx added a comment - Dan, Following your questions you can find: A content package for testing the loading time. We will use this package for testing the issue from now on. The fastfileparser.php The loadsco.php that we use initiate the fastfileparser.php We are aware that this does not cover all security issues yet, but we think that we can look at this part of the code for performance improvement and then again at the whole (security issues included).
          Hide
          Dan Marsden added a comment -

          Hi Gerd,

          as it stands, your patch introduces more issues than it solves.

          Security is very important for most Scorm designers. - using your patch anyone could obtain a scorm package.... in fact... using your patch, anyone could obtain any data they wanted from moodledata - including submitted assignments, and a range of data that should not be available without authenticating.

          The script you have written introduces this massive security hole into Moodle.

          Also - your script requires manual configuration of the moodledata path.

          the moodledata path is already stored in config.php - any other script should obtain that information from config.php and should not require manual configuration.

          thanks,

          Show
          Dan Marsden added a comment - Hi Gerd, as it stands, your patch introduces more issues than it solves. Security is very important for most Scorm designers. - using your patch anyone could obtain a scorm package.... in fact... using your patch, anyone could obtain any data they wanted from moodledata - including submitted assignments, and a range of data that should not be available without authenticating. The script you have written introduces this massive security hole into Moodle. Also - your script requires manual configuration of the moodledata path. the moodledata path is already stored in config.php - any other script should obtain that information from config.php and should not require manual configuration. thanks,
          Hide
          Gerd Goetschalckx added a comment -

          Dear Dan,

          You were quite right to point out the security issue in regard to the fastfileparser.
          Following your remarks we have:

          • Repeated the test with the most recent CVS version.
          • Have found a solution to make our fastFileParser script secure.

          Steps of the current system:
          1. User clicks on Enter to open the SCORM course
          2. SCORM player loads (player.php)
          3. SCORM player loads the imsmanifest.xml file to build SCORM tree
          4. SCORM player sets up API
          5. SCORM player waits 2 seconds
          6. SCORM player loads loadsco.php in iFrame that checks for SCO or asset to load.
          7. Loadsco.php forwards itself to an asset or SCO via file.php.
          8. That asset or SCO resolves all related content e.g. images throug the file.php.
          9. The page is then displayed.

          Steps of the solution proposed:
          1. Steps 1 to 4 are performed just once, the first time any page from the package is loaded.
          2. Steps 5 to 7 are replaced. The right SCO is immediately loaded in the iFrame through the fastfileparser.php
          3. Fastfileparser.php is secured by the use of session variables created by loadsco.php. This is much faster than config.php
          4. That asset or SCO resolves all related content e.g. images throug the fastfileparser.php
          5. The page is then displayed.
          Each other SCO or asset gets loaded in the iFrame without refreshing the SCORM player. This means that in our setup, steps 1 to 5 from the standard Moodle are redundant on subsequent pages.

          See attachments '20080926_fastFileParser.php' and ''20080926_loadSCO.php'.

          From our test results we think it is worth while to investigate this solution, although we still have the path to the data- folder and mimetypes hardcoded at this time. It would be nice to find a structual solution for this also.

          One last remark: we have noticed that in the new version from CVS uploading SCORM packages is limited to 1MB. At VDAB only 5 of a total of 160 SCORM packages are smaller than 1MB. Is there a special reason for this limit?

          Regards,

          Gerd

          Show
          Gerd Goetschalckx added a comment - Dear Dan, You were quite right to point out the security issue in regard to the fastfileparser. Following your remarks we have: Repeated the test with the most recent CVS version. Have found a solution to make our fastFileParser script secure. Steps of the current system: 1. User clicks on Enter to open the SCORM course 2. SCORM player loads (player.php) 3. SCORM player loads the imsmanifest.xml file to build SCORM tree 4. SCORM player sets up API 5. SCORM player waits 2 seconds 6. SCORM player loads loadsco.php in iFrame that checks for SCO or asset to load. 7. Loadsco.php forwards itself to an asset or SCO via file.php. 8. That asset or SCO resolves all related content e.g. images throug the file.php. 9. The page is then displayed. Steps of the solution proposed: 1. Steps 1 to 4 are performed just once, the first time any page from the package is loaded. 2. Steps 5 to 7 are replaced. The right SCO is immediately loaded in the iFrame through the fastfileparser.php 3. Fastfileparser.php is secured by the use of session variables created by loadsco.php. This is much faster than config.php 4. That asset or SCO resolves all related content e.g. images throug the fastfileparser.php 5. The page is then displayed. Each other SCO or asset gets loaded in the iFrame without refreshing the SCORM player. This means that in our setup, steps 1 to 5 from the standard Moodle are redundant on subsequent pages. See attachments '20080926_fastFileParser.php' and ''20080926_loadSCO.php'. From our test results we think it is worth while to investigate this solution, although we still have the path to the data- folder and mimetypes hardcoded at this time. It would be nice to find a structual solution for this also. One last remark: we have noticed that in the new version from CVS uploading SCORM packages is limited to 1MB. At VDAB only 5 of a total of 160 SCORM packages are smaller than 1MB. Is there a special reason for this limit? Regards, Gerd
          Hide
          Dan Marsden added a comment -

          Hi Gerd,

          haven't had a chance to look at this further - I have a range of other things that need my focus. Also - Hard coding vars isn't a viable option.

          might look at this again in a couple of months.

          thanks,

          Dan

          Show
          Dan Marsden added a comment - Hi Gerd, haven't had a chance to look at this further - I have a range of other things that need my focus. Also - Hard coding vars isn't a viable option. might look at this again in a couple of months. thanks, Dan
          Hide
          Gerd Goetschalckx added a comment -

          Hi Dan,

          I understand that you're very bussy at the moment. We keep on working on our version of the SCORM player because the performance issue is really critical, and even a show stopper, for going in production with Moodle. We will try to follow as closely as possible the guidelines for development for Moodle, this in order to provide in the end an alternative module for the SCORM player.
          We will publish all our results so that you can follow the progress, and we invite other members of the Moodle community to do some testing. Feedback can be sent to me by e-mail: gerd.goetschalckx@vdab.be.

          The current version of our SCORM player includes:

          • a fast SCORM file parser
          • in line security
          • a dynamic player for the SCORM packages (This means that the player and the SCORM tree are loaded only once when opening the SCORM package.)

          The next problem that we will tackle is the SCORM communication API object.

          Remark: in the bottom left corner the activities block should be visible (= a customisation for VDAB)

          Kind regards,
          Gerd

          Show
          Gerd Goetschalckx added a comment - Hi Dan, I understand that you're very bussy at the moment. We keep on working on our version of the SCORM player because the performance issue is really critical, and even a show stopper, for going in production with Moodle. We will try to follow as closely as possible the guidelines for development for Moodle, this in order to provide in the end an alternative module for the SCORM player. We will publish all our results so that you can follow the progress, and we invite other members of the Moodle community to do some testing. Feedback can be sent to me by e-mail: gerd.goetschalckx@vdab.be. The current version of our SCORM player includes: a fast SCORM file parser in line security a dynamic player for the SCORM packages (This means that the player and the SCORM tree are loaded only once when opening the SCORM package.) The next problem that we will tackle is the SCORM communication API object. Remark: in the bottom left corner the activities block should be visible (= a customisation for VDAB) Kind regards, Gerd
          Hide
          Dan Marsden added a comment -

          Hi Gerd,

          Thanks for all the work you put towards this, but I'm now closing this issue as won't fix - The SCORM module in Moodle MUST use core functions for serving files to meet internal coding guidelines/security guidelines etc.

          handling of files in Moodle 2.0 has also changed significantly meaning that the above patches won't apply to Moodle 2.0 either.

          There is a really nice player in the new Moodle 2.0 IMSCP module that would be good to replace the existing SCORM player, and this uses Moodle core functions for it's file handling in a correct manner. If someone has some time to put towards this it will improve the UI significantly.

          thanks again for sharing your work - feel free to provide any other SCORM related patches you might have and we'll review them for inclusion.

          thanks,

          Show
          Dan Marsden added a comment - Hi Gerd, Thanks for all the work you put towards this, but I'm now closing this issue as won't fix - The SCORM module in Moodle MUST use core functions for serving files to meet internal coding guidelines/security guidelines etc. handling of files in Moodle 2.0 has also changed significantly meaning that the above patches won't apply to Moodle 2.0 either. There is a really nice player in the new Moodle 2.0 IMSCP module that would be good to replace the existing SCORM player, and this uses Moodle core functions for it's file handling in a correct manner. If someone has some time to put towards this it will improve the UI significantly. thanks again for sharing your work - feel free to provide any other SCORM related patches you might have and we'll review them for inclusion. thanks,

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: