Moodle
  1. Moodle
  2. MDL-38268

download_file_content fails under certain conditions

    Details

    • Database:
      Any
    • Testing Instructions:
      Hide
      1. Go to SCORM package settings (Site administration ► Plugins ► Activity modules ► SCORM package), and enable "downloaded package type" (scorm | allowtypelocalsync)
      2. Verify you have the default value of 56 set for "curltimeoutkbitrate" in Performance settings (Site administration ► Server ► Performance).
      3. Create a new SCORM module, choosing "Downloaded package" as the Type and entering a remote URL for downloading from
      4. Enter the following URL – http://www.ostyn.com/standards/scorm/samples/proddingsco.zip
      5. Save
      Show
      Go to SCORM package settings ( Site administration ► Plugins ► Activity modules ► SCORM package ), and enable "downloaded package type" (scorm | allowtypelocalsync) Verify you have the default value of 56 set for "curltimeoutkbitrate" in Performance settings ( Site administration ► Server ► Performance ). Create a new SCORM module, choosing "Downloaded package" as the Type and entering a remote URL for downloading from Enter the following URL – http://www.ostyn.com/standards/scorm/samples/proddingsco.zip Save
    • Workaround:
      Hide

      Set the global config setting curltimeoutkbitrate to zero and the SCORM will be created successfully.

      Show
      Set the global config setting curltimeoutkbitrate to zero and the SCORM will be created successfully.
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
    • Fixed Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE
    • Pull from Repository:
    • Pull 2.4 Branch:
      MDL-38268-m24
    • Pull Master Branch:
      MDL-38268-master
    • Rank:
      48118

      Description

      Steps to reproduce::
      1) Go to SCORM package settings, and enable "downloaded package type" (scorm | allowtypelocalsync)
      2) Verify you have the default value of 56 set for "curltimeoutkbitrate" in Settings.
      3) Create a new SCORM module, choosing "Downloaded package" as the Type and entering a remote URL for downloading from
      4) Save

      Now you can set curltimeoutkbitrate to zero and the SCORM will be created successfully. I'm attaching my screenshot. I believe Heather was receiving a slightly different error, but having a similar outcome.

      The problem seems to be in lib/filelib.php around line 1230:

             curl_setopt_array ($ch, array(
                  CURLOPT_RETURNTRANSFER => true,
                  CURLOPT_NOBODY         => false)
              );
      

      The problem here is that simply setting CURLOPT_NOBODY to false is insufficient.

      From the PHP documentation:
      CURLOPT_NOBODY - TRUE to exclude the body from the output. Request method is then set to HEAD. Changing this to FALSE does not change it to GET.

      CURLOPT_HTTPGET - TRUE to reset the HTTP request method to GET. Since GET is the default, this is only necessary if the request method has been changed.

      Changing as such allowed the file to successfully download if the curltimeoutkbitrate value was set:

              curl_setopt_array ($ch, array(
                  CURLOPT_RETURNTRANSFER => true,
                  CURLOPT_NOBODY         => false,
                  CURLOPT_HTTPGET        => true)
              );
      

      It may be worthwhile to check for this in other areas of Moodle also?

        Activity

        Hide
        Dan Marsden added a comment -

        Hi Mike - I see you added triage tag but I don't think you've actually triaged this....
        http://docs.moodle.org/dev/Bug_triage

        this bug also seems to focus more on a solution than reporting the issue the user had - it also relates to the filelib area and not SCORM code so probably shouldn't be assigned to me

        Dakota - can you please provide more details on the issue the user had - what they were doing and how it was causing a problem?

        Show
        Dan Marsden added a comment - Hi Mike - I see you added triage tag but I don't think you've actually triaged this.... http://docs.moodle.org/dev/Bug_triage this bug also seems to focus more on a solution than reporting the issue the user had - it also relates to the filelib area and not SCORM code so probably shouldn't be assigned to me Dakota - can you please provide more details on the issue the user had - what they were doing and how it was causing a problem?
        Hide
        Dan Marsden added a comment -

        assigning to moodle.com and removing triage tag - the solution proposed relates to core filelib code.

        Show
        Dan Marsden added a comment - assigning to moodle.com and removing triage tag - the solution proposed relates to core filelib code.
        Hide
        Heather Williams added a comment - - edited

        The issue was found when a client was using the "Download Package" SCORM type and specifying a URL to a SCORM file on another server. As soon as you would try to save the SCORM activity you would immediately get a cURL timeout and the SCORM activity would not be created.

        Setting the curltimeoutkbitrate to zero is a work around, as it allows the SCORM package to be retrieved from the URL and everything works as it should. But with the curltimeoutkbitrate set to anything other than zero (including default of 56) you immediately get a cURL timeout when trying to save (not a delayed timeout, it happens seconds after hitting save).

        The change in code that Dakota is suggesting, we tested on a site and allowed the SCORM activity to be created with the curltimeoutkbitrate set to default.

        Please let me know if you need more information on this.

        Show
        Heather Williams added a comment - - edited The issue was found when a client was using the "Download Package" SCORM type and specifying a URL to a SCORM file on another server. As soon as you would try to save the SCORM activity you would immediately get a cURL timeout and the SCORM activity would not be created. Setting the curltimeoutkbitrate to zero is a work around, as it allows the SCORM package to be retrieved from the URL and everything works as it should. But with the curltimeoutkbitrate set to anything other than zero (including default of 56) you immediately get a cURL timeout when trying to save (not a delayed timeout, it happens seconds after hitting save). The change in code that Dakota is suggesting, we tested on a site and allowed the SCORM activity to be created with the curltimeoutkbitrate set to default. Please let me know if you need more information on this.
        Hide
        Dakota Duff added a comment - - edited

        Thanks, Heather.

        Show
        Dakota Duff added a comment - - edited Thanks, Heather.
        Hide
        Mike Churchward added a comment - - edited

        Hi Dan... Maybe I don't understand "triage"? I thought that meant confirm that it is a bug. We did do that, and confirmed the solution.

        (edit)
        Ah. I responded without reading thoroughly. I will review the "triage" process link. I was not aware of that.

        I have also noticed that we really didn't provide enough, or the proper information. My mistake. I had a lot of the information presented to me in a back channel.

        Show
        Mike Churchward added a comment - - edited Hi Dan... Maybe I don't understand "triage"? I thought that meant confirm that it is a bug. We did do that, and confirmed the solution. (edit) Ah. I responded without reading thoroughly. I will review the "triage" process link. I was not aware of that. I have also noticed that we really didn't provide enough, or the proper information. My mistake. I had a lot of the information presented to me in a back channel.
        Hide
        Justin Filip added a comment -

        I can't replicate this issue locally on either the latest MOODLE_23_STABLE or MOODLE_24_STABLE code.

        Show
        Justin Filip added a comment - I can't replicate this issue locally on either the latest MOODLE_23_STABLE or MOODLE_24_STABLE code.
        Hide
        Dan Marsden added a comment -

        no worries Mike!

        ideally it would be good if someone updated the description on this bug to:

        • provide replication steps (If Justin can't reproduce this that is a worry)
        • provide error messages that were shown to user or displayed in logs(screenshots are good)

        for a bug to be fixed in core we usually have to have some way of reproducing the bug so that it can go through QA (and someone can verify your fix)

        Show
        Dan Marsden added a comment - no worries Mike! ideally it would be good if someone updated the description on this bug to: provide replication steps (If Justin can't reproduce this that is a worry) provide error messages that were shown to user or displayed in logs(screenshots are good) for a bug to be fixed in core we usually have to have some way of reproducing the bug so that it can go through QA (and someone can verify your fix)
        Hide
        Dakota Duff added a comment -

        Dan,

        I tried to get ahold of Justin, but must have just missed him. I was able to replicate with the instructions provided by Heather.

        Step-by-step:
        1) Go to SCORM package settings, and enable "downloaded package type" (scorm | allowtypelocalsync)
        2) Verify you have the default value of 56 set for "curltimeoutkbitrate" in Settings.
        3) Create a new SCORM module, choosing "Downloaded package" as the Type and entering a remote URL for downloading from
        4) Save

        Now you can set curltimeoutkbitrate to zero and the SCORM will be created successfully. I'm attaching my screenshot. I believe Heather was receiving a slightly different error, but having a similar outcome.

        Here are some more links that document the solution we describe above:
        https://bugs.php.net/bug.php?id=44584
        http://curl.haxx.se/mail/curlphp-2008-03/0072.html
        http://curl.haxx.se/mail/archive-2005-07/0073.html

        Show
        Dakota Duff added a comment - Dan, I tried to get ahold of Justin, but must have just missed him. I was able to replicate with the instructions provided by Heather. Step-by-step: 1) Go to SCORM package settings, and enable "downloaded package type" (scorm | allowtypelocalsync) 2) Verify you have the default value of 56 set for "curltimeoutkbitrate" in Settings. 3) Create a new SCORM module, choosing "Downloaded package" as the Type and entering a remote URL for downloading from 4) Save Now you can set curltimeoutkbitrate to zero and the SCORM will be created successfully. I'm attaching my screenshot. I believe Heather was receiving a slightly different error, but having a similar outcome. Here are some more links that document the solution we describe above: https://bugs.php.net/bug.php?id=44584 http://curl.haxx.se/mail/curlphp-2008-03/0072.html http://curl.haxx.se/mail/archive-2005-07/0073.html
        Hide
        Dan Marsden added a comment -

        thanks Dakota - that's improved the quality of this report a lot more - leaving this for Michael de Radt/Moodle HQ to complete triage and assign to a real person to look into.

        Show
        Dan Marsden added a comment - thanks Dakota - that's improved the quality of this report a lot more - leaving this for Michael de Radt/Moodle HQ to complete triage and assign to a real person to look into.
        Hide
        Justin Filip added a comment -

        The problem and the solution make complete sense but, yeah, I can't replicate it on my own dev machine. The steps Dakota posted are what I did to try and replicate the bug. This might be one of those weird ones where it's dependent on the specific version of cURL or PHP used on the server.

        Show
        Justin Filip added a comment - The problem and the solution make complete sense but, yeah, I can't replicate it on my own dev machine. The steps Dakota posted are what I did to try and replicate the bug. This might be one of those weird ones where it's dependent on the specific version of cURL or PHP used on the server.
        Hide
        Justin Filip added a comment -

        Added patch repository info / links.

        Show
        Justin Filip added a comment - Added patch repository info / links.
        Hide
        Matteo Scaramuccia added a comment -

        For the record, I'm able to reproduce the cURL unexpected behaviour using the code from https://bugs.php.net/bug.php?id=44584 in my dev machine (CentOS/5.9, PHP/5.3.8): bool(false) Operation timed out after 3 seconds with 0 out of 2523 bytes received, note the number of bytes received but not moved to the output.
        I'll try to reproduce it in the next days using a Moodle instance.

        Show
        Matteo Scaramuccia added a comment - For the record, I'm able to reproduce the cURL unexpected behaviour using the code from https://bugs.php.net/bug.php?id=44584 in my dev machine ( CentOS/5.9, PHP/5.3.8 ): bool(false) Operation timed out after 3 seconds with 0 out of 2523 bytes received , note the number of bytes received but not moved to the output. I'll try to reproduce it in the next days using a Moodle instance.
        Hide
        Michael de Raadt added a comment -

        Thanks to the people who have worked on this so far.

        I've assigned this to you, Justin. You can push this up for peer review if you think it is ready.

        Show
        Michael de Raadt added a comment - Thanks to the people who have worked on this so far. I've assigned this to you, Justin. You can push this up for peer review if you think it is ready.
        Hide
        Justin Filip added a comment -

        Thanks, Michael. Yeah, this looks good. I'm pretty sure it's related to a version of something(s) somewhere as I tried the test code in that bug that Matteo Scaramuccia linked to but it still worked correctly for me. The change to fix the problem doesn't cause any issues for me, though so I'm confident this won't cause any issues.

        Show
        Justin Filip added a comment - Thanks, Michael. Yeah, this looks good. I'm pretty sure it's related to a version of something(s) somewhere as I tried the test code in that bug that Matteo Scaramuccia linked to but it still worked correctly for me. The change to fix the problem doesn't cause any issues for me, though so I'm confident this won't cause any issues.
        Hide
        Matteo Scaramuccia added a comment -

        Hi All,
        here is my test using Moodle 2.5dev (Build: 20130228) in my guest picking up the ZIP package from my laptop (host).

        • W/o the fix, shortly:
          cURL request for "http://matteo-nb.scaramuccia.home:81/demo%20TOC.zip" failed with: transfer closed with 20028 bytes remaining to read (18)

          Note the pattern: failed but with bytes, bingo!

        • W/ the fix: no error at all and the package has been published.

        HTH,
        Matteo

        Show
        Matteo Scaramuccia added a comment - Hi All, here is my test using Moodle 2.5dev (Build: 20130228) in my guest picking up the ZIP package from my laptop (host). W/o the fix, shortly: cURL request for "http://matteo-nb.scaramuccia.home:81/demo%20TOC.zip" failed with: transfer closed with 20028 bytes remaining to read (18) Note the pattern: failed but with bytes, bingo! W/ the fix: no error at all and the package has been published. HTH, Matteo
        Hide
        Andrew Davis added a comment -

        Hi. Justin, if you haven't already would you mind checking the other places we mention CURLOPT_NOBODY and checking that they don't have the same problem.

        [Y] Syntax
        [NA] Output
        [Y] Whitespace
        [NA] Language
        [NA] Databases
        [?] Testing
        [NA] Security
        [NA] Documentation
        [Y] Git
        [Y] Sanity check

        In the testing instructions would it be possible to provide the tester with a sample URL to use. I suspect it's unlikely that a tester will have a remote scorm URL handy.

        Otherwise, submit for integration whenever you're ready.

        Show
        Andrew Davis added a comment - Hi. Justin, if you haven't already would you mind checking the other places we mention CURLOPT_NOBODY and checking that they don't have the same problem. [Y] Syntax [NA] Output [Y] Whitespace [NA] Language [NA] Databases [?] Testing [NA] Security [NA] Documentation [Y] Git [Y] Sanity check In the testing instructions would it be possible to provide the tester with a sample URL to use. I suspect it's unlikely that a tester will have a remote scorm URL handy. Otherwise, submit for integration whenever you're ready.
        Hide
        Justin Filip added a comment -

        Hi Andrew, thanks for the feedback. I put in the URL for a test SCORM package and the only other place where CURLOPT_NOBODY is used is in the Amazon S3 plugin (/repository/s3/S3.php) and it's not used in the same way as it is in filelib. In the S3 file it's only ever enabled. The problem with filelib was that it was enabled to issue a HEAD request and then disabled to switch back to a GET within the context of the same block of code.

        Show
        Justin Filip added a comment - Hi Andrew, thanks for the feedback. I put in the URL for a test SCORM package and the only other place where CURLOPT_NOBODY is used is in the Amazon S3 plugin ( /repository/s3/S3.php ) and it's not used in the same way as it is in filelib. In the S3 file it's only ever enabled. The problem with filelib was that it was enabled to issue a HEAD request and then disabled to switch back to a GET within the context of the same block of code.
        Hide
        Dakota Duff added a comment -

        Thanks!

        Show
        Dakota Duff added a comment - Thanks!
        Hide
        Andrew Davis added a comment -

        Cool. I'm not sure if you are able to submit issues for integration so I have done that for you as it sounds like this is all ready to go.

        Show
        Andrew Davis added a comment - Cool. I'm not sure if you are able to submit issues for integration so I have done that for you as it sounds like this is all ready to go.
        Hide
        Eloy Lafuente (stronk7) added a comment -

        Integrated (23, 24 and master), thanks!

        Show
        Eloy Lafuente (stronk7) added a comment - Integrated (23, 24 and master), thanks!
        Hide
        Rossiani Wijaya added a comment -

        Hi Justin,

        I followed your testing instructions and the page still displaying the error.

        Unknown exception related to local files (Can not fetch file form URL)
        
        More information about this error
        Debug info:
        Error code: storedfileproblem
        Stack trace:
        
            line 933 of /lib/filestorage/file_storage.php: file_exception thrown
            line 206 of /mod/scorm/locallib.php: call to file_storage->create_file_from_url()
            line 151 of /mod/scorm/lib.php: call to scorm_parse()
            line 447 of /course/modedit.php: call to scorm_add_instance()
        
        Output buffer: <div class="notifytiny">cURL request for "http://www.ostyn.com/standards/scorm/samples/proddingsco.zip" failed with: Couldn't resolve proxy 'does.not.work.example.com' (5)<ul style="text-align: left"><li>line 1268 of /lib/filelib.php: call to debugging()</li><li>line 931 of /lib/filestorage/file_storage.php: call to download_file_content()</li><li>line 206 of /mod/scorm/locallib.php: call to file_storage->create_file_from_url()</li><li>line 151 of /mod/scorm/lib.php: call to scorm_parse()</li><li>line 447 of /course/modedit.php: call to scorm_add_instance()</li></ul></div>
        

        Could you confirm this for integration branches?

        Thanks
        Rosie

        Show
        Rossiani Wijaya added a comment - Hi Justin, I followed your testing instructions and the page still displaying the error. Unknown exception related to local files (Can not fetch file form URL) More information about this error Debug info: Error code: storedfileproblem Stack trace: line 933 of /lib/filestorage/file_storage.php: file_exception thrown line 206 of /mod/scorm/locallib.php: call to file_storage->create_file_from_url() line 151 of /mod/scorm/lib.php: call to scorm_parse() line 447 of /course/modedit.php: call to scorm_add_instance() Output buffer: <div class= "notifytiny" >cURL request for "http: //www.ostyn.com/standards/scorm/samples/proddingsco.zip" failed with: Couldn't resolve proxy 'does.not.work.example.com' (5)<ul style= "text-align: left" ><li>line 1268 of /lib/filelib.php: call to debugging()</li><li>line 931 of /lib/filestorage/file_storage.php: call to download_file_content()</li><li>line 206 of /mod/scorm/locallib.php: call to file_storage->create_file_from_url()</li><li>line 151 of /mod/scorm/lib.php: call to scorm_parse()</li><li>line 447 of /course/modedit.php: call to scorm_add_instance()</li></ul></div> Could you confirm this for integration branches? Thanks Rosie
        Hide
        Matteo Scaramuccia added a comment -

        Hi Rosie,
        it seems you've configured a fake proxy in your Moodle instance: Couldn't resolve proxy 'does.not.work.example.com' (5). The test requires just curltimeoutkbitrate to be equal to the default value, at least different from 0 i.e. no proxy at all.

        HTH,
        Matteo

        Show
        Matteo Scaramuccia added a comment - Hi Rosie, it seems you've configured a fake proxy in your Moodle instance: Couldn't resolve proxy 'does.not.work.example.com' (5) . The test requires just curltimeoutkbitrate to be equal to the default value, at least different from 0 i.e. no proxy at all. HTH, Matteo
        Hide
        Rossiani Wijaya added a comment -

        Doh.... I forgot I have it setup last week. Sorry..

        Re-testing and this is working as expected.

        Tested for 2.3, 2.4 and master.

        Test passed.

        Show
        Rossiani Wijaya added a comment - Doh.... I forgot I have it setup last week. Sorry.. Re-testing and this is working as expected. Tested for 2.3, 2.4 and master. Test passed.
        Hide
        Eloy Lafuente (stronk7) added a comment -

        This issue has been integrated upstream and is now available both via git and cvs (and in some hours, via mirrors and downloads).

        Thanks!

        PS: Yay, legacy template messages. Yes, you're ok, we don't have CVS anymore!

        Show
        Eloy Lafuente (stronk7) added a comment - This issue has been integrated upstream and is now available both via git and cvs (and in some hours, via mirrors and downloads). Thanks! PS: Yay, legacy template messages. Yes, you're ok, we don't have CVS anymore!

          People

          • Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: