Details

    • Database:
      MySQL
    • Testing Instructions:
      Hide

      Download a file from moodle that is larger than the php memory_limit value in php.ini

      Show
      Download a file from moodle that is larger than the php memory_limit value in php.ini
    • Workaround:
      Hide

      Increase the php memory limit to be larger than any of the files that need to be downloaded.

      Show
      Increase the php memory limit to be larger than any of the files that need to be downloaded.
    • Affected Branches:
      MOODLE_21_STABLE
    • Rank:
      18842

      Description

      Files cannot be downloaded unless the php memory limit is increased to be equal or larger than the file.

        Issue Links

          Activity

          Hide
          Dongsheng Cai added a comment -

          We use readfile() to send file content, this function won't read the entire file to memory, it might be something else, does you error log indicate the memory issue?
          http://php.net/manual/en/function.readfile.php

          Show
          Dongsheng Cai added a comment - We use readfile() to send file content, this function won't read the entire file to memory, it might be something else, does you error log indicate the memory issue? http://php.net/manual/en/function.readfile.php
          Hide
          George Chen added a comment -

          We have a client faced the same issues here with 230 Mbytes of course backup can't be download and ported to another Moodle site. Do anyone have better idea to work around?

          Show
          George Chen added a comment - We have a client faced the same issues here with 230 Mbytes of course backup can't be download and ported to another Moodle site. Do anyone have better idea to work around?
          Hide
          Marko Vidberg added a comment -

          I notice that Moodle 1.9 used to use the custom function readfile_chunked() to get around the bugginess of PHP's readfile() function. Perhaps that needs to be re-implemented in Moodle 2.x?

          Show
          Marko Vidberg added a comment - I notice that Moodle 1.9 used to use the custom function readfile_chunked() to get around the bugginess of PHP's readfile() function. Perhaps that needs to be re-implemented in Moodle 2.x?
          Hide
          Jonathan Moore added a comment -

          I have been reviewing this issue. I believe we found that M 2.1 was not memory bound when tested on an Ubuntu workstation build, but does exhibit this problem consistently on our production builds using both CentOS with php 5.3.10 with eAccellerator and on RHEL 5.x KVM VMs using PHP 5.3.x with APC. I am double checking but I believe in testing disabling the PHP cache didn't resolve the issue. I have also checked and output buffer is disabled. As far as we can tell this is a legitimate problem with the change to readfile(). We have had at least 4 clients hit this problem. Anything more that we can do to move this forward? We are happy to provide more information or direct access to a test environment.

          Show
          Jonathan Moore added a comment - I have been reviewing this issue. I believe we found that M 2.1 was not memory bound when tested on an Ubuntu workstation build, but does exhibit this problem consistently on our production builds using both CentOS with php 5.3.10 with eAccellerator and on RHEL 5.x KVM VMs using PHP 5.3.x with APC. I am double checking but I believe in testing disabling the PHP cache didn't resolve the issue. I have also checked and output buffer is disabled. As far as we can tell this is a legitimate problem with the change to readfile(). We have had at least 4 clients hit this problem. Anything more that we can do to move this forward? We are happy to provide more information or direct access to a test environment.
          Hide
          Petr Škoda added a comment -

          Hello, this problem will be finally completely resolved in 2.3 if you configure your servers to support X-Sendfile and enable it in config.php.

          Show
          Petr Škoda added a comment - Hello, this problem will be finally completely resolved in 2.3 if you configure your servers to support X-Sendfile and enable it in config.php.
          Hide
          Petr Škoda added a comment -

          I have just successfully downloaded ubuntu install iso (700MB) file with 128MB of allowed PHP memory (with buffering enabled).

          Show
          Petr Škoda added a comment - I have just successfully downloaded ubuntu install iso (700MB) file with 128MB of allowed PHP memory (with buffering enabled).
          Hide
          Petr Škoda added a comment -

          I am sorry I can not diagnose or fix this if I can not reproduce it, so far it looks like a buggy PHP or a server configuration issue. I am proposing to close this as "not a bug". Please test 2.3 with X-Sendfile enabled and report if it resolves your problems with file downloads, thanks.

          Show
          Petr Škoda added a comment - I am sorry I can not diagnose or fix this if I can not reproduce it, so far it looks like a buggy PHP or a server configuration issue. I am proposing to close this as "not a bug". Please test 2.3 with X-Sendfile enabled and report if it resolves your problems with file downloads, thanks.
          Hide
          Dakota Duff added a comment -

          Petr, we're still experiencing the problem and from what I recall it may be in part do to zlib compression. Would you be able to test your setup with the following php.ini settings before marking as "not a bug"?

          zlib.output_compression => On => On
          zlib.output_compression_level => 2 => 2
          

          Thank you.

          Show
          Dakota Duff added a comment - Petr, we're still experiencing the problem and from what I recall it may be in part do to zlib compression. Would you be able to test your setup with the following php.ini settings before marking as "not a bug"? zlib.output_compression => On => On zlib.output_compression_level => 2 => 2 Thank you.
          Hide
          Petr Škoda added a comment -

          Oh, yes, zlib compression is not much supported in moodle and I am afraid no core developer is testing it. PHP needs memory to compress the output and it needs to buffer it somewhere...

          Could you please try

          @ini_set('zlib.output_compression', 'Off');

          at the beginning of your /pluginfile.php ? That might resolve it for you.

          Show
          Petr Škoda added a comment - Oh, yes, zlib compression is not much supported in moodle and I am afraid no core developer is testing it. PHP needs memory to compress the output and it needs to buffer it somewhere... Could you please try @ini_set('zlib.output_compression', 'Off'); at the beginning of your /pluginfile.php ? That might resolve it for you.
          Hide
          Petr Škoda added a comment -

          But at the same time it might break file serving headers completely...

          Show
          Petr Škoda added a comment - But at the same time it might break file serving headers completely...
          Hide
          Dakota Duff added a comment -

          I know we had tried setting at runtime and had the same results (download failing, or rather downloading with zero bytes).

          I believe earlier testing showed that zlib.output_compression disabled at the system level (i.e. disabled in php.ini and service restarted) allowed the downloads to work as expected.

          Show
          Dakota Duff added a comment - I know we had tried setting at runtime and had the same results (download failing, or rather downloading with zero bytes). I believe earlier testing showed that zlib.output_compression disabled at the system level (i.e. disabled in php.ini and service restarted) allowed the downloads to work as expected.
          Hide
          Petr Škoda added a comment -

          Thanks a lot for the confirmation, that is what I expected. The new X-Sendfile options should be compatible with zlib compression PHP setting.

          Jonathan and George, could you please verify your server configuration and disable zlib compression or any other output handlers and retry if it works without it?

          Show
          Petr Škoda added a comment - Thanks a lot for the confirmation, that is what I expected. The new X-Sendfile options should be compatible with zlib compression PHP setting. Jonathan and George, could you please verify your server configuration and disable zlib compression or any other output handlers and retry if it works without it?
          Hide
          Mike Churchward added a comment -

          Hi Petr... Trying to understand the crux of the issue here. Are you saying that in 2.3, zlib can be enabled and this problem won't exist? That X-Sendfile will play fine with zlib?

          Show
          Mike Churchward added a comment - Hi Petr... Trying to understand the crux of the issue here. Are you saying that in 2.3, zlib can be enabled and this problem won't exist? That X-Sendfile will play fine with zlib?
          Hide
          Petr Škoda added a comment -

          Hi, there are probably problems that might affect file serving memory use and each server is different, the point of x-sendfile to offload the content serving to web server, the gzipping would is ignored there together with all the buffering - only headers are used.

          Show
          Petr Škoda added a comment - Hi, there are probably problems that might affect file serving memory use and each server is different, the point of x-sendfile to offload the content serving to web server, the gzipping would is ignored there together with all the buffering - only headers are used.
          Hide
          Petr Škoda added a comment -

          Hmm, there are several zlib issues to be fixed soon in PHP 5.4.4:
          . Fixed bug #61820 (using ob_gzhandler will complain about headers already
          sent when no compression). (Mike)
          . Fixed bug #61443 (can't change zlib.output_compression on the fly). (Mike)
          . Fixed bug #60761 (zlib.output_compression fails on refresh). (Mike)

          Show
          Petr Škoda added a comment - Hmm, there are several zlib issues to be fixed soon in PHP 5.4.4: . Fixed bug #61820 (using ob_gzhandler will complain about headers already sent when no compression). (Mike) . Fixed bug #61443 (can't change zlib.output_compression on the fly). (Mike) . Fixed bug #60761 (zlib.output_compression fails on refresh). (Mike)
          Hide
          Jonathan Moore added a comment -

          I can confirm what Dakota posted earlier. It is definitely zlib compression. We have a pre-pend file to disable compression on problem files. I can also confirm that adding any of the following does not fix the problem

          '/pluginfile.php',
          '/lib/filelib.php',
          '/lib/filestorage/file_exceptions.php',
          '/lib/filestorage/file_storage.php',
          '/lib/filestorage/zip_packer.php',
          '/lib/filebrowser/file_browser.php',

          Show
          Jonathan Moore added a comment - I can confirm what Dakota posted earlier. It is definitely zlib compression. We have a pre-pend file to disable compression on problem files. I can also confirm that adding any of the following does not fix the problem '/pluginfile.php', '/lib/filelib.php', '/lib/filestorage/file_exceptions.php', '/lib/filestorage/file_storage.php', '/lib/filestorage/zip_packer.php', '/lib/filebrowser/file_browser.php',
          Hide
          Michael de Raadt added a comment -

          A potential solution for 2.2 has been proposed by Jason Ilicic in a duplicate issue.

          https://github.com/jasonilicic/moodle/commit/dee939a8a4d7a818b4ca212ed788793e283253ef

          Show
          Michael de Raadt added a comment - A potential solution for 2.2 has been proposed by Jason Ilicic in a duplicate issue. https://github.com/jasonilicic/moodle/commit/dee939a8a4d7a818b4ca212ed788793e283253ef
          Hide
          Petr Škoda added a comment -

          Hello, I am not responsible for the file api any more, reassigning. In any case this has to be fixed by somebody who can replicate the problem, I can not, sorry.

          Show
          Petr Škoda added a comment - Hello, I am not responsible for the file api any more, reassigning. In any case this has to be fixed by somebody who can replicate the problem, I can not, sorry.
          Hide
          Dakota Duff added a comment -

          We switched to Apache's mod_deflate and are no longer experiencing this problem. It was, however, still an issue when using zlib compression in PHP.

          Show
          Dakota Duff added a comment - We switched to Apache's mod_deflate and are no longer experiencing this problem. It was, however, still an issue when using zlib compression in PHP.
          Hide
          Martin Dougiamas added a comment -

          Can someone check if Jason's patch makes sense here?

          Show
          Martin Dougiamas added a comment - Can someone check if Jason's patch makes sense here?
          Hide
          Petr Škoda added a comment -

          We could also add the zlib.output_compression setting test to the environment page and discourage it on production sites, that should hopefully eliminate this problem or at least admins would know where is the problem.

          Show
          Petr Škoda added a comment - We could also add the zlib.output_compression setting test to the environment page and discourage it on production sites, that should hopefully eliminate this problem or at least admins would know where is the problem.

            People

            • Votes:
              10 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated: