Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Not a bug
    • Affects Version/s: 2.1
    • Fix Version/s: None
    • Component/s: Course, Files API, Unknown
    • Labels:
    • Environment:
      Centos 5
    • Database:
      MySQL
    • Testing Instructions:
      Hide

      Download a file from moodle that is larger than the php memory_limit value in php.ini

      Show
      Download a file from moodle that is larger than the php memory_limit value in php.ini
    • Workaround:
      Hide

      Increase the php memory limit to be larger than any of the files that need to be downloaded.

      Show
      Increase the php memory limit to be larger than any of the files that need to be downloaded.
    • Affected Branches:
      MOODLE_21_STABLE

      Description

      Files cannot be downloaded unless the php memory limit is increased to be equal or larger than the file.

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            Dongsheng Cai added a comment -

            We use readfile() to send file content, this function won't read the entire file to memory, it might be something else, does you error log indicate the memory issue?
            http://php.net/manual/en/function.readfile.php

            Show
            Dongsheng Cai added a comment - We use readfile() to send file content, this function won't read the entire file to memory, it might be something else, does you error log indicate the memory issue? http://php.net/manual/en/function.readfile.php
            Hide
            George Chen added a comment -

            We have a client faced the same issues here with 230 Mbytes of course backup can't be download and ported to another Moodle site. Do anyone have better idea to work around?

            Show
            George Chen added a comment - We have a client faced the same issues here with 230 Mbytes of course backup can't be download and ported to another Moodle site. Do anyone have better idea to work around?
            Hide
            Marko Vidberg added a comment -

            I notice that Moodle 1.9 used to use the custom function readfile_chunked() to get around the bugginess of PHP's readfile() function. Perhaps that needs to be re-implemented in Moodle 2.x?

            Show
            Marko Vidberg added a comment - I notice that Moodle 1.9 used to use the custom function readfile_chunked() to get around the bugginess of PHP's readfile() function. Perhaps that needs to be re-implemented in Moodle 2.x?
            Hide
            Jonathan Moore added a comment -

            I have been reviewing this issue. I believe we found that M 2.1 was not memory bound when tested on an Ubuntu workstation build, but does exhibit this problem consistently on our production builds using both CentOS with php 5.3.10 with eAccellerator and on RHEL 5.x KVM VMs using PHP 5.3.x with APC. I am double checking but I believe in testing disabling the PHP cache didn't resolve the issue. I have also checked and output buffer is disabled. As far as we can tell this is a legitimate problem with the change to readfile(). We have had at least 4 clients hit this problem. Anything more that we can do to move this forward? We are happy to provide more information or direct access to a test environment.

            Show
            Jonathan Moore added a comment - I have been reviewing this issue. I believe we found that M 2.1 was not memory bound when tested on an Ubuntu workstation build, but does exhibit this problem consistently on our production builds using both CentOS with php 5.3.10 with eAccellerator and on RHEL 5.x KVM VMs using PHP 5.3.x with APC. I am double checking but I believe in testing disabling the PHP cache didn't resolve the issue. I have also checked and output buffer is disabled. As far as we can tell this is a legitimate problem with the change to readfile(). We have had at least 4 clients hit this problem. Anything more that we can do to move this forward? We are happy to provide more information or direct access to a test environment.
            Hide
            Petr Skoda added a comment -

            Hello, this problem will be finally completely resolved in 2.3 if you configure your servers to support X-Sendfile and enable it in config.php.

            Show
            Petr Skoda added a comment - Hello, this problem will be finally completely resolved in 2.3 if you configure your servers to support X-Sendfile and enable it in config.php.
            Hide
            Petr Skoda added a comment -

            I have just successfully downloaded ubuntu install iso (700MB) file with 128MB of allowed PHP memory (with buffering enabled).

            Show
            Petr Skoda added a comment - I have just successfully downloaded ubuntu install iso (700MB) file with 128MB of allowed PHP memory (with buffering enabled).
            Hide
            Petr Skoda added a comment -

            I am sorry I can not diagnose or fix this if I can not reproduce it, so far it looks like a buggy PHP or a server configuration issue. I am proposing to close this as "not a bug". Please test 2.3 with X-Sendfile enabled and report if it resolves your problems with file downloads, thanks.

            Show
            Petr Skoda added a comment - I am sorry I can not diagnose or fix this if I can not reproduce it, so far it looks like a buggy PHP or a server configuration issue. I am proposing to close this as "not a bug". Please test 2.3 with X-Sendfile enabled and report if it resolves your problems with file downloads, thanks.
            Hide
            Dakota Duff added a comment -

            Petr, we're still experiencing the problem and from what I recall it may be in part do to zlib compression. Would you be able to test your setup with the following php.ini settings before marking as "not a bug"?

            zlib.output_compression => On => On
            zlib.output_compression_level => 2 => 2
            

            Thank you.

            Show
            Dakota Duff added a comment - Petr, we're still experiencing the problem and from what I recall it may be in part do to zlib compression. Would you be able to test your setup with the following php.ini settings before marking as "not a bug"? zlib.output_compression => On => On zlib.output_compression_level => 2 => 2 Thank you.
            Hide
            Petr Skoda added a comment -

            Oh, yes, zlib compression is not much supported in moodle and I am afraid no core developer is testing it. PHP needs memory to compress the output and it needs to buffer it somewhere...

            Could you please try

            @ini_set('zlib.output_compression', 'Off');

            at the beginning of your /pluginfile.php ? That might resolve it for you.

            Show
            Petr Skoda added a comment - Oh, yes, zlib compression is not much supported in moodle and I am afraid no core developer is testing it. PHP needs memory to compress the output and it needs to buffer it somewhere... Could you please try @ini_set('zlib.output_compression', 'Off'); at the beginning of your /pluginfile.php ? That might resolve it for you.
            Hide
            Petr Skoda added a comment -

            But at the same time it might break file serving headers completely...

            Show
            Petr Skoda added a comment - But at the same time it might break file serving headers completely...
            Hide
            Dakota Duff added a comment -

            I know we had tried setting at runtime and had the same results (download failing, or rather downloading with zero bytes).

            I believe earlier testing showed that zlib.output_compression disabled at the system level (i.e. disabled in php.ini and service restarted) allowed the downloads to work as expected.

            Show
            Dakota Duff added a comment - I know we had tried setting at runtime and had the same results (download failing, or rather downloading with zero bytes). I believe earlier testing showed that zlib.output_compression disabled at the system level (i.e. disabled in php.ini and service restarted) allowed the downloads to work as expected.
            Hide
            Petr Skoda added a comment -

            Thanks a lot for the confirmation, that is what I expected. The new X-Sendfile options should be compatible with zlib compression PHP setting.

            Jonathan and George, could you please verify your server configuration and disable zlib compression or any other output handlers and retry if it works without it?

            Show
            Petr Skoda added a comment - Thanks a lot for the confirmation, that is what I expected. The new X-Sendfile options should be compatible with zlib compression PHP setting. Jonathan and George, could you please verify your server configuration and disable zlib compression or any other output handlers and retry if it works without it?
            Hide
            Mike Churchward added a comment -

            Hi Petr... Trying to understand the crux of the issue here. Are you saying that in 2.3, zlib can be enabled and this problem won't exist? That X-Sendfile will play fine with zlib?

            Show
            Mike Churchward added a comment - Hi Petr... Trying to understand the crux of the issue here. Are you saying that in 2.3, zlib can be enabled and this problem won't exist? That X-Sendfile will play fine with zlib?
            Hide
            Petr Skoda added a comment -

            Hi, there are probably problems that might affect file serving memory use and each server is different, the point of x-sendfile to offload the content serving to web server, the gzipping would is ignored there together with all the buffering - only headers are used.

            Show
            Petr Skoda added a comment - Hi, there are probably problems that might affect file serving memory use and each server is different, the point of x-sendfile to offload the content serving to web server, the gzipping would is ignored there together with all the buffering - only headers are used.
            Hide
            Petr Skoda added a comment -

            Hmm, there are several zlib issues to be fixed soon in PHP 5.4.4:
            . Fixed bug #61820 (using ob_gzhandler will complain about headers already
            sent when no compression). (Mike)
            . Fixed bug #61443 (can't change zlib.output_compression on the fly). (Mike)
            . Fixed bug #60761 (zlib.output_compression fails on refresh). (Mike)

            Show
            Petr Skoda added a comment - Hmm, there are several zlib issues to be fixed soon in PHP 5.4.4: . Fixed bug #61820 (using ob_gzhandler will complain about headers already sent when no compression). (Mike) . Fixed bug #61443 (can't change zlib.output_compression on the fly). (Mike) . Fixed bug #60761 (zlib.output_compression fails on refresh). (Mike)
            Hide
            Jonathan Moore added a comment -

            I can confirm what Dakota posted earlier. It is definitely zlib compression. We have a pre-pend file to disable compression on problem files. I can also confirm that adding any of the following does not fix the problem

            '/pluginfile.php',
            '/lib/filelib.php',
            '/lib/filestorage/file_exceptions.php',
            '/lib/filestorage/file_storage.php',
            '/lib/filestorage/zip_packer.php',
            '/lib/filebrowser/file_browser.php',

            Show
            Jonathan Moore added a comment - I can confirm what Dakota posted earlier. It is definitely zlib compression. We have a pre-pend file to disable compression on problem files. I can also confirm that adding any of the following does not fix the problem '/pluginfile.php', '/lib/filelib.php', '/lib/filestorage/file_exceptions.php', '/lib/filestorage/file_storage.php', '/lib/filestorage/zip_packer.php', '/lib/filebrowser/file_browser.php',
            Hide
            Michael de Raadt added a comment -

            A potential solution for 2.2 has been proposed by Jason Ilicic in a duplicate issue.

            https://github.com/jasonilicic/moodle/commit/dee939a8a4d7a818b4ca212ed788793e283253ef

            Show
            Michael de Raadt added a comment - A potential solution for 2.2 has been proposed by Jason Ilicic in a duplicate issue. https://github.com/jasonilicic/moodle/commit/dee939a8a4d7a818b4ca212ed788793e283253ef
            Hide
            Petr Skoda added a comment -

            Hello, I am not responsible for the file api any more, reassigning. In any case this has to be fixed by somebody who can replicate the problem, I can not, sorry.

            Show
            Petr Skoda added a comment - Hello, I am not responsible for the file api any more, reassigning. In any case this has to be fixed by somebody who can replicate the problem, I can not, sorry.
            Hide
            Dakota Duff added a comment -

            We switched to Apache's mod_deflate and are no longer experiencing this problem. It was, however, still an issue when using zlib compression in PHP.

            Show
            Dakota Duff added a comment - We switched to Apache's mod_deflate and are no longer experiencing this problem. It was, however, still an issue when using zlib compression in PHP.
            Hide
            Martin Dougiamas added a comment -

            Can someone check if Jason's patch makes sense here?

            Show
            Martin Dougiamas added a comment - Can someone check if Jason's patch makes sense here?
            Hide
            Petr Skoda added a comment -

            We could also add the zlib.output_compression setting test to the environment page and discourage it on production sites, that should hopefully eliminate this problem or at least admins would know where is the problem.

            Show
            Petr Skoda added a comment - We could also add the zlib.output_compression setting test to the environment page and discourage it on production sites, that should hopefully eliminate this problem or at least admins would know where is the problem.
            Hide
            Marina Glancy added a comment -

            I'm pretty sure this is not an issue any more but will leave it open for a while in case somebody has another opinion.

            Show
            Marina Glancy added a comment - I'm pretty sure this is not an issue any more but will leave it open for a while in case somebody has another opinion.
            Hide
            Marina Glancy added a comment -

            Closing as it does not seem to affect supported versions

            Show
            Marina Glancy added a comment - Closing as it does not seem to affect supported versions

              People

              • Votes:
                10 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: