Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65693

Support http HEAD requests for pluginfile

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      This is easiest to compare by getting a baseline for before and after mostly to check that the result is the same. Or you can swap between two branches.

      It can also help make things more obvious if you add a sleep(2) immediately after where this patch is to make it really obvious when testing when a real HEAD is being honored or not, and this also simulates the performance gains if you were working with a massive file and / or a slow or remote file system.

      1) You will need a suite of files of different file types, these can be uploaded however but mod_resource is the easiest. We need to test files which do and do not use compression, and ideally at various sizes (eg a very large video file). All of this can be subtly different depending on your apache / nginx / cdn / proxy stack settings. The minimum should be one binary file which is already compressed such as a png or a video file which is not re-compressed by your web server, and for completely another file which is not natively compressed such as html, which will be compressed by your web server.

      2) For each file load it's url such as:

      https://example.com/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png

      3) In the browser dev tools copy this url as a curl command. In chrome this is Dev tools > Network tab, right click on the url, Copy > Copy as cURL (example below has been edited for brevity)

      curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate'  -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed

      4) Append a '-i' (which shows the response headers) to the start of the command, then run it and note the Content-Length: 

      curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed 
      

      Content-Length: 351584

      There are some valid reasons why you don't get a Content-Length, if you don't get one then we don't need to keep continue testing with this particular file. The most common would be your webserver has compressed the result, and / or the response size was large enough it may have swapped to Transfer-Encoding: chunked

      5) Now repeat that command again with both '-i' and '-I' which does a head request. You should end up with the same Content-Length as before.

      curl -iI 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed 

       

      6) Now for completeness repeat this for a non binary file type which we know is going to be compressed by your webserver, eg upload a text, html or xml file. Then download it, note the v and > /dev/null to show the headers this time.

      curl -v 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/null

      Note we get a content length AND a Content-Encoding of gzip:

      Content-Encoding: gzip
      Content-Length: 23355

      7) If we run that with -I for head:

       

      curl -v -I 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/null
      

      The we get a different content length, but it's a different encoding so this is ok ( and this should be consistent without with patch applied too)

      Content-Length: 176096

       

      Show
      This is easiest to compare by getting a baseline for before and after mostly to check that the result is the same. Or you can swap between two branches. It can also help make things more obvious if you add a sleep(2) immediately after where this patch is to make it really obvious when testing when a real HEAD is being honored or not, and this also simulates the performance gains if you were working with a massive file and / or a slow or remote file system. 1) You will need a suite of files of different file types, these can be uploaded however but mod_resource is the easiest. We need to test files which do and do not use compression, and ideally at various sizes (eg a very large video file). All of this can be subtly different depending on your apache / nginx / cdn / proxy stack settings. The minimum should be one binary file which is already compressed such as a png or a video file which is not re-compressed by your web server, and for completely another file which is not natively compressed such as html, which will be compressed by your web server. 2) For each file load it's url such as: https://example.com/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png 3) In the browser dev tools copy this url as a curl command. In chrome this is Dev tools > Network tab, right click on the url, Copy > Copy as cURL (example below has been edited for brevity) curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed 4) Append a '-i' (which shows the response headers) to the start of the command, then run it and note the Content-Length:  curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed  Content-Length: 351584 There are some valid reasons why you don't get a Content-Length, if you don't get one then we don't need to keep continue testing with this particular file. The most common would be your webserver has compressed the result, and / or the response size was large enough it may have swapped to Transfer-Encoding: chunked 5) Now repeat that command again with both '-i' and '-I' which does a head request. You should end up with the same Content-Length as before. curl -iI 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed    6) Now for completeness repeat this for a non binary file type which we know is going to be compressed by your webserver, eg upload a text, html or xml file. Then download it, note the v and > /dev/null to show the headers this time. curl -v 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/ null Note we get a content length AND a Content-Encoding of gzip: Content-Encoding: gzip Content-Length: 23355 7) If we run that with -I for head:   curl -v -I 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/ null The we get a different content length, but it's a different encoding so this is ok ( and this should be consistent without with patch applied too) Content-Length: 176096  
    • Pull Master Branch:
      MDL-65693-http-head

      Description

      Often when a browser converts from a full download to an incremental download using range requests (ie for streaming audio / video) it will first request a HEAD to get the content length.

      Moodle doesn't handle this itself in any way so it will just serve the whole file normally and then apache or nginx will throw away the entire response and convert it to a head request just to grab the content length.

      If you use xsend file then it's probably smart enough to figure this out, but when you use the new alternate filesystem api it will first stream the file from elsewhere (eg objectfs -> aws s3) and then throw it away. This could be a massive video file. But we know the content length from the mdl_files, we should be able to blindly send this without touching the real file.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brendanheywood Brendan Heywood
                Reporter:
                brendanheywood Brendan Heywood
                Peer reviewer:
                Matt Porritt
                Participants:
                Component watchers:
                Matteo Scaramuccia, Jake Dallimore, Jun Pataleta, Matteo Scaramuccia, Jake Dallimore, Jun Pataleta
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: