Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65693

Support http HEAD requests for pluginfile

XMLWordPrintable

    • MOODLE_39_STABLE
    • MDL-65693-http-head
    • Hide

      This is easiest to compare by getting a baseline for before and after mostly to check that the result is the same. Or you can swap between two branches.

      It can also help make things more obvious if you add a sleep(2) immediately after where this patch is to make it really obvious when testing when a real HEAD is being honored or not, and this also simulates the performance gains if you were working with a massive file and / or a slow or remote file system.

      1) You will need a suite of files of different file types, these can be uploaded however but mod_resource is the easiest. We need to test files which do and do not use compression, and ideally at various sizes (eg a very large video file). All of this can be subtly different depending on your apache / nginx / cdn / proxy stack settings. The minimum should be one binary file which is already compressed such as a png or a video file which is not re-compressed by your web server, and for completely another file which is not natively compressed such as html, which will be compressed by your web server.
      One method of creating these 2 files:

      • Take a screenshot and save as a png.
      • Visit example.com and save the page as an html file.
      • Upload both files as 'File' resources in Moodle, setting the 'Display' (under 'Appearance') to 'Force download'.

      2) For each file load it's url such as:

      "https://example.com/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png"

      To do this:

      • Open dev tools and focus the network tab
      • Click the name of the resource to load it.
      • In dev tools, notice a request for the name of the file. Mouse over this and you'll see it's a request to pluginfile.php. This is the entry we want.

      3) In the browser dev tools copy this url as a curl command. In chrome this is Dev tools > Network tab, right click on the url, Copy > Copy as cURL (example below has been edited for brevity)

      curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate'  -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed

      4) Append a '-i' (which shows the response headers) to the start of the command, then run it and note the Content-Length. Do this for the png file only.

      curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed 
      

      Content-Length: 351584
      Note down this content length.

      Note: If you don't see a content length,there are some valid reasons why. If you don't get one then we don't need to keep continue testing with this particular file. The most common would be your webserver has compressed the result, and / or the response size was large enough it may have swapped to Transfer-Encoding: chunked

      5) Now repeat that command again for the png file with both '-i' and '-I' which does a head request. 

      curl -iI 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed 

      Verify you end up with the same Content-Length as before. 

      6) Now for completeness repeat this for a non binary file type which we know is going to be compressed by your webserver (we'll use the html file we uploaded earlier). Then download it, note the v and > /dev/null to show the headers this time.

      curl -v 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/null

      Note we get a content length AND a Content-Encoding of gzip:

      Content-Encoding: gzip
      Content-Length: 23355

      7) Now run that with -I for head:

       

      curl -v -I 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/null
      

      The we get a different content length, but it's a different encoding so this is ok ( and this should be consistent without with patch applied too)

      Content-Length: 176096

       

      Show
      This is easiest to compare by getting a baseline for before and after mostly to check that the result is the same. Or you can swap between two branches. It can also help make things more obvious if you add a sleep(2) immediately after where this patch is to make it really obvious when testing when a real HEAD is being honored or not, and this also simulates the performance gains if you were working with a massive file and / or a slow or remote file system. 1) You will need a suite of files of different file types, these can be uploaded however but mod_resource is the easiest. We need to test files which do and do not use compression, and ideally at various sizes (eg a very large video file). All of this can be subtly different depending on your apache / nginx / cdn / proxy stack settings. The minimum should be one binary file which is already compressed such as a png or a video file which is not re-compressed by your web server, and for completely another file which is not natively compressed such as html, which will be compressed by your web server. One method of creating these 2 files: Take a screenshot and save as a png. Visit example.com and save the page as an html file. Upload both files as 'File' resources in Moodle, setting the 'Display' (under 'Appearance') to 'Force download'. 2) For each file load it's url such as: "https://example.com/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png" To do this: Open dev tools and focus the network tab Click the name of the resource to load it. In dev tools, notice a request for the name of the file. Mouse over this and you'll see it's a request to pluginfile.php. This is the entry we want. 3) In the browser dev tools copy this url as a curl command. In chrome this is Dev tools > Network tab, right click on the url, Copy > Copy as cURL (example below has been edited for brevity) curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed 4) Append a '-i' (which shows the response headers) to the start of the command, then run it and note the Content-Length. Do this for the png file only. curl -i 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed  Content-Length: 351584 Note down this content length. Note: If you don't see a content length,there are some valid reasons why. If you don't get one then we don't need to keep continue testing with this particular file. The most common would be your webserver has compressed the result, and / or the response size was large enough it may have swapped to Transfer-Encoding: chunked 5) Now repeat that command again for the png file with both '-i' and '-I' which does a head request.  curl -iI 'http://moodle.local/pluginfile.php/82/mod_label/intro/Workspace%201_999%28004%29.png' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed  Verify you end up with the same Content-Length as before.  6) Now for completeness repeat this for a non binary file type which we know is going to be compressed by your webserver (we'll use the html file we uploaded earlier). Then download it, note the v and > /dev/null to show the headers this time. curl -v 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/ null Note we get a content length AND a Content-Encoding of gzip: Content-Encoding: gzip Content-Length: 23355 7) Now run that with -I for head:   curl -v -I 'http://moodle.local/pluginfile.php/11969/mod_resource/content/0/UNKN_%20Course_%20comp%20100%20test%20course.html?embed=1' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: MDL_SSP_SessID=74732f14dae627ba688088b2a6f7da2a; MoodleSession=1cto3rht7q27dd7dijt68m8fsb; MOODLEID1_=%2596K%2589%25A37mY%251F%25E1%25F8-' --compressed > /dev/ null The we get a different content length, but it's a different encoding so this is ok ( and this should be consistent without with patch applied too) Content-Length: 176096  

      Often when a browser converts from a full download to an incremental download using range requests (ie for streaming audio / video) it will first request a HEAD to get the content length.

      Moodle doesn't handle this itself in any way so it will just serve the whole file normally and then apache or nginx will throw away the entire response and convert it to a head request just to grab the content length.

      If you use xsend file then it's probably smart enough to figure this out, but when you use the new alternate filesystem api it will first stream the file from elsewhere (eg objectfs -> aws s3) and then throw it away. This could be a massive video file. But we know the content length from the mdl_files, we should be able to blindly send this without touching the real file.

            brendanheywood Brendan Heywood
            brendanheywood Brendan Heywood
            Matt Porritt Matt Porritt
            Jake Dallimore Jake Dallimore
            Gladys Basiana Gladys Basiana
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours
                3h

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.