Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-73734

Improve the way long running web pages stream output to the browser aka BigPipe

XMLWordPrintable

    • MOODLE_403_STABLE
    • MDL-73734-streaming-output-api
    • Hide

      1) Enable $CFG->perfdebug 

       

      2) Visit /admin/tool/httpsreplace/tool.php

      2.2) Confirm the footer is rendered while the progress bar is running 

      2.3) Confirm you see a placeholder in the footer for the performance debugging instead of the tables

      2.4) Confirm the additional content and 'Confirm form' are shown at the end of the script execution

      2.5) Confirm that at the end of the script execution the placeholder text is removed

      2.6) Confirm that at the end of the script execution the performance table appears in the footer

       

      3) Install this plugin to make testing easier

      https://github.com/catalyst/moodle-tool_testtasks

      3.1) Visit /admin/tool/task/schedule_task.php

      3.2) Click 'Run now' for the tool_testtasks/task/slow_task, then 'Run now'

      3.3) Confirm the footer is rendered while task is running

      3.4) Confirm the addition 'Run again' form is shown at the end when the task is complete

       

      4) Visit /admin/tool/customlang/index.php

      4.1) Select a language eg 'en', then click 'Open language pack for editing'

      4.2) Confirm the footer is rendered while the progress bar is running

       

      5) Visit https://master.localhost/admin/tool/generator/maketestcourse.php

      5.1) Choose small (large is fine but takes longer) and click Create course

      5.2) Confirm that you see the full page rendered including the footer while the generation happens

      5.3) Create a course using the CLI to check we have not broken display in cli scripts:

       php admin/tool/generator/cli/maketestcourse.php --shortname=SIZE_S --size=S

      • Creating course SIZE_S
      • Creating assignments (10): .done (1.2s)
      • Creating pages (50): .done (0.9s)
      • Creating small files (64): done (0.3s)
      • Creating big files (2): done (0.2s)
      • Checking user accounts (100)
      • Enrolling users into course (100): ..done (1.4s)
      • Creating forum (20 posts): done (0.2s)
      • Course completed (6.5s)

      Generated course: https://master.localhost/course/view.php?id=58

      5.4) Confirm you get normal output with no random html

       

      6) Visit /course/management.php

      6.1) Find the new course you just created

      6.2) Click the delete icon then Delete

      6.3) Confirm the next page is fully rendered while the delete is in progress

       

      7) Download and run this page:

      MDL-73734-stream-output.php

      7.1) Confirm that the 6 'waiting...' spinners are gradually replaced with 'preparing...' and then replaced again with a green DONE

      7.2) Confirm that the stream then incrementally appends strings like 'Printing 1 into area number 1' into each of the 6 areas in turn

      7.3) Edit this file and comment out the NO_OUTPUT_BUFFERING line

      7.4) Reload the page and confirm you get a coding_exception stating that buffering must be off

       

      8) Visit any normal page with buffering on, eg /my/courses.php

      8.1) Confirm that the performance footer renders instantly without the placeholder from 2.3

       

      Show
      1) Enable $CFG->perfdebug    2) Visit /admin/tool/httpsreplace/tool.php 2.2) Confirm the footer is rendered while the progress bar is running  2.3) Confirm you see a placeholder in the footer for the performance debugging instead of the tables 2.4) Confirm the additional content and 'Confirm form' are shown at the end of the script execution 2.5) Confirm that at the end of the script execution the placeholder text is removed 2.6) Confirm that at the end of the script execution the performance table appears in the footer   3) Install this plugin to make testing easier https://github.com/catalyst/moodle-tool_testtasks 3.1) Visit /admin/tool/task/schedule_task.php 3.2) Click 'Run now' for the tool_testtasks/task/slow_task, then 'Run now' 3.3) Confirm the footer is rendered while task is running 3.4) Confirm the addition 'Run again' form is shown at the end when the task is complete   4) Visit /admin/tool/customlang/index.php 4.1) Select a language eg 'en', then click 'Open language pack for editing' 4.2) Confirm the footer is rendered while the progress bar is running   5) Visit https://master.localhost/admin/tool/generator/maketestcourse.php 5.1) Choose small (large is fine but takes longer) and click Create course 5.2) Confirm  that you see the full page rendered including the footer while the generation happens 5.3) Create a course using the CLI to check we have not broken display in cli scripts:  php admin/tool/generator/cli/maketestcourse.php --shortname=SIZE_S --size=S Creating course SIZE_S Creating assignments (10): .done (1.2s) Creating pages (50): .done (0.9s) Creating small files (64): done (0.3s) Creating big files (2): done (0.2s) Checking user accounts (100) Enrolling users into course (100): ..done (1.4s) Creating forum (20 posts): done (0.2s) Course completed (6.5s) Generated course: https://master.localhost/course/view.php?id=58 5.4) Confirm you get normal output with no random html   6) Visit /course/management.php 6.1) Find the new course you just created 6.2) Click the delete icon then Delete 6.3) Confirm the next page is fully rendered while the delete is in progress   7) Download and run this page: MDL-73734 -stream-output.php 7.1) Confirm that the 6 'waiting...' spinners are gradually replaced with 'preparing...' and then replaced again with a green DONE 7.2) Confirm that the stream then incrementally appends strings like 'Printing 1 into area number 1' into each of the 6 areas in turn 7.3) Edit this file and comment out the NO_OUTPUT_BUFFERING line 7.4) Reload the page and confirm you get a coding_exception stating that buffering must be off   8) Visit any normal page with buffering on, eg /my/courses.php 8.1) Confirm that the performance footer renders instantly without the placeholder from 2.3  

      There are a couple places such as running upgrades and running a task which slowly output a long running stream of stuff, but the way this works means the page layout is broken in the interim: the footer isn't sent yet and the header can be weirdly aligned. 

      It would be much better if there was a new page rendering method very similar to the way the progress bar sends incremental updates so that the page can be fully rendered and then one or possibly many parts of the page can be updated in any order by streaming html chunk updates from the backend.

      This would be a relatively small api but is effectively implementing the 'BigPipe' pattern which could have a lot of uses and performance improvements in various places:

      https://www.drupal.org/docs/8/core/modules/big-pipe/overview

      https://engineering.fb.com/2010/06/04/web/bigpipe-pipelining-web-pages-for-high-performance/

      The $PAGE method could look vaguely like these:

      https://github.com/williankeller/bigpipe-pipeline#methods-to-use-it

      Places which would benefit from this:

      • backup and restore when doing the synchronously
      • course generator tool
      • cron 'run now' pages
      • tool_httpsreplace
      • dashboard, site home, and course pages with progressively rendered blocks without resorting to them all being another ajax request

       

            brendanheywood Brendan Heywood
            brendanheywood Brendan Heywood
            Peter Burnett Peter Burnett
            Andrew Lyons Andrew Lyons
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 day, 59 minutes
                1d 59m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.