Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-79285

Enable XHProf runs to be reduced for easier consume (mustache problem)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • 4.1.6, 4.2.3
    • 4.1.5, 4.2.2, 4.3
    • Libraries
    • MOODLE_401_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE
    • MOODLE_401_STABLE, MOODLE_402_STABLE
    • MDL-79285_401
    • MDL-79285_402
    • Hide

      Copied from MDL-76397

      Important note

      While looking to this issue I detected that there are some problems with this extension and PHP 8.0 and up (see https://github.com/tideways/php-xhprof-extension/issues/107). To workaround that, you can:

      • Use the PECL xhprof extension instead of tideways one (not tested here!)
      • Alternatively, use any of our php docker images that come with PECL xhprof installed (via moodle-docker or similar solution).

      Requirements

      • To have some of the 2 xhprof extensions (tideways or PECL) enabled.
      • To have "dot" (that comes with the Graphviz package) installed.
      • Add the path to "dot" in the config.php

        $CFG->pathtodot = '/opt/local/bin/dot';

      Install the tideways_xhprof PHP extension

      Install the tideways extension for PHP7, see https://github.com/tideways/php-profiler-extension for Installation instructions. Or if you're on Ubuntu/LinuxMint, follow the instructions below

      Ubuntu/LinuxMint install instructions

      1. Update your system

        sudo apt update && sudo apt upgrade -y

      2. Download the precompiled tideways_xhprof extension installer for Debian-based systems:

        wget https://github.com/tideways/php-xhprof-extension/releases/download/v5.0.4/tideways-xhprof_5.0.4_amd64.deb

      3. Install the package

        sudo dpkg -i tideways-xhprof_5.0.4_amd64.deb
        

      4. Check your PHP version

        php -v
        

      5. Check the contents of /usr/lib/tideways_xhprof

        ls /usr/lib/tideways_xhprof

      6. Make sure that there is a .so file that matches your current PHP version. For example, if you're on PHP 7.4, then confirm that tideways_xhprof-7.4.so is available in that folder.
      7. Using your favourite editor, create a "tideways_xhprof.ini" under /etc/php/[YOUR_PHPVERSION]/mods-available. e.g. if you're on PHP 7.4 on LinuxMint with xed as the editor:

        sudo xed /etc/php/7.4/mods-available/tideways_xhprof.ini
        

      8. Paste the following line in tideways_xhprof.ini pointing to the path of the tideways_xhprof extension file matching your PHP version.

        extension=/usr/lib/tideways_xhprof/tideways_xhprof-7.4.so
        

      9. Enable the tideways_xhprof extension

        sudo phpenmod tideways_xhprof
        

      10. Install Graphviz. (This is required for generating the callgraph of the profiling run)

        sudo apt install graphviz
        

      11. Restart your webserver

        sudo service apache2 restart

      12. Check your phpinfo() page. Search for tideways_xhprof and confirm that it is present.

      Mac install instructions

      1. Start the apple font if you encounter error regarding apple font

        sudo launchctl load -w /System/Library/LaunchAgents/com.apple.fontd.useragent.plist

      2. Restart your webserver

        brew services restart httpd 

      3. Make sure xhprof is enabled by going to "admin/phpinfo.php"

      Enabling xhprof

      1. Make sure "Developer mode" is turned on.
      2. Go to "Site administration / Development / Profiling"
      3. Tick "Enable profiling".
      4. Under "Profile these", enter "/index.php"
      5. Save changes

      Test export & graphs performance

      1. Go to the "Site home".
      2. Scroll down to the bottom of the page. Confirm that there is a link named "This script has been profiled"
      3. Click the "This script has been profiled" link.
      4. Confirm that it loads okay.
      5. Click "Export this profiling run"
      6. Confirm that a .mpr file archive is downloaded containing XML files is generated (rename it to .zip and uncompress if needed).
      7. Click "View profiling details".
      8. Confirm that a table of function calls is displayed.
      9. Click on "View Full Callgraph"
      10. Confirm that the call graph is generated.
      11. Confirm that it's quick (1-2 seconds at most, not minutes/hours/crashing anymore).

      Test the new "reducedata" parameter.

      1. Go to the "Site home".
      2. Scroll down to the bottom of the page. Confirm that there is a link named "This script has been profiled"
      3. Click the "This script has been profiled" link.
      4. Confirm that it loads okay.
      5. Click "View profiling details".
      6. Confirm that a table of function calls is displayed.
      7. Confirm that you can see a lot of "__Mustache_xxxxxx" entries in the table, with long hexadecimal suffixes. Some of them are shown together.
      8. On separate tabe Edit the URL of the page and add (without the quotes): "&reducedata=1"
      9. Press enter.
      10. Confirm that now there are a lot less of "__Mustache_xxxx" entries in the table. Easiest was to verify is to click 'display all' link in the bottom and sort table by function name
      11. Confirm that all the links in the page have that "&reducedata=1" added to them.
      12. Navigate following various links to entries in the table.
      13. Confirm that the "&reducedata=1" persists in the URL all the time.
      14. Click on the "View Top Level Run Report" link.
      15. Confirm that the "&reducedata=1" is part of it.
      16. Edit the URL of the page and add (without the quotes): "&reducedata=0"
      17. Press enter.
      18. Confirm that all the links in the page don't have any "reducedata" any more.
      19. Confirm that the table is again showing a lot of "__Mustache" entries, like it was in step 7.
      Show
      Copied from MDL-76397 Important note While looking to this issue I detected that there are some problems with this extension and PHP 8.0 and up (see https://github.com/tideways/php-xhprof-extension/issues/107 ). To workaround that, you can: Use the PECL xhprof extension instead of tideways one (not tested here!) Alternatively, use any of our php docker images that come with PECL xhprof installed (via moodle-docker or similar solution). Requirements To have some of the 2 xhprof extensions (tideways or PECL) enabled. To have "dot" (that comes with the Graphviz package) installed. Add the path to "dot" in the config.php $CFG->pathtodot = '/opt/local/bin/dot'; Install the tideways_xhprof PHP extension Install the tideways extension for PHP7, see https://github.com/tideways/php-profiler-extension for Installation instructions. Or if you're on Ubuntu/LinuxMint, follow the instructions below Ubuntu/LinuxMint install instructions Update your system sudo apt update && sudo apt upgrade -y Download the precompiled tideways_xhprof extension installer for Debian-based systems: wget https://github.com/tideways/php-xhprof-extension/releases/download/v5.0.4/tideways-xhprof_5.0.4_amd64.deb Install the package sudo dpkg -i tideways-xhprof_5.0.4_amd64.deb Check your PHP version php -v Check the contents of /usr/lib/tideways_xhprof ls /usr/lib/tideways_xhprof Make sure that there is a .so file that matches your current PHP version. For example, if you're on PHP 7.4, then confirm that tideways_xhprof-7.4.so is available in that folder. Using your favourite editor, create a " tideways_xhprof.ini " under /etc/php/ [YOUR_PHPVERSION] /mods-available. e.g. if you're on PHP 7.4 on LinuxMint with xed as the editor: sudo xed /etc/php/7.4/mods-available/tideways_xhprof.ini Paste the following line in tideways_xhprof.ini pointing to the path of the tideways_xhprof extension file matching your PHP version. extension=/usr/lib/tideways_xhprof/tideways_xhprof-7.4.so Enable the tideways_xhprof extension sudo phpenmod tideways_xhprof Install Graphviz. (This is required for generating the callgraph of the profiling run) sudo apt install graphviz Restart your webserver sudo service apache2 restart Check your phpinfo() page. Search for tideways_xhprof and confirm that it is present. Mac install instructions Start the apple font if you encounter error regarding apple font sudo launchctl load -w /System/Library/LaunchAgents/com.apple.fontd.useragent.plist Restart your webserver brew services restart httpd Make sure xhprof is enabled by going to "admin/phpinfo.php" Enabling xhprof Make sure " Developer mode " is turned on. Go to " Site administration / Development / Profiling " Tick " Enable profiling ". Under " Profile these ", enter " /index.php " Save changes Test export & graphs performance Go to the " Site home ". Scroll down to the bottom of the page. Confirm that there is a link named " This script has been profiled " Click the " This script has been profiled " link. Confirm that it loads okay. Click " Export this profiling run " Confirm that a .mpr file archive is downloaded containing XML files is generated (rename it to .zip and uncompress if needed). Click " View profiling details ". Confirm that a table of function calls is displayed. Click on " View Full Callgraph " Confirm that the call graph is generated. Confirm that it's quick (1-2 seconds at most, not minutes/hours/crashing anymore). Test the new "reducedata" parameter. Go to the " Site home ". Scroll down to the bottom of the page. Confirm that there is a link named " This script has been profiled " Click the " This script has been profiled " link. Confirm that it loads okay. Click " View profiling details ". Confirm that a table of function calls is displayed. Confirm that you can see a lot of "__Mustache_xxxxxx" entries in the table, with long hexadecimal suffixes. Some of them are shown together. On separate tabe Edit the URL of the page and add (without the quotes): "&reducedata=1" Press enter. Confirm that now there are a lot less of "__Mustache_xxxx" entries in the table. Easiest was to verify is to click 'display all' link in the bottom and sort table by function name Confirm that all the links in the page have that "&reducedata=1" added to them. Navigate following various links to entries in the table. Confirm that the "&reducedata=1" persists in the URL all the time. Click on the "View Top Level Run Report" link. Confirm that the "&reducedata=1" is part of it. Edit the URL of the page and add (without the quotes): "&reducedata=0" Press enter. Confirm that all the links in the page don't have any "reducedata" any more. Confirm that the table is again showing a lot of "__Mustache" entries, like it was in step 7.

      Detected @ MDL-76397, and also when trying to use XHProf here and there...

      The complexity of our requests has grown enormously, with way more code involved on each (aka, super big stack traces, basically).

      Also, more specifically, the use of Mustache templates, with all their partials, nested... have made XHProf profiling runs to near double in size.

      While the visualisation in table mode of those profiling runs in "table mode" is still acceptable (no matter the enormous noise that the Mustache templates add), right now it's impossible to render a "callgraph" with the critical paths and other useful information.

      So, this issue is about to:

      1. Introduce some way to "reduce" the information available in XHProf profiling runs.
      2. Apply such a reduction specifically to the Mustache templates calls.
      3. By default, use that "reduced" information when displaying call-graphs.
      4. By default, use the "full" information when displaying in table mode.
      5. Optional: Be able to pick between full and reduced in table mode.

      This has some requirements, namely:

      1. The backed (database) always will store the full runs.
      2. The "reduction" will happen on demand when presenting the information (table or graph).
      3. The "reduction" needs to be consistent, not leading to "orphaned" data that can lead to misleading information.
      4. The need of hacking the xhprof library needs to be the smallest possible. Ideally zero (although I bet we'll need to add some line here and there).
      5. Covered with tests as much as possible.

      And the goals are:

      1. General: To have a working system in place to "reduce" profiling runs information.
      2. Specific: To be able to get the call-graphs back to work, right now it's virtually impossible because of the complexity added by the Mustache templates.

      Ciao

        1. 01_xhprof_reduce_initial.jpg
          01_xhprof_reduce_initial.jpg
          41 kB
        2. 02_xhprof_reduce_remove.jpg
          02_xhprof_reduce_remove.jpg
          38 kB
        3. 03_xhprof_reduce_orphaned1.jpg
          03_xhprof_reduce_orphaned1.jpg
          34 kB
        4. 04_xhprof_reduce_orphaned2.jpg
          04_xhprof_reduce_orphaned2.jpg
          33 kB
        5. 05_xhprof_reduce_orphaned3.jpg
          05_xhprof_reduce_orphaned3.jpg
          29 kB
        6. MDL-79285 - 401 - Export test - export file.png
          MDL-79285 - 401 - Export test - export file.png
          145 kB
        7. MDL-79285 - 401 - Export test - View Full Callgraph.png
          MDL-79285 - 401 - Export test - View Full Callgraph.png
          273 kB
        8. MDL-79285 - 401 - Export test - view profiling details.png
          MDL-79285 - 401 - Export test - view profiling details.png
          345 kB
        9. MDL-79285 - 401 - Reduce data 0.png
          MDL-79285 - 401 - Reduce data 0.png
          265 kB
        10. MDL-79285 - 401 - Reduce data 1.png
          MDL-79285 - 401 - Reduce data 1.png
          182 kB
        11. MDL-79285 - 401 - Reduce data 1 persists URL.png
          MDL-79285 - 401 - Reduce data 1 persists URL.png
          154 kB
        12. MDL-79285 - 402 - Export test - export file.png
          MDL-79285 - 402 - Export test - export file.png
          131 kB
        13. MDL-79285 - 402 - Export test - View Full Callgraph.png
          MDL-79285 - 402 - Export test - View Full Callgraph.png
          237 kB
        14. MDL-79285 - 402 - Export test - view profiling details.png
          MDL-79285 - 402 - Export test - view profiling details.png
          193 kB
        15. MDL-79285 - 402 - Reduce data 0.png
          MDL-79285 - 402 - Reduce data 0.png
          202 kB
        16. MDL-79285 - 402 - Reduce data 1.png
          MDL-79285 - 402 - Reduce data 1.png
          186 kB
        17. MDL-79285 - 402 - Reduce data 1 persists URL.png
          MDL-79285 - 402 - Reduce data 1 persists URL.png
          94 kB
        18. MDL-79285 - master - Export test - export file.png
          MDL-79285 - master - Export test - export file.png
          146 kB
        19. MDL-79285 -  master - Export test - View Full Callgraph.png
          MDL-79285 - master - Export test - View Full Callgraph.png
          223 kB
        20. MDL-79285 -  master - Export test - view profiling details.png
          MDL-79285 - master - Export test - view profiling details.png
          313 kB
        21. MDL-79285 -  master - Reduce data 0.png
          MDL-79285 - master - Reduce data 0.png
          366 kB
        22. MDL-79285 -  master - Reduce data 1.png
          MDL-79285 - master - Reduce data 1.png
          185 kB
        23. MDL-79285 -  master - Reduce data 1 persists URL.png
          MDL-79285 - master - Reduce data 1 persists URL.png
          37 kB

            stronk7 Eloy Lafuente (stronk7)
            stronk7 Eloy Lafuente (stronk7)
            Andrew Lyons Andrew Lyons
            Ilya Tregubov Ilya Tregubov
            Carlos Escobedo Carlos Escobedo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 days, 6 hours, 17 minutes
                3d 6h 17m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.