Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67449

Grunt amd:dist very slow

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Build all from root directory

      1. Change directory into your moodle root
      2. Run grunt:

        grunt
        

      3. Confirm there were no files changes:

        git status
        

      Watch from root directory

      1. Change directory into your moodle root
      2. Run grunt watch:

        grunt watch
        

      3. Open a second terminal
      4. Touch some files. For each file confirm that the file builds.

        touch mod/forum/amd/src/discussion.js
        touch mod/forum/amd/src/local/layouts.js
        touch mod/forum/amd/src/local/layout/fullscreen.js
        touch mod/assign/yui/src/history/js/history.js
        touch mod/forum/styles.css
        touch theme/boost/scss/moodle/*.scss
        touch theme/boost/scss/moodle/editor.scss
        

      5. Touch some more files and ensure that thy do not trigger a build:

        touch theme/boost/style/*.css
        

        1. Confirm that there were no changes:

          git status
          

      Build all from component directory

      1. Change directory into mod/assign:

        cd mod/assign
        

      2. Run grunt:

        grunt
        

      3. Confirm there were no files changes:

        git status
        

      Watch from component directory

      1. Change directory into mod/assign:

        cd mod/assign
        

      2. Run grunt watch:

        grunt watch
        

      3. Open a second terminal
      4. Touch some files. For each file confirm that the file builds.

        touch mod/assign/yui/src/history/js/history.js
        touch mod/assign/styles.css
        touch mod/assign/amd/src/grading_panel.js
        

      5. Touch some more files and ensure that thy do not trigger a build:

        touch mod/forum/amd/src/discussion.js
        touch mod/forum/amd/src/local/layouts.js
        touch mod/forum/amd/src/local/layout/fullscreen.js
        touch theme/boost/style/*.css
        touch theme/boost/scss/moodle/*.scss
        touch theme/boost/scss/moodle/editor.scss
        touch mod/forum/styles.css
        

        1. Confirm that there were no changes:

          git status
          

      Watch from a different component directory

      1. Change directory into mod/forum:

        cd mod/forum
        

      2. Run grunt watch:

        grunt watch
        

      3. Open a second terminal
      4. Touch some files. For each file confirm that the file builds.

        touch mod/forum/amd/src/discussion.js
        touch mod/forum/amd/src/local/layouts.js
        touch mod/forum/amd/src/local/layout/fullscreen.js
        touch mod/forum/styles.css
        

      5. Touch some more files and ensure that thy do not trigger a build:

        touch mod/assign/yui/src/history/js/history.js
        touch mod/assign/styles.css
        touch mod/assign/amd/src/grading_panel.js
        touch theme/boost/style/*.css
        touch theme/boost/scss/moodle/*.scss
        touch theme/boost/scss/moodle/editor.scss
        

        1. Confirm that there were no changes:

          git status
          

      Watch and build from a theme directory

      1. Change directory into theme/boost

        cd theme/boost
        

      2. Run grunt:

        grunt
        

      3. Confirm there were no files changes:

        git status
        

      4. Run grunt watch
      5. Open a second terminal
      6. Touch some files. For each file confirm that the file builds.

        touch theme/boost/scss/moodle/*.scss
        touch theme/boost/scss/moodle/editor.scss
        

      7. Touch some more files and ensure that thy do not trigger a build:

        touch theme/boost/style/*.css
        

        1. Confirm that there were no changes:

          git status
          

      Watch and build from an AMD directory

      1. Change directory into mod/forum/amd:

        cd mod/forum/amd
        

      2. Run grunt:

        grunt
        

        1. Confirm that only the startup, eslint:amd, and babel:dist tasks ran
      3. Run grunt watch
      4. Open a second terminal
      5. Touch some files. For each file confirm that the file builds.

        touch mod/forum/amd/src/discussion.js
        touch mod/forum/amd/src/local/layouts.js
        touch mod/forum/amd/src/local/layout/fullscreen.js
        touch mod/forum/styles.css
        

      6. Touch some more files and ensure that thy do not trigger a build:

        touch mod/assign/yui/src/history/js/history.js
        touch mod/assign/styles.css
        touch mod/assign/amd/src/grading_panel.js
        touch theme/boost/style/*.css
        touch theme/boost/scss/moodle/*.scss
        touch theme/boost/scss/moodle/editor.scss
        

        1. Confirm that there were no changes:

          git status
          

      Build a thirdpartylibs.xml file

      1. Run `grunt ignorefiles`
        1. Confirm that there were no changes:

          git status
          

      2. Open lib/thirdpartylibs.xml in your editor
      3. Go to the bottom of the file and add a new entry to a new directory, for example:

          <library>
            <location>foo</location>
            <name>foo</name>
            <license>GPL-3.0</license>
            <version>1.00</version>
          </library>
        

      4. Run `grunt ignorefiles`
        1. Confirm that the .stylelintignore and .eslintignore files were changed:

          git status
          

        2. Confirm that the new foo directory was added to both
      Show
      Build all from root directory Change directory into your moodle root Run grunt : grunt Confirm there were no files changes: git status Watch from root directory Change directory into your moodle root Run grunt watch : grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch mod/assign/yui/src/history/js/history.js touch mod/forum/styles.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Touch some more files and ensure that thy do not trigger a build: touch theme/boost/style/*.css Confirm that there were no changes : git status Build all from component directory Change directory into mod/assign: cd mod/assign Run grunt : grunt Confirm there were no files changes: git status Watch from component directory Change directory into mod/assign: cd mod/assign Run grunt watch : grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/assign/yui/src/history/js/history.js touch mod/assign/styles.css touch mod/assign/amd/src/grading_panel.js Touch some more files and ensure that thy do not trigger a build: touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch theme/boost/style/*.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss touch mod/forum/styles.css Confirm that there were no changes : git status Watch from a different component directory Change directory into mod/forum: cd mod/forum Run grunt watch : grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch mod/forum/styles.css Touch some more files and ensure that thy do not trigger a build: touch mod/assign/yui/src/history/js/history.js touch mod/assign/styles.css touch mod/assign/amd/src/grading_panel.js touch theme/boost/style/*.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Confirm that there were no changes : git status Watch and build from a theme directory Change directory into theme/boost cd theme/boost Run grunt : grunt Confirm there were no files changes: git status Run grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Touch some more files and ensure that thy do not trigger a build: touch theme/boost/style/*.css Confirm that there were no changes : git status Watch and build from an AMD directory Change directory into mod/forum/amd: cd mod/forum/amd Run grunt : grunt Confirm that only the startup, eslint:amd, and babel:dist tasks ran Run grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch mod/forum/styles.css Touch some more files and ensure that thy do not trigger a build: touch mod/assign/yui/src/history/js/history.js touch mod/assign/styles.css touch mod/assign/amd/src/grading_panel.js touch theme/boost/style/*.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Confirm that there were no changes : git status Build a thirdpartylibs.xml file Run `grunt ignorefiles` Confirm that there were no changes : git status Open lib/thirdpartylibs.xml in your editor Go to the bottom of the file and add a new entry to a new directory, for example: <library> <location>foo</location> <name>foo</name> <license>GPL-3.0</license> <version>1.00</version> </library> Run `grunt ignorefiles` Confirm that the .stylelintignore and .eslintignore files were changed : git status Confirm that the new foo directory was added to both
    • Affected Branches:
      MOODLE_38_STABLE
    • Fixed Branches:
      MOODLE_39_STABLE
    • Pull Master Branch:
      MDL-67449-master

      Description

      When we run `grunt amd` it is very slow. It should not be. There's a bug:
      Every time we call the eslint or dist commands we are generating a list of files to work on. That list uses globstar (*) patterns like /amd/src//.j.

      Ultimately these are slow and non-performant. We already have a list of all subsystems, and plugintypes, and we can easily turn that into a list of components (subsystems + plugins + subplugins). From there we can create a smaller list whcih, while still using the globstar format, has a much smaller surface area.

      Whilst I discovered this with babel:dist, I also discovered that it applies to several other places, and there was some confusing behaviours too:

      • eslint:amd suffered the same/similar issue
      • eslint:yui suffered the same/similar issue
      • stylelint:css suffered the same/similar issue
      • stylelint:scss suffered the same/similar issue
      • in a directory [path/to/component] the grunt command had different behaviours:
        • grunt builds the whole of Moodle
        • grunt watch watches just the current directory
      • in a directory [path/to/component/amd] the grunt command had different behaviours:
        • grunt builds only the current directory
        • grunt watch claims to watch just the current directory but does nothing

      Therefore I've taken the liberty of fixing these issues too. They all had a similar root cause and similar fix.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dobedobedoh Andrew Lyons
              Reporter:
              dobedobedoh Andrew Lyons
              Peer reviewer:
              Peter Dias Peter Dias
              Integrator:
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Tester:
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Participants:
              Component watchers:
              Andrew Lyons, Dongsheng Cai, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                15/Jun/20

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 7 hours, 27 minutes
                  7h 27m