[MDL-67449] Grunt amd:dist very slow - Moodle Tracker

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Minor
Resolution: Fixed
Affects Version/s: 3.8.1
Fix Version/s: 3.9
Component/s: JavaScript
Labels:
- ci
- hqteam_international

Affected Branches:
MOODLE_38_STABLE
Fixed Branches:
MOODLE_39_STABLE
Pull from Repository:
git://github.com/andrewnicols/moodle.git
Pull Master Branch:
~~MDL-67449~~-master
Pull Master Diff URL:
https://github.com/andrewnicols/moodle/compare/c55402486b...MDL-67449-master

Testing Instructions:

Hide

Build all from root directory

Change directory into your moodle root
Run grunt:
grunt
Confirm there were no files changes:
git status

Watch from root directory

Change directory into your moodle root
Run grunt watch:
grunt watch
Open a second terminal

Touch some files. For each file confirm that the file builds.

touch mod/forum/amd/src/discussion.js

touch mod/forum/amd/src/local/layouts.js

touch mod/forum/amd/src/local/layout/fullscreen.js

touch mod/assign/yui/src/history/js/history.js

touch mod/forum/styles.css

touch theme/boost/scss/moodle/*.scss

touch theme/boost/scss/moodle/editor.scss

Touch some more files and ensure that thy do not trigger a build:
touch theme/boost/style/*.css
1. Confirm that there were no changes:
  git status

Build all from component directory

Change directory into mod/assign:
cd mod/assign
Run grunt:
grunt
Confirm there were no files changes:
git status

Watch from component directory

Change directory into mod/assign:
cd mod/assign
Run grunt watch:
grunt watch
Open a second terminal

Touch some files. For each file confirm that the file builds.

touch mod/assign/yui/src/history/js/history.js

touch mod/assign/styles.css

touch mod/assign/amd/src/grading_panel.js

Touch some more files and ensure that thy do not trigger a build:

touch mod/forum/amd/src/discussion.js

touch mod/forum/amd/src/local/layouts.js

touch mod/forum/amd/src/local/layout/fullscreen.js

touch theme/boost/style/*.css

touch theme/boost/scss/moodle/*.scss

touch theme/boost/scss/moodle/editor.scss

touch mod/forum/styles.css

Confirm that there were no changes:
git status

Watch from a different component directory

Change directory into mod/forum:
cd mod/forum
Run grunt watch:
grunt watch
Open a second terminal

Touch some files. For each file confirm that the file builds.

touch mod/forum/amd/src/discussion.js

touch mod/forum/amd/src/local/layouts.js

touch mod/forum/amd/src/local/layout/fullscreen.js

touch mod/forum/styles.css

Touch some more files and ensure that thy do not trigger a build:

touch mod/assign/yui/src/history/js/history.js

touch mod/assign/styles.css

touch mod/assign/amd/src/grading_panel.js

touch theme/boost/style/*.css

touch theme/boost/scss/moodle/*.scss

touch theme/boost/scss/moodle/editor.scss

Confirm that there were no changes:
git status

Watch and build from a theme directory

Change directory into theme/boost
cd theme/boost
Run grunt:
grunt
Confirm there were no files changes:
git status
Run grunt watch
Open a second terminal

Touch some files. For each file confirm that the file builds.

touch theme/boost/scss/moodle/*.scss

touch theme/boost/scss/moodle/editor.scss

Touch some more files and ensure that thy do not trigger a build:
touch theme/boost/style/*.css
1. Confirm that there were no changes:
  git status

Watch and build from an AMD directory

Change directory into mod/forum/amd:
cd mod/forum/amd
Run grunt:
grunt
1. Confirm that only the startup, eslint:amd, and babel:dist tasks ran
Run grunt watch
Open a second terminal

Touch some files. For each file confirm that the file builds.

touch mod/forum/amd/src/discussion.js

touch mod/forum/amd/src/local/layouts.js

touch mod/forum/amd/src/local/layout/fullscreen.js

touch mod/forum/styles.css

Touch some more files and ensure that thy do not trigger a build:

touch mod/assign/yui/src/history/js/history.js

touch mod/assign/styles.css

touch mod/assign/amd/src/grading_panel.js

touch theme/boost/style/*.css

touch theme/boost/scss/moodle/*.scss

touch theme/boost/scss/moodle/editor.scss

Confirm that there were no changes:
git status

Build a thirdpartylibs.xml file

Run `grunt ignorefiles`
1. Confirm that there were no changes:
  git status
Open lib/thirdpartylibs.xml in your editor

Go to the bottom of the file and add a new entry to a new directory, for example:

  <library>

    <location>foo</location>

    <name>foo</name>

    <license>GPL-3.0</license>

    <version>1.00</version>

  </library>

Run `grunt ignorefiles`
1. Confirm that the .stylelintignore and .eslintignore files were changed:
  git status
2. Confirm that the new foo directory was added to both

Show

Build all from root directory Change directory into your moodle root Run grunt : grunt Confirm there were no files changes: git status Watch from root directory Change directory into your moodle root Run grunt watch : grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch mod/assign/yui/src/history/js/history.js touch mod/forum/styles.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Touch some more files and ensure that thy do not trigger a build: touch theme/boost/style/*.css Confirm that there were no changes : git status Build all from component directory Change directory into mod/assign: cd mod/assign Run grunt : grunt Confirm there were no files changes: git status Watch from component directory Change directory into mod/assign: cd mod/assign Run grunt watch : grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/assign/yui/src/history/js/history.js touch mod/assign/styles.css touch mod/assign/amd/src/grading_panel.js Touch some more files and ensure that thy do not trigger a build: touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch theme/boost/style/*.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss touch mod/forum/styles.css Confirm that there were no changes : git status Watch from a different component directory Change directory into mod/forum: cd mod/forum Run grunt watch : grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch mod/forum/styles.css Touch some more files and ensure that thy do not trigger a build: touch mod/assign/yui/src/history/js/history.js touch mod/assign/styles.css touch mod/assign/amd/src/grading_panel.js touch theme/boost/style/*.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Confirm that there were no changes : git status Watch and build from a theme directory Change directory into theme/boost cd theme/boost Run grunt : grunt Confirm there were no files changes: git status Run grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Touch some more files and ensure that thy do not trigger a build: touch theme/boost/style/*.css Confirm that there were no changes : git status Watch and build from an AMD directory Change directory into mod/forum/amd: cd mod/forum/amd Run grunt : grunt Confirm that only the startup, eslint:amd, and babel:dist tasks ran Run grunt watch Open a second terminal Touch some files. For each file confirm that the file builds. touch mod/forum/amd/src/discussion.js touch mod/forum/amd/src/local/layouts.js touch mod/forum/amd/src/local/layout/fullscreen.js touch mod/forum/styles.css Touch some more files and ensure that thy do not trigger a build: touch mod/assign/yui/src/history/js/history.js touch mod/assign/styles.css touch mod/assign/amd/src/grading_panel.js touch theme/boost/style/*.css touch theme/boost/scss/moodle/*.scss touch theme/boost/scss/moodle/editor.scss Confirm that there were no changes : git status Build a thirdpartylibs.xml file Run `grunt ignorefiles` Confirm that there were no changes : git status Open lib/thirdpartylibs.xml in your editor Go to the bottom of the file and add a new entry to a new directory, for example: <library> <location>foo</location> <name>foo</name> <license>GPL-3.0</license> <version>1.00</version> </library> Run `grunt ignorefiles` Confirm that the .stylelintignore and .eslintignore files were changed : git status Confirm that the new foo directory was added to both

Description

When we run `grunt amd` it is very slow. It should not be. There's a bug:
Every time we call the eslint or dist commands we are generating a list of files to work on. That list uses globstar (*) patterns like /amd/src//.j.

Ultimately these are slow and non-performant. We already have a list of all subsystems, and plugintypes, and we can easily turn that into a list of components (subsystems + plugins + subplugins). From there we can create a smaller list whcih, while still using the globstar format, has a much smaller surface area.

Whilst I discovered this with babel:dist, I also discovered that it applies to several other places, and there was some confusing behaviours too:

eslint:amd suffered the same/similar issue
eslint:yui suffered the same/similar issue
stylelint:css suffered the same/similar issue
stylelint:scss suffered the same/similar issue
in a directory [path/to/component] the grunt command had different behaviours:
- grunt builds the whole of Moodle
- grunt watch watches just the current directory
in a directory [path/to/component/amd] the grunt command had different behaviours:
- grunt builds only the current directory
- grunt watch claims to watch just the current directory but does nothing

Therefore I've taken the liberty of fixing these issues too. They all had a similar root cause and similar fix.

Attachments

Issue Links

caused a regression

MDL-67712 grunt does not work plugin types that are inside another component

Closed

MDL-67878 grunt does not run on Windows

Closed

MDL-67949 Windows issues with generating ignoresfiles, and with watching in themes

Closed

Activity

People

Assignee:: Andrew Lyons

Reporter:: Andrew Lyons

Peer reviewer:: Peter Dias

Integrator:: Eloy Lafuente (stronk7)

Tester:: Eloy Lafuente (stronk7)

Participants:: Andrew Lyons, CiBoT, Eloy Lafuente (stronk7), Peter Dias

Component watchers:: David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 04/Dec/19 5:28 PM

Updated:: 13/Feb/20 11:50 AM

Resolved:: 09/Jan/20 11:31 PM

Fix Release Date:: 15/Jun/20

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

7h 27m