Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-68405

Regression: grunt gherkinlint not providing correct exit status anymore

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.7.5, 3.8.2, 3.9
    • Fix Version/s: 3.7.6, 3.8.3
    • Component/s: JavaScript
    • Labels:
    • Testing Instructions:
      Hide

      Setup

      1. Open your terminal and go to your Moodle root directory
      2. Run "npm install"

      Test all is good

      1. Run Gherkinlint

        grunt gherkinlint
        

        1. Confirm that no errors were reported
      2. Immediately check the exit code:

        echo $?
        

        1. Confirm that it was zero
      3. Change directory into mod/forum and run the check again:

        cd mod/forum
        grunt gherkinlint ; echo $?
        

        1. Confirm that there were no errors
        2. Confirm that the exit code was zero

      Master only:

      1. Change directory into mod/h5pactivity and run the check again:

        cd ../h5pactivity
        grunt gherkinlint ; echo $?
        

        1. Confirm that there were no errors
        2. Confirm that the exit code was zero

      Test an introduced failure

      1. Open mod/forum/tests/behat/add_forum.feature in your editor
      2. At the first empty line, add another empty line and save the file
      3. Ensure that you are in the root of the moodle installation
      4. Run Gherkinlint

        grunt gherkinlint
        

        1. Confirm that the test reported a failing line
      5. Immediately check the exit code:

        echo $?
        

        1. Confirm that it was non-zero (probably 3)
      6. Change directory into mod/forum and run the check again:

        cd mod/forum
        grunt gherkinlint ; echo $?
        

        1. Confirm that errors were found
        2. Confirm the non-zero exit code again
      7. Change directory into mod/chat and run the check again:

        cd ../chat
        grunt gherkinlint ; echo $?
        

        1. Confirm that there were no errors
        2. Confirm that the exit code was zero
      Show
      Setup Open your terminal and go to your Moodle root directory Run " npm install " Test all is good Run Gherkinlint grunt gherkinlint Confirm that no errors were reported Immediately check the exit code: echo $? Confirm that it was zero Change directory into mod/forum and run the check again: cd mod/forum grunt gherkinlint ; echo $? Confirm that there were no errors Confirm that the exit code was zero Master only: Change directory into mod/h5pactivity and run the check again: cd ../h5pactivity grunt gherkinlint ; echo $? Confirm that there were no errors Confirm that the exit code was zero Test an introduced failure Open mod/forum/tests/behat/add_forum.feature in your editor At the first empty line, add another empty line and save the file Ensure that you are in the root of the moodle installation Run Gherkinlint grunt gherkinlint Confirm that the test reported a failing line Immediately check the exit code: echo $? Confirm that it was non-zero (probably 3) Change directory into mod/forum and run the check again: cd mod/forum grunt gherkinlint ; echo $? Confirm that errors were found Confirm the non-zero exit code again Change directory into mod/chat and run the check again: cd ../chat grunt gherkinlint ; echo $? Confirm that there were no errors Confirm that the exit code was zero
    • Affected Branches:
      MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_37_STABLE, MOODLE_38_STABLE
    • Pull 3.7 Branch:
    • Pull 3.8 Branch:
    • Pull Master Branch:
      MDL-68405-master

      Description

      This was detected when working on MDLSITE-6103 because I was getting non-consistent results in some of the tests being added there.

      In that issue (see details) the problem was bisected and found that the change of behavior came with MDL-67953.

      Basically, before that patch... whenever the gherkinlint task finds errors, they are reported, then some warning text is shown and the execution ends with error and exit code (not 0).

      After the patch, the errors are reported exactly the same, but the warning text is not shown and the execution ends without error and with exit code (= 0).

      To reproduce:

      • edit any feature file, change some indentation (to force a gherkin-lint error).
      • run grunt gherkinklint; echo $?
      • expectation: exit code should not be zero.
      • current: exit code is zero.

      Have confirmed that reverting the commit does lead to the expected result.

      Haven't checked if there are other tasks affected the same way. Say stylelint, eslint... all them should lead to error exit code if there are errors.

      Ciao

      PS: I think this explains pretty much why, recently, some gherkin violations have met core and we haven't detected them (in fact that's ultimately the cause I prepared MDLSITE-6103). Since this regression, grunt is not failing any more... hence CiBoT says all ok, so there wasn't any reason to look. Before the regression, grunt was failing and, although it was hard to find the cause (well-hidden in log/xml files in the job) it was enough ro raise alarms and make us look for it.

        Attachments

        1. MDL-68405.jpg
          50 kB
          Anna Carissa Sadia
        2. MDL-68405 (2).jpg
          68 kB
          Anna Carissa Sadia

          Issue Links

            Activity

              People

              Assignee:
              dobedobedoh Andrew Nicols
              Reporter:
              stronk7 Eloy Lafuente (stronk7)
              Peer reviewer:
              Simey Lameze
              Integrator:
              Jun Pataleta
              Tester:
              Anna Carissa Sadia
              Participants:
              Component watchers:
              Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                11/May/20

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 6 hours, 24 minutes
                  2d 6h 24m