Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-55072

Upgrades to Behat so it can work with different themes

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      NOTE: Features in Snap theme might fail, please ignore them.

      Test if behat.yml is being created properly.

      NOTE:

      1. behat_theme_snap_* context should only be available in snap suite
      2. behat_theme_snap_behat_selectors should not be included in behat.yml
      1. Test 1:
        • Initialise behat without patch (admin/tool/behat/cli/init.php)
        • Initialise behat with patch (admin/tool/behat/cli/init.php)
        • Compare behat.yml for default suite section and ensure they are same.
      1. Test 2:
        • Initialise behat without patch for parallel run (admin/tool/behat/cli/init.php -j=2)
        • Initialise behat with patch for parallel run (admin/tool/behat/cli/init.php -j=2)
        • compare behat.yml for run 1 and run2, and ensure default suite section is same for both behat.yml
      1. Test 3: clone https://github.com/rajeshtaneja/snap in your theme/ directory
        • Initilise behat without patch (admin/tool/behat/cli/init.php)
        • Initilise behat with patch (admin/tool/behat/cli/init.php)
        • Compare behat.yml for above and make sure contexts and features under suites > default is same (except theme snap features)
        • Ensure you can see paths and contexts under snap suite and it only contains missing themes ^
      1. Test 4: clone https://github.com/rajeshtaneja/snap in your theme/ directory
        • Initilise behat without patch (admin/tool/behat/cli/init.php)
        • Initilise behat with patch (admin/tool/behat/cli/init.php --run-with-theme)
        • Compare behat.yml for each run and make sure contexts and features under suites > default is same (except theme snap features)
        • Ensure you can see paths and contexts under snap suite and it only contains missing themes ^ + features from ^ - features mentioned in blacklist.json
      1. Test 5: Add "grade/tests/behat/behat_grade.php" in contexts in file /theme/snap/tests/behat/blacklist.json
        • Initialise behat with patch (admin/tool/behat/cli/init.php)
        • Ensure in behat.yml behat_grade, behat_navigation and behat_auth should not exist under contexts for snap suite, and there should be behat_theme_snap_behat_navigation and behat_theme_snap_behat_auth
        • Ensure in behat.yml features blacklisted in blacklist.json should not exist under paths for snap suite
        • change contexts name from "grade/tests/behat/behat_grade.php" to "behat_grade" and ensure after initialising behat_grade is not in contexts for snap suite
        • change contexts name from "behat_grade" to "behat_grade.php" and ensure after initialising behat_grade is not in contexts for snap suite

      Test all suites are run.

      1. Test 1:
        • Initialise behat (without --run-with-theme)
        • run scenario --name="Bold some text"
        • Ensure only 1 test is executed.
      2. Test 2:
        • Initialise behat (with --run-with-theme)
        • run scenario --name="Bold some text"
        • Ensure 3 test are executed, one with core theme, one with more theme and last with snap theme
      3. Test 3:
        • Initialise behat (with --run-with-theme)
        • run scenario --name="Student does not see admin block on any page."
        • Ensure only 1 test is executed with theme snap.
      4. Test 4:
        • Drop old behat site and Initialise behat with 2 runs (-j=2)
        • run php admin/tool/behat/cli/run.php --tags=@mod_assign
        • Ensure you can only see 1 runs with default theme only.
        • Check behat.yml for both runs and ensure there is no features under path for snap or more suite.
      5. Test 5:
        • run php admin/tool/behat/cli/run.php --tags=@mod_assign --run-with-theme
        • Ensure you can only see 2 runs with default theme, more theme and then with snap theme.
        • Check behat.yml for both runs and ensure there are features under path for snap suite.
      6. Test 6:
        • run php admin/tool/behat/cli/run.php --tags=@mod_assign --run-with-theme --suite=snap
        • Ensure you can only see 2 runs with snap theme only.
        • Check behat.yml for both runs and ensure there are features under path for snap suite.
      7. Test 7:
        • run php admin/tool/behat/cli/run.php --tags=@mod_assign --suite=snap
        • Ensure you can only see 2 runs started with no scenarios.
        • Check behat.yml for both runs and ensure there is no features under path for snap suite.

      Unit tests:

      1. Run unit test admin/tool/behat/tests/manager_test.php

      Extra credit

      1. Initialise behat (without --run-with-theme)
      2. Run behat (--format=pretty --out=~/pretty.txt --format=progress --out=std)
      3. After the run ensure all features in core were executed, default and theme specific. (Ignore behat failures.)

      Important: Test above on windows.

      Show
      NOTE: Features in Snap theme might fail, please ignore them. Test if behat.yml is being created properly. NOTE: behat_theme_snap_* context should only be available in snap suite behat_theme_snap_behat_selectors should not be included in behat.yml Test 1: Initialise behat without patch (admin/tool/behat/cli/init.php) Initialise behat with patch (admin/tool/behat/cli/init.php) Compare behat.yml for default suite section and ensure they are same. Test 2: Initialise behat without patch for parallel run (admin/tool/behat/cli/init.php -j=2) Initialise behat with patch for parallel run (admin/tool/behat/cli/init.php -j=2) compare behat.yml for run 1 and run2, and ensure default suite section is same for both behat.yml Test 3: clone https://github.com/rajeshtaneja/snap in your theme/ directory Initilise behat without patch (admin/tool/behat/cli/init.php) Initilise behat with patch (admin/tool/behat/cli/init.php) Compare behat.yml for above and make sure contexts and features under suites > default is same (except theme snap features) Ensure you can see paths and contexts under snap suite and it only contains missing themes ^ Test 4: clone https://github.com/rajeshtaneja/snap in your theme/ directory Initilise behat without patch (admin/tool/behat/cli/init.php) Initilise behat with patch (admin/tool/behat/cli/init.php --run-with-theme) Compare behat.yml for each run and make sure contexts and features under suites > default is same (except theme snap features) Ensure you can see paths and contexts under snap suite and it only contains missing themes ^ + features from ^ - features mentioned in blacklist.json Test 5: Add "grade/tests/behat/behat_grade.php" in contexts in file /theme/snap/tests/behat/blacklist.json Initialise behat with patch (admin/tool/behat/cli/init.php) Ensure in behat.yml behat_grade, behat_navigation and behat_auth should not exist under contexts for snap suite, and there should be behat_theme_snap_behat_navigation and behat_theme_snap_behat_auth Ensure in behat.yml features blacklisted in blacklist.json should not exist under paths for snap suite change contexts name from "grade/tests/behat/behat_grade.php" to "behat_grade" and ensure after initialising behat_grade is not in contexts for snap suite change contexts name from "behat_grade" to "behat_grade.php" and ensure after initialising behat_grade is not in contexts for snap suite Test all suites are run. Test 1: Initialise behat (without --run-with-theme) run scenario --name="Bold some text" Ensure only 1 test is executed. Test 2: Initialise behat (with --run-with-theme) run scenario --name="Bold some text" Ensure 3 test are executed, one with core theme, one with more theme and last with snap theme Test 3: Initialise behat (with --run-with-theme) run scenario --name="Student does not see admin block on any page." Ensure only 1 test is executed with theme snap. Test 4: Drop old behat site and Initialise behat with 2 runs (-j=2) run php admin/tool/behat/cli/run.php --tags=@mod_assign Ensure you can only see 1 runs with default theme only. Check behat.yml for both runs and ensure there is no features under path for snap or more suite. Test 5: run php admin/tool/behat/cli/run.php --tags=@mod_assign --run-with-theme Ensure you can only see 2 runs with default theme, more theme and then with snap theme. Check behat.yml for both runs and ensure there are features under path for snap suite. Test 6: run php admin/tool/behat/cli/run.php --tags=@mod_assign --run-with-theme --suite=snap Ensure you can only see 2 runs with snap theme only. Check behat.yml for both runs and ensure there are features under path for snap suite. Test 7: run php admin/tool/behat/cli/run.php --tags=@mod_assign --suite=snap Ensure you can only see 2 runs started with no scenarios. Check behat.yml for both runs and ensure there is no features under path for snap suite. Unit tests: Run unit test admin/tool/behat/tests/manager_test.php Extra credit Initialise behat (without --run-with-theme) Run behat (--format=pretty --out=~/pretty.txt --format=progress --out=std) After the run ensure all features in core were executed, default and theme specific. (Ignore behat failures.) Important: Test above on windows.
    • Affected Branches:
      MOODLE_32_STABLE
    • Fixed Branches:
      MOODLE_32_STABLE
    • Pull Master Branch:
      wip-mdl-55072

      Description

      Change behat infrastructure so steps can be overridden by themes. Should allow whole suite to pass on different themes even with different navigation etc.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                3 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  5/Dec/16