Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-49456

Improve behat parallel run

    XMLWordPrintable

Details

    • MOODLE_29_STABLE
    • MOODLE_29_STABLE
    • wip-mdl-49456
    • Easy
    • Hide

      Test 1: Enure behat process without feature file exit sucessfully without error

      1. Add @wip tag to some feature file (availability/condition/completion/tests/behat/availability_completion.feature)
      2. Install behat parallel site (php admin/tool/behat/cli/init.php -j=2)
      3. Run behat with tag @wip (php admin/tool/behat/cli/run.php --tags=@wip)
      4. Ensure both process exit with sucess and there is no error/notice.

      Test 2: Check behat_wwwroot is set properly for each process

      1. Set following in config.php 

        $CFG->behat_parallel_run = array (
        		 
            array ('wd_host' => 'http://127.0.0.1:4444/wd/hub', 'behat_wwwroot' => "http://127.0.0.1/behat1", 'behat_dataroot' => "/PATH_TO_DATAROOT/behat_test"),
        		 
            array ('wd_host' => 'http://127.0.0.1:4445/wd/hub', 'behat_wwwroot' => "http://127.0.0.1/behat3", 'behat_dataroot' => "/PATH_TO_DATAROOT/behat_secondtest")
        		 
        );

      2. Initilise behat with 2 parallel runs (php admin/tool/behat/cli/init.php -j=2)
      3. Check behat.yml files for both paralle runs and ensure
        • base_url points to correct behat_wwwroot
        • wd_host points to correct selenium wd_host
      4. Run behat with tag @wip
      5. Check behat.yml files for both paralle runs again.

      Test 3: Check exit codes are properly formatted

      1. Run behat (php admin/tool/behat/cli/run.php --tags=@wip --verbose)
      2. Ensure exit codes are shown properly for each process

      Test 4: Check --name and --feature works with parallel behat

      1. Run php admin/tool/behat/cli/run.php --verbose --name="Test condition"
      2. Ensure only 1 scenario is executed.
      3. Run php admin/tool/behat/cli/run.php --verbose --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature
      4. Ensure only process is executed and confirm behat command has above feature file path.

      Test 5: Tag || and && support

      1. php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="~@wip&&@javascript" should not execute any scenario
      2. php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="@wip&&@javascript" shoud run 1 scenario
      3. php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="~@wip,@javascript" shoud run 1 scenario
      4. php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="@wip&&@javascript,@availability" should not execute any scenario

      Test 6: Test singe run

      1. Drop old site (php admin/tool/behat/cli/utl.php --drop)
      2. Initialize single behat site (php admin/tool/behat/cli/init.php)
      3. php admin/tool/behat/cli/run.php (It should show help)
      4. php admin/tool/behat/cli/run.php --single-run , it should not ask you anything and single run should start

      Test 7: Get proper message, when running behat without initilising it

      1. Drop old site (php admin/tool/behat/cli/utl.php --drop)
      2. Try run behat (php admin/tool/behat/cli/run.php)
      3. Ensure you get proper message, saying you haven't installed site for parallel run.

      Test 8: Check help contains proper usage message

      1. Run php admin/tool/behat/cli/util.php -h
      2. Ensure usage is properly explained.
      3. Run php admin/tool/behat/cli/util_single_run.php -h
      4. Ensure usage is properly explained.
      5. Run php admin/tool/behat/cli/init.php -h
      6. Ensure usage is properly explained.
      7. Run php admin/tool/behat/cli/run.php -h
      8. Ensure usage is properly explained.

      Test 9: run.php should be able to run from any path

      1. Change working directory to some path other then moodle root (cd admin/tool)
      2. Initialise site (php behat/cli/init.php -j=2)
      3. Ensure site is initialized
      4. Run parallel behat (php behat/cli/run.php)
      5. Ensure it works.
      6. Try same for single run.

      Test 10: Check behat options are supported

      1. Check if behat options like --format etc. are supported (http://behat.readthedocs.org/en/v2.5/guides/6.cli.html)
      Show
      Test 1: Enure behat process without feature file exit sucessfully without error Add @wip tag to some feature file (availability/condition/completion/tests/behat/availability_completion.feature) Install behat parallel site (php admin/tool/behat/cli/init.php -j=2) Run behat with tag @wip (php admin/tool/behat/cli/run.php --tags=@wip) Ensure both process exit with sucess and there is no error/notice. Test 2: Check behat_wwwroot is set properly for each process Set following in config.php $CFG->behat_parallel_run = array ( array ('wd_host' => 'http://127.0.0.1:4444/wd/hub', 'behat_wwwroot' => "http://127.0.0.1/behat1", 'behat_dataroot' => "/PATH_TO_DATAROOT/behat_test"), array ('wd_host' => 'http://127.0.0.1:4445/wd/hub', 'behat_wwwroot' => "http://127.0.0.1/behat3", 'behat_dataroot' => "/PATH_TO_DATAROOT/behat_secondtest") ); Initilise behat with 2 parallel runs (php admin/tool/behat/cli/init.php -j=2) Check behat.yml files for both paralle runs and ensure base_url points to correct behat_wwwroot wd_host points to correct selenium wd_host Run behat with tag @wip Check behat.yml files for both paralle runs again. Test 3: Check exit codes are properly formatted Run behat (php admin/tool/behat/cli/run.php --tags=@wip --verbose) Ensure exit codes are shown properly for each process Test 4: Check --name and --feature works with parallel behat Run php admin/tool/behat/cli/run.php --verbose --name="Test condition" Ensure only 1 scenario is executed. Run php admin/tool/behat/cli/run.php --verbose --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature Ensure only process is executed and confirm behat command has above feature file path. Test 5: Tag || and && support php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="~@wip&&@javascript" should not execute any scenario php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="@wip&&@javascript" shoud run 1 scenario php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags="~@wip,@javascript" shoud run 1 scenario php admin/tool/behat/cli/run.php --feature=/var/www/m/availability/condition/completion/tests/behat/availability_completion.feature --tags=" @wip&&@javascript, @availability" should not execute any scenario Test 6: Test singe run Drop old site (php admin/tool/behat/cli/utl.php --drop) Initialize single behat site (php admin/tool/behat/cli/init.php) php admin/tool/behat/cli/run.php (It should show help) php admin/tool/behat/cli/run.php --single-run , it should not ask you anything and single run should start Test 7: Get proper message, when running behat without initilising it Drop old site (php admin/tool/behat/cli/utl.php --drop) Try run behat (php admin/tool/behat/cli/run.php) Ensure you get proper message, saying you haven't installed site for parallel run. Test 8: Check help contains proper usage message Run php admin/tool/behat/cli/util.php -h Ensure usage is properly explained. Run php admin/tool/behat/cli/util_single_run.php -h Ensure usage is properly explained. Run php admin/tool/behat/cli/init.php -h Ensure usage is properly explained. Run php admin/tool/behat/cli/run.php -h Ensure usage is properly explained. Test 9: run.php should be able to run from any path Change working directory to some path other then moodle root (cd admin/tool) Initialise site (php behat/cli/init.php -j=2) Ensure site is initialized Run parallel behat (php behat/cli/run.php) Ensure it works. Try same for single run. Test 10: Check behat options are supported Check if behat options like --format etc. are supported ( http://behat.readthedocs.org/en/v2.5/guides/6.cli.html )

    Description

      With behat parallel run in place, it will be nice to have following feature as well:
      1. Support --name / feature
      2. Add option to run single run without asking user
      3. Support for re-run option, so you don't have to do it manually ...
      4. Allow manual distribution of features, as current way of step count and timing has limitations.

      • Some steps take longer then others, so it really don't work so optimal
      • Timing will vary on system and need to run whole suite once, before getting that information

      5. Add support for tag or'ing.
      6. Exit codes are reversed and they should contain o/p run appended to show correct information.
      7. Add support for feature files in sub-folder (This is supported by behat and before MDL-49622, we were setting the folder path so they were indirectly supported)
      8. Handle cases where 1 process have feature files and rest don't. 

      Documents/Moodle/instances/im/moodle/lib/behat/classes/behat_config_manager.php on line 357
      		 
      PHP Stack trace:
      		 
      PHP   1. {main}() /home/damyonw/Documents/Moodle/instances/im/moodle/admin/tool/behat/cli/run.php:0
      		 
      PHP   2. behat_config_manager::update_config_file() /home/damyonw/Documents/Moodle/instances/im/moodle/admin/tool/behat/cli/run.php:167
      		 
      PHP   3. behat_config_manager::get_config_file_contents() /home/damyonw/Documents/Moodle/instances/im/moodle/lib/behat/classes/behat_config_manager.php:112

      Attachments

        Issue Links

          has a non-specific relationship to

          Bug - A problem which impairs or prevents Moodle from functioning correctly. MDL-49622 Some annoyances with behat run.php

          • Minor - Minor loss of function where workaround is possible
          • Closed

          Activity

            People

              rajeshtaneja Rajesh Taneja
              rajeshtaneja Rajesh Taneja
              Adrian Greeve Adrian Greeve
              David Monllaó David Monllaó
              Simey Lameze Simey Lameze
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                11/May/15