Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-39752

Behat tests runs in parallel

XMLWordPrintable

    • MOODLE_25_STABLE, MOODLE_26_STABLE, MOODLE_27_STABLE, MOODLE_28_STABLE
    • MOODLE_29_STABLE
    • wip-mdl-39752
    • Hide

      Test on 1 db.

      1. php admin/tool/behat/cli/init.php --parallel=2
      2. php admin/tool/behat/cli/run.php
      3. Ensure you see 2 behat process.
      4. Check behat.yml and ensure they have different features.

      Test on different db.

      1. Set following variable in config: 

        $CFG->behat_parallel_run = array (
        		 
            array (
        		 
                'dbtype'    => 'mysqli',
        		 
                'dblibrary' => 'native',
        		 
                'dbhost'    => 'localhost',
        		 
                'dbname'    => 'moodle',
        		 
                'dbuser'    => 'moodle',
        		 
                'dbpass'    => 'moodle',
        		 
                'behat_prefix' => 'mdl_',
        		 
                'wd_host' => 'http://127.0.0.1:4444/wd/hub'
        		 
            ),
        		 
            array (
        		 
                 'dbtype'    => 'pgsql',
        		 
                'dblibrary' => 'native',
        		 
                'dbhost'    => 'localhost',
        		 
                'dbname'    => 'moodle',
        		 
                'dbuser'    => 'moodle',
        		 
                'dbpass'    => 'moodle',
        		 
                'behat_prefix' => 'mdl_',
        		 
                'wd_host' => 'http://127.0.0.1:4445/wd/hub',
        		 
            ),
        		 
        );

      2. Run 2 instance of selenium on pot 4444 and 4445
      3. php admin/tool/behat/cli/init.php --parallel=2
      4. php admin/tool/behat/cli/run.php
      5. Ensure you see 2 behat process on different wd.
      6. Check db's and make sure behat was installed.

      Test with different options

      1. php admin/tool/behat/cli/run.php -p=phantomjs-linux --format='progress,pretty' --out=',/home/rajesh/ot {runprocess}.txt' --replace="{runprocess}

        "

        1. This should allow you to output multiple formats
      2. php admin/tool/behat/cli/util.php --drop
        1. This should drop all parallel sites.

      Check step count formatter

      1. define('BEHAT_FEATURE_STEP_FILE', SOMEPATHTOFILE.json);
      2. php admin/tool/behat/cli/util.php --updatesteps
      3. Make sure BEHAT_FEATURE_STEP_FILE file gets updated with step count information.

      Test From run and to run

      1. Install site with 4 parallel runs
        1. php admin/tool/behat/cli/init.php --parallel=4
      2. Run following commands in 2 different consoles and makes sure only 2 processes run in each console and you get 4 windows.
        1. php admin/tool/behat/cli/run.php --fromrun=1 --torun=2
        2. php admin/tool/behat/cli/run.php --fromrun=2 --torun=4

      Test increasing command line output

      1. Drop previous site
      2. set define('BEHAT_MAX_CMD_LINE_OUTPUT', 500)
      3. Install site with 8 parallel runs
        1. php admin/tool/behat/cli/init.php --parallel=8
      4. Make sure installation output is expanded.

      Test installing few sites at once.

      1. Drop previous site
      2. Install site with 8 parallel runs in bunch of 2 each
        1. php admin/tool/behat/cli/init.php -j=8 -m=2
      3. Ensure 2 sites gets installed at once.

      Test single site.

      1. Drop previous site
      2. Install single site
        1. php admin/tool/behat/cli/init.php
      3. make sure after install, it shows to run vendor/bin/behat --config=...
      4. Run vendo/bin/behat command and make sure behat execution works as expected.
      5. php admin/tool/behat/cli/run.php and it should run behat as well.

      Check on Windows and Mac

      1. Check installing and running parallel behat execution on windows and mac and make sure it works as expected.
      2. Don't have to run all above tests, but would be nice to have most of them covered.
      Show
      Test on 1 db. php admin/tool/behat/cli/init.php --parallel=2 php admin/tool/behat/cli/run.php Ensure you see 2 behat process. Check behat.yml and ensure they have different features. Test on different db. Set following variable in config: $CFG->behat_parallel_run = array ( array ( 'dbtype' => 'mysqli', 'dblibrary' => 'native', 'dbhost' => 'localhost', 'dbname' => 'moodle', 'dbuser' => 'moodle', 'dbpass' => 'moodle', 'behat_prefix' => 'mdl_', 'wd_host' => 'http://127.0.0.1:4444/wd/hub' ), array ( 'dbtype' => 'pgsql', 'dblibrary' => 'native', 'dbhost' => 'localhost', 'dbname' => 'moodle', 'dbuser' => 'moodle', 'dbpass' => 'moodle', 'behat_prefix' => 'mdl_', 'wd_host' => 'http://127.0.0.1:4445/wd/hub', ), ); Run 2 instance of selenium on pot 4444 and 4445 php admin/tool/behat/cli/init.php --parallel=2 php admin/tool/behat/cli/run.php Ensure you see 2 behat process on different wd. Check db's and make sure behat was installed. Test with different options php admin/tool/behat/cli/run.php -p=phantomjs-linux --format='progress,pretty' --out=',/home/rajesh/ot {runprocess}.txt' --replace="{runprocess} " This should allow you to output multiple formats php admin/tool/behat/cli/util.php --drop This should drop all parallel sites. Check step count formatter define('BEHAT_FEATURE_STEP_FILE', SOMEPATHTOFILE.json); php admin/tool/behat/cli/util.php --updatesteps Make sure BEHAT_FEATURE_STEP_FILE file gets updated with step count information. Test From run and to run Install site with 4 parallel runs php admin/tool/behat/cli/init.php --parallel=4 Run following commands in 2 different consoles and makes sure only 2 processes run in each console and you get 4 windows. php admin/tool/behat/cli/run.php --fromrun=1 --torun=2 php admin/tool/behat/cli/run.php --fromrun=2 --torun=4 Test increasing command line output Drop previous site set define('BEHAT_MAX_CMD_LINE_OUTPUT', 500) Install site with 8 parallel runs php admin/tool/behat/cli/init.php --parallel=8 Make sure installation output is expanded. Test installing few sites at once. Drop previous site Install site with 8 parallel runs in bunch of 2 each php admin/tool/behat/cli/init.php -j=8 -m=2 Ensure 2 sites gets installed at once. Test single site. Drop previous site Install single site php admin/tool/behat/cli/init.php make sure after install, it shows to run vendor/bin/behat --config=... Run vendo/bin/behat command and make sure behat execution works as expected. php admin/tool/behat/cli/run.php and it should run behat as well. Check on Windows and Mac Check installing and running parallel behat execution on windows and mac and make sure it works as expected. Don't have to run all above tests, but would be nice to have most of them covered.

      After talking with Damyon about possible approaches to speed up the tests execution, runs in parallel as Petr also commented could help a lot. This requires multiple sites to be created.

      We can not keep the current simplicity as the 2nd step runs directly the behat CLI command without passing through any of our scripts, if we want to keep the 2 steps process (1.- install, 2.- run tests) we can add an extra CLI script and passthru() to vendor/bin/behat specifying which run should run which tags.

      I'm adding a proof of concept to ensure there are no major issues

            rajeshtaneja Rajesh Taneja
            dmonllao David Monllaó
            Damyon Wiese Damyon Wiese
            Dan Poltawski Dan Poltawski
            Dave Cooper Dave Cooper
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.