Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-48302

Behat tests execution not observing 100% isolation

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Pre-requisite:

      1. Open error log in one terminal to ensure you can see php errors

      Test 1: Run individual scenarios, ensuring you don't see any db error.

      1. Add @test tags to
        1. blocks/activity_modules/tests/behat/block_activity_modules.feature
        2. mod/quiz/tests/behat/editing_set_marks_with_attempts.feature
      2. Modify Course 1 to Course 2 in mod/quiz/tests/behat/editing_set_marks_with_attempts.feature
      3. Run behat with tags @test
      4. Repeat above few times, ensuring you don't see any error in error log.

      Test 2: Run whole suite

      1. Run whole suite ensuring you don't see any error related to dataroot dir/file creation (see comments for exact error)
      Show
      Pre-requisite: Open error log in one terminal to ensure you can see php errors Test 1: Run individual scenarios, ensuring you don't see any db error. Add @test tags to blocks/activity_modules/tests/behat/block_activity_modules.feature mod/quiz/tests/behat/editing_set_marks_with_attempts.feature Modify Course 1 to Course 2 in mod/quiz/tests/behat/editing_set_marks_with_attempts.feature Run behat with tags @test Repeat above few times, ensuring you don't see any error in error log. Test 2: Run whole suite Run whole suite ensuring you don't see any error related to dataroot dir/file creation (see comments for exact error)
    • Affected Branches:
      MOODLE_28_STABLE, MOODLE_29_STABLE
    • Fixed Branches:
      MOODLE_27_STABLE, MOODLE_28_STABLE
    • Pull Master Branch:
      wip-mdl-48302

      Description

      This is a followup of MDL-48251, where a "mysterious" interdependency between 2 tests was discovered (and fixed with a "nonsense" fix).

      In any case, it demoed that, at some point, there are unwanted dependencies between tests, so they are not running 100% isolated.

      This is about to detect what is causing isolation to fail and fix it. Note that it also may affect unit tests (if the problem is in some part of the "reset between tests" code).

      To be able to reproduce the problem (only tried with Postgres here):

      1. In master revert this (the "nonsense") commit: 7b9fb26
      2. Edit "blocks/activity_modules/tests/behat/block_activity_modules.feature" and add the @isolated tag to it.
      3. Edit "mod/quiz/tests/behat/editing_set_marks_with_attempts.feature" and add the @isolated tag to it.
      4. Run behat with --tags='@isolated'
      5. Failure happens and execution runs forever (it does not if the commit in point #1 is not reverted).

      Posible points to review can be (note this is just a bet):

      1) Some request/session/static MUC cache not being reset.
      2) Some problem in generators incorrectly using static data.
      3) Some problem with transactions, not properly closed or rolled-back incorrectly..

      It would be great to, apart from the solution, provide some tests ensuring it does not happen anymore (behat or phpunit).

      Ciao

      Note: I've set only 2.8.1 and 2.9 as affected versions (to match MDL-48251)... but once discovered the culprit this surely should be extended to all supported versions.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  12/Jan/15