Details

    • Rank:
      44337

      Description

      The aim is to allow moodle components to have they own features making use of the steps definitions, allowing a behavioural driven development.

      More info about Behat & Moodle in http://docs.moodle.org/dev/Behat

        Issue Links

          Activity

          Hide
          David Monllaó added a comment -

          Editing name and description

          Show
          David Monllaó added a comment - Editing name and description
          Hide
          David Monllaó added a comment -

          As talked with Dan this integration will require more discussion to avoid adding external CLI tools in moodle codebase.

          The way phpunit is integrated could be a good way to do, Behat should be installed separately and it's execution should be external but could be good to allow moodle components to add it's own features files (and steps definitions if possible) in the moodle codebase instead of keeping them in the moodle-behat-features project.

          Show
          David Monllaó added a comment - As talked with Dan this integration will require more discussion to avoid adding external CLI tools in moodle codebase. The way phpunit is integrated could be a good way to do, Behat should be installed separately and it's execution should be external but could be good to allow moodle components to add it's own features files (and steps definitions if possible) in the moodle codebase instead of keeping them in the moodle-behat-features project.
          Hide
          Dan Poltawski added a comment -

          The way I see it, is that we have the possibility to have 'infrastructure' so let developers and plugin writers to distribute their feature plugins in a standard way like with phpunit. Say /tests/behat/ subdirectory could contain the feature files.

          Then, in core we have the functionality to run all the behat tests in all the plugins. Much like the phpunit integration, which generates the appropiate phpunit.xml file.

          Then, if all falls into place in the future, developers can submit both their feature files (and perhaps step definitions) with their plugins and it can also become part of the integration process. Together as one.

          Show
          Dan Poltawski added a comment - The way I see it, is that we have the possibility to have 'infrastructure' so let developers and plugin writers to distribute their feature plugins in a standard way like with phpunit. Say /tests/behat/ subdirectory could contain the feature files. Then, in core we have the functionality to run all the behat tests in all the plugins. Much like the phpunit integration, which generates the appropiate phpunit.xml file. Then, if all falls into place in the future, developers can submit both their feature files (and perhaps step definitions) with their plugins and it can also become part of the integration process. Together as one.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          +1

          • to create a "similar/parallel to phpunit" (install, reset, search, run...) behat support/infrastructure in core. In fact, surely it could reuse a lot of (install, reset) logic from it.
          • find some way to automate the documentation of available steps, so they can be reused easily.
          • incorporate all working behat tests within xxx/tests/behat

          yay!

          Show
          Eloy Lafuente (stronk7) added a comment - +1 to create a "similar/parallel to phpunit" (install, reset, search, run...) behat support/infrastructure in core. In fact, surely it could reuse a lot of (install, reset) logic from it. find some way to automate the documentation of available steps, so they can be reused easily. incorporate all working behat tests within xxx/tests/behat yay!
          Hide
          David Monllaó added a comment -

          I just realize (yes, I'm slow) that probably all the features/steps should be executed in a test environment (db + moodledata) like phpunit does, but I'm not sure if creating a new set of tables and another moodledata for behat is redundant, could not be used the same phpunit test environment?

          Show
          David Monllaó added a comment - I just realize (yes, I'm slow) that probably all the features/steps should be executed in a test environment (db + moodledata) like phpunit does, but I'm not sure if creating a new set of tables and another moodledata for behat is redundant, could not be used the same phpunit test environment?
          Hide
          Aparup Banerjee added a comment - - edited

          atm there are tests/fixtures/ and tests/generators/ all pertaining to phpunit stuff.
          having tests/behat seems to rank those phpunit test support directories at the same level as behat.

          perhaps we should have a quick refactor of stuff into */tests/phpunit/(stuff only for phpunit) so that we can have */test/behat without any confusion within the */tests/ level going on.

          Show
          Aparup Banerjee added a comment - - edited atm there are tests/fixtures/ and tests/generators/ all pertaining to phpunit stuff. having tests/behat seems to rank those phpunit test support directories at the same level as behat. perhaps we should have a quick refactor of stuff into */tests/phpunit/(stuff only for phpunit) so that we can have */test/behat without any confusion within the */tests/ level going on.
          Hide
          Petr Škoda added a comment -

          Let's not over-organize the /tests/ subdirectories, I agree with Eloy that /tests/behat/* should be enough.

          I suppose we could support phpunit installation via composer, I personally never liked PEAR much.

          Show
          Petr Škoda added a comment - Let's not over-organize the /tests/ subdirectories, I agree with Eloy that /tests/behat/* should be enough. I suppose we could support phpunit installation via composer, I personally never liked PEAR much.
          Hide
          David Monllaó added a comment -

          Continuing the integration room chat about MDL-36701 and using composer also for behat dependencies, it seems that ATM composer does not accept multiple profiles or multiple deployment-configurations, is just one composer.json per project with all the dependencies and versions and an optional composer.lock to specific versions of the dependencies (based on the commit hash) http://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file.

          There are a lot of benefits on using composer for dependencies management and one of them is to have a single location for each dependency (behat also depends on the phpunit package) and it seems perfect to specify the behat dependencies there:

          • Easier to install
          • All packaged together
          • Same installation process than phpunit (also testing stuff)

          The only inconvenient is for people who wants to use unit testing and not functional testing, they would have to download more data.

          The other options are what we are currently doing (separate project) or maybe we can add another composer.json inside tool/behat, a moodledata folder or whatever, but this second option sounds tricky and probably will conflict with classes/namespaces autoloading also duplicates dependencies code (phpunit and his children at the moment)

          Show
          David Monllaó added a comment - Continuing the integration room chat about MDL-36701 and using composer also for behat dependencies, it seems that ATM composer does not accept multiple profiles or multiple deployment-configurations, is just one composer.json per project with all the dependencies and versions and an optional composer.lock to specific versions of the dependencies (based on the commit hash) http://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file . There are a lot of benefits on using composer for dependencies management and one of them is to have a single location for each dependency (behat also depends on the phpunit package) and it seems perfect to specify the behat dependencies there: Easier to install All packaged together Same installation process than phpunit (also testing stuff) The only inconvenient is for people who wants to use unit testing and not functional testing, they would have to download more data. The other options are what we are currently doing (separate project) or maybe we can add another composer.json inside tool/behat, a moodledata folder or whatever, but this second option sounds tricky and probably will conflict with classes/namespaces autoloading also duplicates dependencies code (phpunit and his children at the moment)
          Hide
          Aparup Banerjee added a comment - - edited

          David had mentioned about using the composer.json file added in MDL-36701 for behat too.
          we had a short discussion in integration chat and there seemed to be the case where the system didn't have behat stuff setup so just noting that it might not be desirable to have behat dependencies as well as phpunit dependencies in the same file.

          (David has looked up if composer would accept installation options for correlations to depedencies needed and apparently thats not there)

          edit: or what David just commented above :-p

          Show
          Aparup Banerjee added a comment - - edited David had mentioned about using the composer.json file added in MDL-36701 for behat too. we had a short discussion in integration chat and there seemed to be the case where the system didn't have behat stuff setup so just noting that it might not be desirable to have behat dependencies as well as phpunit dependencies in the same file. (David has looked up if composer would accept installation options for correlations to depedencies needed and apparently thats not there) edit: or what David just commented above :-p
          Hide
          David Monllaó added a comment -

          For the record, the nearest I've found so far for this use case is http://getcomposer.org/doc/articles/scripts.md

          Show
          David Monllaó added a comment - For the record, the nearest I've found so far for this use case is http://getcomposer.org/doc/articles/scripts.md
          Hide
          David Monllaó added a comment -

          Adding more info, I've attached links to discussions of the lead devs. there are 2 ways to declare dependencies: require and require-dev, and there are no intentions of adding more ways to do it (see links below)

          I'm writting a proposal using require-dev to allow separate installation for unit tests and behat tests, but IMO both phpunit and behat dependencies should be at the same require level. Composer provides commands to add additional requirements but requires users who wants to run acceptance tests to execute a bunch of composer commands to add the behat dependencies, which could be good to avoid to ease the installation.

          Show
          David Monllaó added a comment - Adding more info, I've attached links to discussions of the lead devs. there are 2 ways to declare dependencies: require and require-dev, and there are no intentions of adding more ways to do it (see links below) I'm writting a proposal using require-dev to allow separate installation for unit tests and behat tests, but IMO both phpunit and behat dependencies should be at the same require level. Composer provides commands to add additional requirements but requires users who wants to run acceptance tests to execute a bunch of composer commands to add the behat dependencies, which could be good to avoid to ease the installation. Removed old option to install suggested packages: https://github.com/composer/composer/issues/662#issuecomment-5562336 and https://github.com/composer/composer/issues/510 Pull request related with having composer.json heritance rejected: https://github.com/composer/composer/pull/1013 require-dev attribute addition discussion: https://github.com/composer/composer/issues/78 require-dev docu: https://github.com/composer/composer/blob/master/doc/04-schema.md#require-dev-root-only
          Hide
          David Monllaó added a comment -

          The Moodle extension of behat (package moodlehq/behat-extension, used to declare alternative behaviours to behat's default ones, like the steps printer, features list loader...) also has it's own composer.json and does not have sense without a Moodle site behind as a standalone behat extension, so I've specified there the dependencies instead of in Moodle's composer.json. For future Moodle releases and major changes on the behat side we can specify different versions of moodlehq/behat-extension for different releases (we just have to create a git tag and point to it) As a alternative we can specify the versions in Moodle's composer.json and leave the moodlehq/behat-extension composer.json without dependencies.

          Show
          David Monllaó added a comment - The Moodle extension of behat (package moodlehq/behat-extension, used to declare alternative behaviours to behat's default ones, like the steps printer, features list loader...) also has it's own composer.json and does not have sense without a Moodle site behind as a standalone behat extension, so I've specified there the dependencies instead of in Moodle's composer.json. For future Moodle releases and major changes on the behat side we can specify different versions of moodlehq/behat-extension for different releases (we just have to create a git tag and point to it) As a alternative we can specify the versions in Moodle's composer.json and leave the moodlehq/behat-extension composer.json without dependencies.
          Hide
          Dan Poltawski added a comment -

          As discussed today in the office, I do not think we need to make this into a '1 step' process if that makes it much harder to write and maintain the tool.

          From my understanding, if we setup the environment initially using our tools (e.g. admin/tool/behat/init.php ) and then are able to use the behat cli itself it works better. That way, we are using the tools in the way they are supposed to be used.

          Also, this means that IDE's, and things like jenkins plugins will just work using behat, rather than us having to modify them to use our own custom scripts.

          Show
          Dan Poltawski added a comment - As discussed today in the office, I do not think we need to make this into a '1 step' process if that makes it much harder to write and maintain the tool. From my understanding, if we setup the environment initially using our tools (e.g. admin/tool/behat/init.php ) and then are able to use the behat cli itself it works better. That way, we are using the tools in the way they are supposed to be used. Also, this means that IDE's, and things like jenkins plugins will just work using behat, rather than us having to modify them to use our own custom scripts.
          Hide
          David Monllaó added a comment -

          To contextualize what Dan is talking about I've added pull branches to http://tracker.moodle.org/browse/MDL-37046 and http://tracker.moodle.org/browse/MDL-36269 (on top of MDL-37046, this one can be more useful because has more tests) The documentation (http://docs.moodle.org/dev/Acceptance_testing) is up to date with the current steps to install and run the tests, splitting the current runner into different parts would only change the usage, the installation would be the same.

          The whole process, currently wrapped in a single PHP CLI command to make it's usage as simple as possible is:

          1. Start the PHP 5.4 built-in server through proc_open() & proc_close()
          2. Gather steps definitions and .feature files (acceptance tests) to update the behat.yml file (some kind of phpunit.xml manifest)
          3. Create a file in dataroot to switch to the test environment when the site is accessed through the built-in server (it's only purpose is to allow Moodle to run on the built-in server without using the test environment)
          4. Run the behat tests through passthru() receiving the output when the tests finishes
          5. Delete the file in dataroot (switching off test environment when accessing through the built-in server)
          6. Stop the PHP built-in server

          The limitations of this approach are:

          • We can't see the progress of the tests execution until it's done
          • behat command can not run as a standalone commands easily (it can be done specifying CLI options)
          • Possible issues when stopping the CLI command before the end and the built-in server, so it is not stopped properly

          The are different options to split the process:

          • Start the server in a different shell window
          • Running a admin/tool/behat/cli/init.php before running the tests and a admin/tool/behat/cli/stop.php when tests are done OR restrict the execution of Moodle in the built-in server to the test environment

          Feedback would be appreciated, the idea is to make it as easy and quick to run as possible, but having a more ad-hoc solution also have limitations as Dan says.

          Show
          David Monllaó added a comment - To contextualize what Dan is talking about I've added pull branches to http://tracker.moodle.org/browse/MDL-37046 and http://tracker.moodle.org/browse/MDL-36269 (on top of MDL-37046 , this one can be more useful because has more tests) The documentation ( http://docs.moodle.org/dev/Acceptance_testing ) is up to date with the current steps to install and run the tests, splitting the current runner into different parts would only change the usage, the installation would be the same. The whole process, currently wrapped in a single PHP CLI command to make it's usage as simple as possible is: Start the PHP 5.4 built-in server through proc_open() & proc_close() Gather steps definitions and .feature files (acceptance tests) to update the behat.yml file (some kind of phpunit.xml manifest) Create a file in dataroot to switch to the test environment when the site is accessed through the built-in server (it's only purpose is to allow Moodle to run on the built-in server without using the test environment) Run the behat tests through passthru() receiving the output when the tests finishes Delete the file in dataroot (switching off test environment when accessing through the built-in server) Stop the PHP built-in server The limitations of this approach are: We can't see the progress of the tests execution until it's done behat command can not run as a standalone commands easily (it can be done specifying CLI options) Possible issues when stopping the CLI command before the end and the built-in server, so it is not stopped properly The are different options to split the process: Start the server in a different shell window Running a admin/tool/behat/cli/init.php before running the tests and a admin/tool/behat/cli/stop.php when tests are done OR restrict the execution of Moodle in the built-in server to the test environment Feedback would be appreciated, the idea is to make it as easy and quick to run as possible, but having a more ad-hoc solution also have limitations as Dan says.
          Hide
          David Monllaó added a comment - - edited

          MDL-37046 and MDL-36269 branches updated splitting the process in 3 parts:

          Documentation in http://docs.moodle.org/dev/Acceptance_testing up-to-date according to it.

          All the tests runs in localhost:8000 by default (you can overwrite this value in config.php with CFG->behat_wwwtoor) The test database and dataroot is reset before each scenario runs, so is important that Behat, running as a standalone command, does not reset nor executes any test in the regular environment, this is prevented by behat hooks before the whole test suite begins and before each scenario, and also by phpunit reset methods.

          To play with it use MDL-36269 pull branch, MDL-37046 don't has tests, is just the Moodle integration.

          Show
          David Monllaó added a comment - - edited MDL-37046 and MDL-36269 branches updated splitting the process in 3 parts: Start the built-in web server ( http://php.net/manual/en/features.commandline.webserver.php ) Enable the test environment (updates the list of tests and steps definitions reading the Moodle codebase) Run tests with Behat ( http://docs.behat.org/guides/6.cli.html ) Documentation in http://docs.moodle.org/dev/Acceptance_testing up-to-date according to it. All the tests runs in localhost:8000 by default (you can overwrite this value in config.php with CFG->behat_wwwtoor) The test database and dataroot is reset before each scenario runs, so is important that Behat, running as a standalone command, does not reset nor executes any test in the regular environment, this is prevented by behat hooks before the whole test suite begins and before each scenario, and also by phpunit reset methods. To play with it use MDL-36269 pull branch, MDL-37046 don't has tests, is just the Moodle integration.
          Hide
          David Monllaó added a comment -

          Sending MDL-37046 to peer review

          Show
          David Monllaó added a comment - Sending MDL-37046 to peer review
          Hide
          David Scotson added a comment -

          Just wondering if people were aware of Codeception, which appears to be a Behat rival:

          http://codeception.com/

          They have an entry on their blog about where they differ from Behat in philosophy, though as far as I can tell most of the lower layers of both use the same technology (e.g. PHPUnit and Mink).

          Just FYI.

          Show
          David Scotson added a comment - Just wondering if people were aware of Codeception, which appears to be a Behat rival: http://codeception.com/ They have an entry on their blog about where they differ from Behat in philosophy, though as far as I can tell most of the lower layers of both use the same technology (e.g. PHPUnit and Mink). Just FYI.
          Hide
          Martin Dougiamas added a comment -

          Things seem to be superceded faster and faster these days! Unfortunately I think if we always chase everything new we'll never have anything to actually use.

          Show
          Martin Dougiamas added a comment - Things seem to be superceded faster and faster these days! Unfortunately I think if we always chase everything new we'll never have anything to actually use.
          Hide
          Matt Gibson added a comment -

          Having recently learned Ruby and gotten addicted to Cucumber tests, I can't wait to see Behat make it into core, the same way PHPUnit is. It's going to dramaitically cut the amount of dev time lost on finding bugs.

          I'd prefer Behat over codeception, simply because the tests are written the same way as Cucumber - easy to understand plain English that you can discuss with non-coders really easily. Interesting tool, though.

          Show
          Matt Gibson added a comment - Having recently learned Ruby and gotten addicted to Cucumber tests, I can't wait to see Behat make it into core, the same way PHPUnit is. It's going to dramaitically cut the amount of dev time lost on finding bugs. I'd prefer Behat over codeception, simply because the tests are written the same way as Cucumber - easy to understand plain English that you can discuss with non-coders really easily. Interesting tool, though.
          Hide
          Petr Škoda added a comment -

          Hi, we have discussed the behat issues again today with David. We agreed that I will review everything again at the end of this week and we will try to find some compromises in phpunit x behat code.

          Show
          Petr Škoda added a comment - Hi, we have discussed the behat issues again today with David. We agreed that I will review everything again at the end of this week and we will try to find some compromises in phpunit x behat code.
          Hide
          David Monllaó added a comment -

          Converting MDL-36269 to an issue to create there subtasks

          Show
          David Monllaó added a comment - Converting MDL-36269 to an issue to create there subtasks
          Hide
          David Monllaó added a comment -

          Closing as all subtasks are closed

          Show
          David Monllaó added a comment - Closing as all subtasks are closed
          Hide
          Michael de Raadt added a comment -

          Well done, David.

          Now the real work begins.

          Show
          Michael de Raadt added a comment - Well done, David. Now the real work begins.
          Hide
          Ruslan Kabalin added a comment -

          I support the idea of keeping both behat and phpunit. Though, BDD concept is based on principles of TDD, it has a different focus. With BDD acceptance tests (Behat) we do validation (i.e. testing if we built the right thing), while with TDD (PHPunit) we do verification (i.e. if the thing built right). With BDD scenarios it is difficult to identify the code design issues, while it is perfect to test the functionality from user perspective in general and the interface for example. When it comes to more low-level testing, the particular, specific sections of code, functions behaviour and outcomes, the unit testing should come into play.

          Having said that, the main principle, however, of both BDD and TDD is that tests should be designed before writing the code

          You may find useful a book of Armando Fox & David Patterson that explain these differences using Ruby-on-Rails tools (RSpec/Cucumber/Capybara).

          Show
          Ruslan Kabalin added a comment - I support the idea of keeping both behat and phpunit. Though, BDD concept is based on principles of TDD, it has a different focus. With BDD acceptance tests (Behat) we do validation (i.e. testing if we built the right thing), while with TDD (PHPunit) we do verification (i.e. if the thing built right). With BDD scenarios it is difficult to identify the code design issues, while it is perfect to test the functionality from user perspective in general and the interface for example. When it comes to more low-level testing, the particular, specific sections of code, functions behaviour and outcomes, the unit testing should come into play. Having said that, the main principle, however, of both BDD and TDD is that tests should be designed before writing the code You may find useful a book of Armando Fox & David Patterson that explain these differences using Ruby-on-Rails tools (RSpec/Cucumber/Capybara).

            People

            • Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: