Moodle
  1. Moodle
  2. MDL-41421

Add a JMeter test plan generator based on a course contents

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6
    • Fix Version/s: 2.6
    • Component/s: Performance
    • Labels:
    • Testing Instructions:
      Hide

      Pre-requisite: A course without page activities, a course with page activities but without forum activity/ies and a course with page activity/ies and forum activity/ies with discussions and replies. All of them with, at least, 1 enrolled user.

      1. Set your debugging level to developer
      2. Edit config.php and add a new line with $CFG->tool_generator_users_password = 'imatest';
      3. Generate courses
        1. cd /to/your/moodle/dirroot && php admin/tool/generator/cli/maketestsite.php --size="S" --fixeddataset
      4. Browse to admin/tool/generator/maketestplan.php
      5. Test 1
        1. Select size XS
        2. Select the course without page activities
        3. Submit the form
        4. You SHOULD see an error message stating that there are no page activities
      6. Test 2
        1. Select size XS
        2. Select the course with page activities but without forum activities
        3. Submit the form
        4. You SHOULD see an error message stating that there are no forum activities
      7. Test 3
        1. Select size XS
        2. Select the course with page activities and with forum activity/ies
        3. Submit the form
        4. You SHOULD see a couple of links to download the test plan and the users file
      8. Test 4
        1. Download jmeter binaries from https://jmeter.apache.org/download_jmeter.cgi and store them in /opt/apache-jmeter-2.9 with fancy permissions
        2. cd /dir/inside/public/www
        3. git clone https://github.com/moodlehq/moodle-performance-comparison.git moodle-performance-comparison
        4. cd moodle-performance-comparison
        5. Follow https://github.com/moodlehq/moodle-performance-comparison/blob/master/README.md install instructions, filling the webserver_config.properties repository section with:
          1. a new site settings (dataroot, db data, wwwroot...)
          2. repository="git://git.moodle.org/integration.git"
          3. basecommit="d0b63cbde580750e116d2c4b2832eda4ae0db907"
          4. beforebranch="master"
          5. afterbranch="master"
          6. the services and the backup part according to your needs
        6. Follow usage instructions
          1. https://github.com/moodlehq/moodle-performance-comparison/blob/master/README.md#web-and-jmeter-servers-in-the-same-computer-usually-a-development-computer-1
          2. You don't really need to run restart_services.sh as this is not a serious performance comparison test
        7. Browse to http://path/to/moodle-performance-comparison
        8. Consider that this tool is not being tested here, it still has a few issues to solve
        9. You SHOULD see a web interface with a couple of drop down menus
        10. There SHOULD be a couple of options in each drop down
        11. Select a different option in each menu
        12. Press 'Load'
        13. You SHOULD see a few big percent numbers colored in red and green
        14. You SHOULD see the test plan steps in the left side, something like View course or Frontpage not logged
        15. You SHOULD see a few graphs
      9. Test 5
        1. Comment the $CFG->tool_generator_users_password = 'imatest'; line in config.php
        2. Open a CLI and cd /to/your/moodle/dirroot
        3. php admin/tool/generator/cli/maketestplan.php --size="S" --shortname="testcourse_12"
        4. You SHOULD see Error: You need to set $CFG->tool_generator_users_password in config.php to generate the test plan
      Show
      Pre-requisite: A course without page activities, a course with page activities but without forum activity/ies and a course with page activity/ies and forum activity/ies with discussions and replies. All of them with, at least, 1 enrolled user. Set your debugging level to developer Edit config.php and add a new line with $CFG->tool_generator_users_password = 'imatest'; Generate courses cd /to/your/moodle/dirroot && php admin/tool/generator/cli/maketestsite.php --size="S" --fixeddataset Browse to admin/tool/generator/maketestplan.php Test 1 Select size XS Select the course without page activities Submit the form You SHOULD see an error message stating that there are no page activities Test 2 Select size XS Select the course with page activities but without forum activities Submit the form You SHOULD see an error message stating that there are no forum activities Test 3 Select size XS Select the course with page activities and with forum activity/ies Submit the form You SHOULD see a couple of links to download the test plan and the users file Test 4 Download jmeter binaries from https://jmeter.apache.org/download_jmeter.cgi and store them in /opt/apache-jmeter-2.9 with fancy permissions cd /dir/inside/public/www git clone https://github.com/moodlehq/moodle-performance-comparison.git moodle-performance-comparison cd moodle-performance-comparison Follow https://github.com/moodlehq/moodle-performance-comparison/blob/master/README.md install instructions, filling the webserver_config.properties repository section with: a new site settings (dataroot, db data, wwwroot...) repository="git://git.moodle.org/integration.git" basecommit="d0b63cbde580750e116d2c4b2832eda4ae0db907" beforebranch="master" afterbranch="master" the services and the backup part according to your needs Follow usage instructions https://github.com/moodlehq/moodle-performance-comparison/blob/master/README.md#web-and-jmeter-servers-in-the-same-computer-usually-a-development-computer-1 You don't really need to run restart_services.sh as this is not a serious performance comparison test Browse to http://path/to/moodle-performance-comparison Consider that this tool is not being tested here, it still has a few issues to solve You SHOULD see a web interface with a couple of drop down menus There SHOULD be a couple of options in each drop down Select a different option in each menu Press 'Load' You SHOULD see a few big percent numbers colored in red and green You SHOULD see the test plan steps in the left side, something like View course or Frontpage not logged You SHOULD see a few graphs Test 5 Comment the $CFG->tool_generator_users_password = 'imatest'; line in config.php Open a CLI and cd /to/your/moodle/dirroot php admin/tool/generator/cli/maketestplan.php --size="S" --shortname="testcourse_12" You SHOULD see Error: You need to set $CFG->tool_generator_users_password in config.php to generate the test plan
    • Affected Branches:
      MOODLE_26_STABLE
    • Fixed Branches:
      MOODLE_26_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-41421_master

      Description

      As described in http://docs.moodle.org/dev/index.php?title=User:David_Monlla%C3%B3_Oliv%C3%A9/Performance-comparison_proposal#How_can_we_code_it.3F could be good to have a script to generate a jmeter test plan (and a csv with the users) so we can compare runs results using different settings or different codebase (different releases or after applying patches)

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            David Monllaó added a comment -

            Marking as blocked by MDL-41402 as the default target course will be based on MDL-41402 outputs

            Show
            David Monllaó added a comment - Marking as blocked by MDL-41402 as the default target course will be based on MDL-41402 outputs
            Hide
            David Monllaó added a comment - - edited

            This is a pretty simple test plan, the objective is to have, AT LEAST, something before the 2.6 release to compare test plan runs, this allows us to create test plans from generated courses and from restored courses or from any course that contains

            • A page activity
            • A forum activity with discussion/s and reply/ies

            The process is explained in http://docs.moodle.org/dev/index.php?title=User:David_Monlla%C3%B3_Oliv%C3%A9/Performance-comparison_proposal#How_can_we_code_it.3F

            About the patches:

            I have concerns about assigning a default password to the generated users, if somebody generates a user in a production site (can be done using the bypasscheck option of maketestcourse.php) it would be straight for a attacker to login as test_generated; right now they are not available as they use a 'not cached' value in the database. As an alternative I would propose to set a $CFG->tool_generator_password password in config.php so all the generated users will use that password to access the site.

            I was going to add extra checkings in make_testplan_form::validation() and testplan_backend::get_course_options() to discard courses without the required data but will slow down the performance and users are supposed to know which courses are they testing before selecting a course, most of the time will be testcourse_XXXX courses.

            We could also pass $size to tool_generator_testplan_backend::create_users_file(), I let all course users to be copied in the csv to allow manual .jmx post-editions.

            Show
            David Monllaó added a comment - - edited This is a pretty simple test plan, the objective is to have, AT LEAST, something before the 2.6 release to compare test plan runs, this allows us to create test plans from generated courses and from restored courses or from any course that contains A page activity A forum activity with discussion/s and reply/ies The process is explained in http://docs.moodle.org/dev/index.php?title=User:David_Monlla%C3%B3_Oliv%C3%A9/Performance-comparison_proposal#How_can_we_code_it.3F About the patches: I have concerns about assigning a default password to the generated users, if somebody generates a user in a production site (can be done using the bypasscheck option of maketestcourse.php) it would be straight for a attacker to login as test_generated; right now they are not available as they use a 'not cached' value in the database. As an alternative I would propose to set a $CFG->tool_generator_password password in config.php so all the generated users will use that password to access the site. I was going to add extra checkings in make_testplan_form::validation() and testplan_backend::get_course_options() to discard courses without the required data but will slow down the performance and users are supposed to know which courses are they testing before selecting a course, most of the time will be testcourse_XXXX courses. We could also pass $size to tool_generator_testplan_backend::create_users_file(), I let all course users to be copied in the csv to allow manual .jmx post-editions.
            Hide
            David Monllaó added a comment -

            I haven't changed the tool_plugin version, I see that, as there are new language strings, we need to purge cache otherwise we see a debugging message about an unexisting text string and the autoloader reports it as a

            Fatal error: Class 'tool_generator_site_backend' not found in /home/davidm/Desktop/moodlecode/master/admin/tool/generator/cli/maketestsite.php on line 45
            

            Show
            David Monllaó added a comment - I haven't changed the tool_plugin version, I see that, as there are new language strings, we need to purge cache otherwise we see a debugging message about an unexisting text string and the autoloader reports it as a Fatal error: Class 'tool_generator_site_backend' not found in /home/davidm/Desktop/moodlecode/master/admin/tool/generator/cli/maketestsite.php on line 45
            Hide
            David Monllaó added a comment -

            Completed test plan with:

            • Constant throughput timer
            • Warm-up threads group without logs report, just to fill moodle caches
            Show
            David Monllaó added a comment - Completed test plan with: Constant throughput timer Warm-up threads group without logs report, just to fill moodle caches
            Hide
            David Monllaó added a comment -

            Other stuff added:

            • CLI script to generate the plan, outputs the generated files paths, according to usual bash scripts needs
            • Compulsory to set users password when generating a test plan as jmeter needs to know it, adding a default one can be dangerous if the tool is used in production environments
            • Added the option to update generated users passwords
            • New assertion in the test plan, it reports errors in case the user credentials are not correct
            Show
            David Monllaó added a comment - Other stuff added: CLI script to generate the plan, outputs the generated files paths, according to usual bash scripts needs Compulsory to set users password when generating a test plan as jmeter needs to know it, adding a default one can be dangerous if the tool is used in production environments Added the option to update generated users passwords New assertion in the test plan, it reports errors in case the user credentials are not correct
            Hide
            Sam Hemelryk added a comment -

            I'm taking a peak at this now.

            Show
            Sam Hemelryk added a comment - I'm taking a peak at this now.
            Hide
            David Monllaó added a comment -

            Copy & paste of some of Sam's comments (commented in jabber):

            (11:53:02) samhemelryk@moodle.org: There was one blocking bug, the check for the $CFG password var is missing a global so it always fails presently
            (11:54:45) samhemelryk@moodle.org: At present the interfaces isn't overly intuitive, but I think most of that could be clarified by improving labels and descriptions etc
            (12:49:33) samhemelryk@moodle.org: 1. You've an odd mix. You have test plan attributes and then you hardcode them in when creating the plan from the template
            (12:49:47) samhemelryk@moodle.org: I think perhaps you would be better to make them properties with the defaults hardcoded
            (12:49:54) samhemelryk@moodle.org: That way you justify having them as attribtues
            (12:50:21) samhemelryk@moodle.org: and you enabled an "advanced" mode whereby if you run the tool from the command line you can override the properties at run time if you wish
            (12:53:27) samhemelryk@moodle.org: 2. It's pretty apparent that this template was produced by Jmeter and they embed version information etc into it
            (12:53:36) samhemelryk@moodle.org: It would be interesting to find out what we could cut out
            (12:54:17) samhemelryk@moodle.org: and also find someway to embed the version of the tool that created the file into both the test plan and perhaps the run file even
            (12:57:09) samhemelryk@moodle.org: 3. There is a bug with the recorder beanshell script if you put the server under real load whereby it may write the header several times into the file corrupting it

            Resume:

            • Move the project to moodlehq organization
            • Fix missing global $CFG;
            • Improve labelling and descriptions in general
            • And specifically about the test plan:
              • Replace user vars hardcoded values for properties with defaults and add options to test_runner.sh
              • Remove the not required parts of the jmx
              • About the recorder I was planning to move the run description (the php tag with the few info vars) to the end of the script and append it to the beginning of the file.
            Show
            David Monllaó added a comment - Copy & paste of some of Sam's comments (commented in jabber): (11:53:02) samhemelryk@moodle.org: There was one blocking bug, the check for the $CFG password var is missing a global so it always fails presently (11:54:45) samhemelryk@moodle.org: At present the interfaces isn't overly intuitive, but I think most of that could be clarified by improving labels and descriptions etc (12:49:33) samhemelryk@moodle.org: 1. You've an odd mix. You have test plan attributes and then you hardcode them in when creating the plan from the template (12:49:47) samhemelryk@moodle.org: I think perhaps you would be better to make them properties with the defaults hardcoded (12:49:54) samhemelryk@moodle.org: That way you justify having them as attribtues (12:50:21) samhemelryk@moodle.org: and you enabled an "advanced" mode whereby if you run the tool from the command line you can override the properties at run time if you wish (12:53:27) samhemelryk@moodle.org: 2. It's pretty apparent that this template was produced by Jmeter and they embed version information etc into it (12:53:36) samhemelryk@moodle.org: It would be interesting to find out what we could cut out (12:54:17) samhemelryk@moodle.org: and also find someway to embed the version of the tool that created the file into both the test plan and perhaps the run file even (12:57:09) samhemelryk@moodle.org: 3. There is a bug with the recorder beanshell script if you put the server under real load whereby it may write the header several times into the file corrupting it Resume: Move the project to moodlehq organization Fix missing global $CFG; Improve labelling and descriptions in general And specifically about the test plan: Replace user vars hardcoded values for properties with defaults and add options to test_runner.sh Remove the not required parts of the jmx About the recorder I was planning to move the run description (the php tag with the few info vars) to the end of the script and append it to the beginning of the file.
            Hide
            Rajesh Taneja added a comment -

            I am removing myself as peer-reviewer, as Sam H has been looking at this.

            Show
            Rajesh Taneja added a comment - I am removing myself as peer-reviewer, as Sam H has been looking at this.
            Hide
            David Monllaó added a comment -

            Patch updated according to Sam's review:

            Show
            David Monllaó added a comment - Patch updated according to Sam's review: Project now owned by moodlehq ( https://github.com/moodlehq/moodle-performance-comparison ) Added more info about what to do with the test plan Fixed missing global $CFG I've found that there is no jmx schema definition ( http://wiki.apache.org/jmeter/JMeterFAQ#Is_there_a_JMX_Schema.2FDTD_available.3F ) so I'm not 100% sure of what we can remove from the GUI-generated test plan. Created MDL-41769 and https://github.com/moodlehq/moodle-performance-comparison/issues/2 for switching to properties with default values Created https://github.com/moodlehq/moodle-performance-comparison/issues/3 to solve the recorder problem
            Hide
            David Monllaó added a comment -

            Requesting peer review again Sam.

            I removed /jmeterTestPlan[@jmeter] as it does not seem to be required (http://wiki.apache.org/jmeter/JmxTestPlan#A2.1_Format_Example) keeping the version and properties values, I haven't found many info about them.

            Show
            David Monllaó added a comment - Requesting peer review again Sam. I removed /jmeterTestPlan [@jmeter] as it does not seem to be required ( http://wiki.apache.org/jmeter/JmxTestPlan#A2.1_Format_Example ) keeping the version and properties values, I haven't found many info about them.
            Hide
            David Monllaó added a comment -

            I closed MDL-41769 and I included the changes here as there is nothing required in moodle-performance-comparison side.

            Show
            David Monllaó added a comment - I closed MDL-41769 and I included the changes here as there is nothing required in moodle-performance-comparison side.
            Hide
            David Monllaó added a comment -

            I've changed the test plan to get users, loops, rampup and throughput as properties, but saving their values in user vars as otherwise the default property value don't arrive to the BeanShell script.

            Show
            David Monllaó added a comment - I've changed the test plan to get users, loops, rampup and throughput as properties, but saving their values in user vars as otherwise the default property value don't arrive to the BeanShell script.
            Hide
            Marina Glancy added a comment -

            Hi David,
            great that we are moving towards automating tests and generators!

            couple of comments:

            1. what does it mean when you call is_bool() on password? If it is allowed to be 'true' - it is never documented and in fact may break other places - so it seems

            if (empty($CFG->tool_generator_users_password) || is_bool($CFG->tool_generator_users_password)) {
            

            2. Another thing that in cli script we usually recommend to run as "sudo -u www-data"
            https://github.com/moodle/moodle/blob/master/admin/cli/purge_caches.php#L44 - existing cli script
            https://github.com/dmonllao/moodle/compare/moodle:master...MDL-41421_master#L4R63 - generator cli script

            I do not reopen because those comments are minor, waiting for your response.

            PS (for the future) please submit whitespace changes in a separate commit - make reviewers/integrators life easier

            Show
            Marina Glancy added a comment - Hi David, great that we are moving towards automating tests and generators! couple of comments: 1. what does it mean when you call is_bool() on password? If it is allowed to be 'true' - it is never documented and in fact may break other places - so it seems if (empty($CFG->tool_generator_users_password) || is_bool($CFG->tool_generator_users_password)) { 2. Another thing that in cli script we usually recommend to run as "sudo -u www-data" https://github.com/moodle/moodle/blob/master/admin/cli/purge_caches.php#L44 - existing cli script https://github.com/dmonllao/moodle/compare/moodle:master...MDL-41421_master#L4R63 - generator cli script I do not reopen because those comments are minor, waiting for your response. PS (for the future) please submit whitespace changes in a separate commit - make reviewers/integrators life easier
            Hide
            David Monllaó added a comment -
            Show
            David Monllaó added a comment - Thanks Marina, I've updated the patch ( https://tracker.moodle.org/secure/attachment/34818/after_integration_review_changes.diff )
            Hide
            Marina Glancy added a comment -

            Thanks David, integrated in master.
            There was a merge conflict in admin/tool/generator/version.php, I changed the version to 2013091201

            Show
            Marina Glancy added a comment - Thanks David, integrated in master. There was a merge conflict in admin/tool/generator/version.php, I changed the version to 2013091201
            Hide
            Damyon Wiese added a comment -

            Noting that after getting jmeter to run, there are still no comparisons listed in the web interface. I think either the testing instructions and the tool are not in sync.

            Looking at the code for the tool - it apparently is looking for some php files in the runs/ folder - but I can't see anywhere where this is supposed to be created.

            Show
            Damyon Wiese added a comment - Noting that after getting jmeter to run, there are still no comparisons listed in the web interface. I think either the testing instructions and the tool are not in sync. Looking at the code for the tool - it apparently is looking for some php files in the runs/ folder - but I can't see anywhere where this is supposed to be created.
            Hide
            Damyon Wiese added a comment -

            Also - the help page for this tool could use a little tidying.

            What does p.e. stand for ?

            This:
            You need to set a password for the course users in config.php (p.e. $CFG->tool_generator_users_password = 'moodle' there is no default value for this password to prevent unintended usages of the tool.

            Needs to be broken into 2 sentences:
            You need to set a password for the course users in config.php (p.e. $CFG->tool_generator_users_password = 'moodle'. There is no default value for this password to prevent unintended usages of the tool.

            The bullet items in this text:

            It is part of tool_generator so it works well with the courses generated by the courses and the site generators, it can also be used with any course that contains, at least: * Enough enrolled users (depends on the test plan size you select) with the password reset to 'moodle' * A page module instance * A forum module instance with at least one discussion and one reply

            Could use some line breaks when displayed in HTML.

            Show
            Damyon Wiese added a comment - Also - the help page for this tool could use a little tidying. What does p.e. stand for ? This: You need to set a password for the course users in config.php (p.e. $CFG->tool_generator_users_password = 'moodle' there is no default value for this password to prevent unintended usages of the tool. Needs to be broken into 2 sentences: You need to set a password for the course users in config.php (p.e. $CFG->tool_generator_users_password = 'moodle' . There is no default value for this password to prevent unintended usages of the tool. The bullet items in this text: It is part of tool_generator so it works well with the courses generated by the courses and the site generators, it can also be used with any course that contains, at least: * Enough enrolled users (depends on the test plan size you select) with the password reset to 'moodle' * A page module instance * A forum module instance with at least one discussion and one reply Could use some line breaks when displayed in HTML.
            Hide
            David Monllaó added a comment - - edited

            Thanks for the comments Damyon, attaching a patch with the proposed changes; the p.e. was a communication problem between my English-side and my Catalan background.

            git pull git://github.com/dmonllao/moodle.git MDL-41421-help-fix

            You will need to purge caches as there is a language string change

            Show
            David Monllaó added a comment - - edited Thanks for the comments Damyon, attaching a patch with the proposed changes; the p.e. was a communication problem between my English-side and my Catalan background. git pull git://github.com/dmonllao/moodle.git MDL-41421 -help-fix You will need to purge caches as there is a language string change
            Hide
            Marina Glancy added a comment -

            fix integrated

            Show
            Marina Glancy added a comment - fix integrated
            Hide
            Damyon Wiese added a comment -

            Thanks David - the help text reads much better now.

            I solved my jmeter problems (need to install from website - not apt-get) and was able to run some tests and view the results.

            Thanks - passing!

            Show
            Damyon Wiese added a comment - Thanks David - the help text reads much better now. I solved my jmeter problems (need to install from website - not apt-get) and was able to run some tests and view the results. Thanks - passing!
            Hide
            Marina Glancy added a comment -

            And THANK YOU again for making Moodle better every day!

            Another weekly release has been released.

            Show
            Marina Glancy added a comment - And THANK YOU again for making Moodle better every day! Another weekly release has been released.

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: