Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-72921

Make Jmeter test plans to behave more stable (using correct number of users)

    XMLWordPrintable

Details

    • MOODLE_310_STABLE, MOODLE_311_STABLE, MOODLE_400_STABLE
    • MOODLE_310_STABLE, MOODLE_311_STABLE
    • MDL-72921_311
    • Hide

      Before the patch

      1. Install a site using PostgreSQL.
      2. Edit config.php and add this line anywhere before the last line: 

        $CFG->tool_generator_users_password = 'i_love_moodle';

      3. Let's generate the testing site. Run: 

        php admin/tool/generator/cli/maketestsite.php  \
        		 
            --size=S --fixeddataset --bypasscheck \
        		 
            --filesizelimit="1000" --quiet

      4. Let's generate the testing plan. Run: 

        php admin/tool/generator/cli/maketestplan.php \
        		 
            --size=S --shortname='testcourse_11' --bypasscheck

      5. Verify the execution ends with 2 lines looking like these (the numbers don't matter, they are current date and time): 

        http://your.site/pluginfile.php/1/tool_generator/testplan/0/testplan_202110270018_1121.jmx
        		 
        http://your.site/pluginfile.php/1/tool_generator/users/0/users_202110270018_5725.csv

      6. Using curl, or wget, download the second file (the users one): 

        curl http://127.0.0.1/~stronk7/git/pluginfile.php/1/tool_generator/users/0/users_202110270018_5725.csv -o users.csv

      7. Verify that the downloaded users.csv file:
        1. Has 100 lines
        2. Each line has 2 values separated by a comma.
        3. The first value (user names) looks like "tool_generator_00000x", with every line getting the number incremented by 1.
        4. The second value (user password) is, always, "i_love_moodle" (that was configured some steps above.

      After the patch

      1. Repeat steps 4, 5 and 6 above.
      2. Verify that all the details of 7 above are the same BUT the file now has only 30 lines (instead of the original 100).
      Show
      Before the patch Install a site using PostgreSQL. Edit config.php and add this line anywhere before the last line: $CFG->tool_generator_users_password = 'i_love_moodle'; Let's generate the testing site. Run: php admin/tool/generator/cli/maketestsite.php \ --size=S --fixeddataset --bypasscheck \ --filesizelimit="1000" --quiet Let's generate the testing plan. Run: php admin/tool/generator/cli/maketestplan.php \ --size=S --shortname='testcourse_11' --bypasscheck Verify the execution ends with 2 lines looking like these (the numbers don't matter, they are current date and time): http://your.site/pluginfile.php/1/tool_generator/testplan/0/testplan_202110270018_1121.jmx http://your.site/pluginfile.php/1/tool_generator/users/0/users_202110270018_5725.csv Using curl, or wget, download the second file (the users one): curl http://127.0.0.1/~stronk7/git/pluginfile.php/1/tool_generator/users/0/users_202110270018_5725.csv -o users.csv Verify that the downloaded users.csv file: Has 100 lines Each line has 2 values separated by a comma. The first value (user names) looks like "tool_generator_00000x", with every line getting the number incremented by 1. The second value (user password) is, always, "i_love_moodle" (that was configured some steps above. After the patch Repeat steps 4, 5 and 6 above. Verify that all the details of 7 above are the same BUT the file now has only 30 lines (instead of the original 100).

    Description

      When a jmeter plan is created:

      1) A site of a given size (XS, S, M....) is created. Now we are using S.
      2) A jmeter plan (testplan.jmx) and a users file (users.csv) are created, using the same size used in previous point, aka, S.

      The there is a problem that has been long with us (since 2013!). And it's that, for the S size, 100 users are enrolled in the courses. But the test plan is created with 30 concurrent threads only. In other words, for the S size... the number of users and the number of plan threads don't match (they used to match before 2013).

      And... it's absolutely important, that, not matter if a course has 100, 1000 or 10^6 enrolled... that the users.csv file generated for the plan... only has exactly as many users as the plan concurrency is being set.

      But, right now, plans for 30 threads are using a users.csv with 100 users, so the loops in the plan aren't really looping 5 times over 30 users... but using the 100 users in the cvs. file, so first users run 4 loops, others only 3...

      And all that leads to random stats here and there (because of problems when performing the "warm-up" loop that the test does only for the 1st 30 users in the file)

      To be comparable... 2 plans and the loops must be 100% the same (same users, against same site, with same concurrency...).

      So this patch only aims to do one thing... and it's to ensure that the users.csv file generated contains exactly as many users and the threads are going to need (defined by the plan size).

      In practice, that means that, for our current S size sites... we'll get users.csv files with 30 users (matching the number of threads in the plan) and not the 100 users that are currently being enrolled in courses by default.

      This must be implemented in a BC way (although I don't think that there will be many uses out from core), but old calls to the changed methods, not passing the new param must behave like they were doing before the patch.

      And applied to all supported branches, so comparisons between them can be performed consistently.

      Attachments

        Activity

          People

            stronk7 Eloy Lafuente (stronk7)
            stronk7 Eloy Lafuente (stronk7)
            Carlos Escobedo Carlos Escobedo
            Andrew Lyons Andrew Lyons
            Sujith Haridasan Sujith Haridasan
            Adrian Greeve, David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo, Matteo Scaramuccia, David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:
              8/Nov/21

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 35 minutes
                3h 35m