Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-47271

Add system status report including tasks with cli status script

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      1) Visit the new report page: /report/status/index.php

      On a properly running site confirm everything is green

      2) Run the cli and confirm it is OK

      php admin/cli/checks.php

      OK: All 'status' checks ok

      Confirm that the shell error code was 0:

      $ echo $?
      0

      3) Run it in verbose mode

       

      Lets break everything

      4) Break environment check, edit admin/environment.xml, find the moodle version 3.9 php version required (line 2998) and change it to php version 17, confirm it is an error:

      Now undo this.

      5) Simulate a pending upgrade, bump the version.php and confirm the error:

      Also confirm that the shell error code was 2 which is an NRPE critical error:

      $ echo $?
      2

      6) Turn cron off. Now simulate it having never run and confirm the error:

      php admin/cli/cfg.php --component=tool_task --name=lastcronstart --unset

      7) Simulate it having run 40 hours ago and confirm an error:

       

      $ php admin/cli/cfg.php --component=tool_task --name=lastcronstart --set=`date -d '40 hour ago' +'%s'`
      $ php admin/cli/checks.php
       CRITICAL: Cron running (tool_task_cronrunning)
          Status | Check
       ----------+--------------------------------------------------------------------
       CRITICAL | Cron running (tool_task_cronrunning) | The admin/cron.php script has not been run for 1 day 16 | hours and should run every 1 min.
      

       

      8) Simulate it having run 1 hour ago and confirm an warning

       

      $ php admin/cli/cfg.php --component=tool_task --name=lastcronstart --set=`date -d '1 hour ago' +'%s'`
      $ php admin/cli/checks.php
      WARNING: Cron running (tool_task_cronrunning)
         Status | Check                                                       
      ----------+--------------------------------------------------------------------
        WARNING | Cron running (tool_task_cronrunning)                        
                |     The admin/cron.php script has not been run for 1 hour and
                |     should run every 1 min.
      

       Also confirm that the shell error code was 1 which is an NRPE warning:

      $ echo $?
      1

       9) Make it good again, and confirm it is ok

       

      $ php admin/cli/cfg.php --component=tool_task --name=lastcronstart --set=`date -d '1 sec ago' +'%s'`
      $ php admin/cli/checks.php
      OK: All 'status' checks ok
      
      

       10) Now simulate gaps between cron 

       

      $ php admin/cli/cfg.php --component=tool_task --name=lastcroninterval --set=600
      $ php admin/cli/checks.php 
      WARNING: Cron running (tool_task_cronrunning)
         Status | Check                                                       
      ----------+--------------------------------------------------------------------
        WARNING | Cron running (tool_task_cronrunning)                        
                |     There was 10 mins between the last two runs of the cron
                |     maintenance script and it should run every 1 min. We
                |     recommend configuring it to run more frequently.
      
      

       10) Install https://github.com/catalyst/moodle-tool_testtasks

      11) Queue up some tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php

      12) Confirm these appear as INFO:

      $ php admin/cli/checks.php -v --filter=adhoc

       

      13) Drop into sql and fudge the dates to be slightly old and confirm you get a warning:

      update mdl_task_adhoc set nextruntime = round(extract(epoch from now()) - 60 * 10);

       

      14) Fudge them to be older and confirm you get a warning

      update mdl_task_adhoc set nextruntime = round(extract(epoch from now()) - 60 * 60);

      15) Update the warning and error threshold and retest:

      php admin/cli/cfg.php --name=adhoctaskagewarn --set=500

      php admin/cli/cfg.php --name=adhoctaskageerror --set=600

       

       

      Show
      1) Visit the new report page: /report/status/index.php On a properly running site confirm everything is green 2) Run the cli and confirm it is OK php admin/cli/checks.php OK: All 'status' checks ok Confirm that the shell error code was 0: $ echo $? 0 3) Run it in verbose mode   Lets break everything 4) Break environment check, edit admin/environment.xml, find the moodle version 3.9 php version required (line 2998) and change it to php version 17, confirm it is an error: Now undo this. 5) Simulate a pending upgrade, bump the version.php and confirm the error: Also confirm that the shell error code was 2 which is an NRPE critical error: $ echo $? 2 6) Turn cron off. Now simulate it having never run and confirm the error: php admin/cli/cfg.php --component=tool_task --name=lastcronstart --unset 7) Simulate it having run 40 hours ago and confirm an error:   $ php admin/cli/cfg.php --component=tool_task --name=lastcronstart --set=`date -d '40 hour ago' + '%s' ` $ php admin/cli/checks.php CRITICAL: Cron running (tool_task_cronrunning) Status | Check ----------+-------------------------------------------------------------------- CRITICAL | Cron running (tool_task_cronrunning) | The admin/cron.php script has not been run for 1 day 16 | hours and should run every 1 min.   8) Simulate it having run 1 hour ago and confirm an warning   $ php admin/cli/cfg.php --component=tool_task --name=lastcronstart --set=`date -d '1 hour ago' +'%s'` $ php admin/cli/checks.php WARNING: Cron running (tool_task_cronrunning) Status | Check ----------+-------------------------------------------------------------------- WARNING | Cron running (tool_task_cronrunning) | The admin/cron.php script has not been run for 1 hour and | should run every 1 min.  Also confirm that the shell error code was 1 which is an NRPE warning: $ echo $? 1  9) Make it good again, and confirm it is ok   $ php admin/cli/cfg.php --component=tool_task --name=lastcronstart --set=`date -d '1 sec ago' +'%s'` $ php admin/cli/checks.php OK: All 'status' checks ok  10) Now simulate gaps between cron    $ php admin/cli/cfg.php --component=tool_task --name=lastcroninterval --set= 600 $ php admin/cli/checks.php WARNING: Cron running (tool_task_cronrunning) Status | Check ----------+-------------------------------------------------------------------- WARNING | Cron running (tool_task_cronrunning) | There was 10 mins between the last two runs of the cron | maintenance script and it should run every 1 min. We | recommend configuring it to run more frequently.  10) Install  https://github.com/catalyst/moodle-tool_testtasks 11) Queue up some tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php 12) Confirm these appear as INFO: $ php admin/cli/checks.php -v --filter=adhoc   13) Drop into sql and fudge the dates to be slightly old and confirm you get a warning: update mdl_task_adhoc set nextruntime = round(extract(epoch from now()) - 60 * 10);   14) Fudge them to be older and confirm you get a warning update mdl_task_adhoc set nextruntime = round(extract(epoch from now()) - 60 * 60); 15) Update the warning and error threshold and retest: php admin/cli/cfg.php --name=adhoctaskagewarn --set=500 php admin/cli/cfg.php --name=adhoctaskageerror --set=600    
    • Affected Branches:
      MOODLE_27_STABLE, MOODLE_33_STABLE, MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_39_STABLE
    • Pull Master Branch:
      MDL-47271-status-api-fixes

      Description

      For the purposes of monitoring, it is good to have a CLI script which can be run to verify the status of scheduled tasks.

      It could report problems such as:

      • Tasks which have been scheduled to run but have not been run recently
      • Tasks with faildelay present

      Taking parameters for tolerances and reporting 0 for no problems.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              brendanheywood Brendan Heywood
              Reporter:
              poltawski Dan Poltawski
              Peer reviewer:
              Peter Burnett
              Integrator:
              Andrew Nicols
              Tester:
              Janelle Barcega
              Participants:
              Component watchers:
              Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona), Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona)
              Votes:
              15 Vote for this issue
              Watchers:
              22 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                15/Jun/20

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour, 36 minutes
                  1h 36m