Moodle
  1. Moodle
  2. MDL-36858

SCORM - attemptsgrading confusing

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.6, 2.3.3, 2.4
    • Fix Version/s: 2.2.7, 2.3.4
    • Component/s: SCORM
    • Labels:
    • Testing Instructions:
      Hide

      install this SCORM package:
      http://scorm.com/wp-content/assets/golf_examples/PIFS/RuntimeBasicCalls_SCORM12.zip
      make sure the setting "display attempt status" is set to yes.
      enter the scorm and go to the quiz at the end and enter a bunch of wrong answers to the quiz - then hit the "exit" button at the bottom (unfortuantely another bug with object/iframe means this won't do anything obvious) then navigate back to the SCORM entry page.
      The SCORM entry page will not show a checkbox to "start a new attempt"
      enter the SCORM again and answer most of the questions correctly (I found if I answered all but 1 wrong the SCORM was set to complete)
      hit the exit button, navigate back to entry page and the "start new attempt" checkbox should show.

      NOTE:SCORM packages internally have different scores that trigger a completion status of complete - I found a score of 33% in this package set it as incomplete but a score of 94% it set as "passed/complete" - I'm not sure what threshold this particular package sets for it's "pass" score.

      hopefully those testing instructions are ok! - ping me on jabber if you need help.

      Show
      install this SCORM package: http://scorm.com/wp-content/assets/golf_examples/PIFS/RuntimeBasicCalls_SCORM12.zip make sure the setting "display attempt status" is set to yes. enter the scorm and go to the quiz at the end and enter a bunch of wrong answers to the quiz - then hit the "exit" button at the bottom (unfortuantely another bug with object/iframe means this won't do anything obvious) then navigate back to the SCORM entry page. The SCORM entry page will not show a checkbox to "start a new attempt" enter the SCORM again and answer most of the questions correctly (I found if I answered all but 1 wrong the SCORM was set to complete) hit the exit button, navigate back to entry page and the "start new attempt" checkbox should show. NOTE:SCORM packages internally have different scores that trigger a completion status of complete - I found a score of 33% in this package set it as incomplete but a score of 94% it set as "passed/complete" - I'm not sure what threshold this particular package sets for it's "pass" score. hopefully those testing instructions are ok! - ping me on jabber if you need help.
    • Affected Branches:
      MOODLE_22_STABLE, MOODLE_23_STABLE, MOODLE_24_STABLE
    • Fixed Branches:
      MOODLE_22_STABLE, MOODLE_23_STABLE
    • Pull Master Branch:
      master_MDL-36858
    • Rank:
      46385

      Description

      When a SCORM package sets the lesson_status as "failed" a new attempt is allowed but the grading method "last completed attempt" doesn't make it completely clear that 'failed' attempts aren't included in the calculation - this is behaviour that has been there since 1.9 and it's possible we should add another grading method that does include failed attempts but no "real" user has asked for this behaviour so we need to update the grading help to make it more explicit.

        Issue Links

          Activity

          Hide
          Dan Marsden added a comment -

          interesting - that may actually be a bug with the ability to generate a new attempt - did you use a cached version of the entry page to generate the 4th attempt?

          Show
          Dan Marsden added a comment - interesting - that may actually be a bug with the ability to generate a new attempt - did you use a cached version of the entry page to generate the 4th attempt?
          Hide
          Dan Marsden added a comment -

          eg - attempt 2 may in fact be the last fully completed attempt and attempt 3 was generated correctly but then instead of refreshing the view page but using back in browser viewing the view page allowed you to generate a new attempt?

          Show
          Dan Marsden added a comment - eg - attempt 2 may in fact be the last fully completed attempt and attempt 3 was generated correctly but then instead of refreshing the view page but using back in browser viewing the view page allowed you to generate a new attempt?
          Hide
          Adrian Greeve added a comment -

          No, I purged the cache and made a complete attempt(visited all pages and answered all questions).

          Show
          Adrian Greeve added a comment - No, I purged the cache and made a complete attempt(visited all pages and answered all questions).
          Hide
          Dan Marsden added a comment -

          weird. I hate grading in SCORM. will try to take a look tonight.

          Show
          Dan Marsden added a comment - weird. I hate grading in SCORM. will try to take a look tonight.
          Hide
          Dan Marsden added a comment -

          could still be a problem with the code that allows you to generate a new attempt rather than a problem with the grading but I'll see if I can reproduce.

          Show
          Dan Marsden added a comment - could still be a problem with the code that allows you to generate a new attempt rather than a problem with the grading but I'll see if I can reproduce.
          Hide
          Dan Marsden added a comment -

          yeah - problem with the check to allow a new attempt to be generated - SCORM is only supposed to show that box when lesson_status is complete or passed - at the moment it's allowing every re-entry to generate a new attempt - bug with the new scorm_get_toc function added in master.

          Show
          Dan Marsden added a comment - yeah - problem with the check to allow a new attempt to be generated - SCORM is only supposed to show that box when lesson_status is complete or passed - at the moment it's allowing every re-entry to generate a new attempt - bug with the new scorm_get_toc function added in master.
          Hide
          Dan Marsden added a comment -

          found it - not a regression with master - it's a bug in stable releases too - that particular package you're using uses passed/failed as the status not complete/incomplete - and the check for incomplete was only looking at "incomplete" value and needed to check "failed" value as well.

          so - grading is happening correctly - it's showing the last completed attempt - but SCORM code isn't preventing a new attempt from being created correctly.

          Show
          Dan Marsden added a comment - found it - not a regression with master - it's a bug in stable releases too - that particular package you're using uses passed/failed as the status not complete/incomplete - and the check for incomplete was only looking at "incomplete" value and needed to check "failed" value as well. so - grading is happening correctly - it's showing the last completed attempt - but SCORM code isn't preventing a new attempt from being created correctly.
          Hide
          Dan Marsden added a comment -

          bouncing up for integration.

          NOTE TO INTEGRATOR: Master only please - this does affect Stable branches so I may decide to backport it if I get requests from real users asking for it.

          not backporting to stable at this stage as although it corrects SCORM behaviour as per the ADL spec there may be people relying on the incorrect behaviour and a change in a stable branch could be unexpected.

          Show
          Dan Marsden added a comment - bouncing up for integration. NOTE TO INTEGRATOR: Master only please - this does affect Stable branches so I may decide to backport it if I get requests from real users asking for it. not backporting to stable at this stage as although it corrects SCORM behaviour as per the ADL spec there may be people relying on the incorrect behaviour and a change in a stable branch could be unexpected.
          Hide
          Ankit Agarwal added a comment -

          Hi Dan,
          Me and Adrian did some more study on the situation in here. We both agree that "failed" shouldn't be considered as "incomplete" . I am not sure if we are missing something obvious with the scorm design here, but "failed" should be considered as "complete" but "failed" imho.
          It might be a better idea to update scorm_get_last_completed_attempt() to pass records with value 'completed', 'passed' or 'failed'. That also fixes and passes the issue reported in the QA.

          Thanks

          Show
          Ankit Agarwal added a comment - Hi Dan, Me and Adrian did some more study on the situation in here. We both agree that "failed" shouldn't be considered as "incomplete" . I am not sure if we are missing something obvious with the scorm design here, but "failed" should be considered as "complete" but "failed" imho. It might be a better idea to update scorm_get_last_completed_attempt() to pass records with value 'completed', 'passed' or 'failed'. That also fixes and passes the issue reported in the QA. Thanks
          Hide
          Dan Marsden added a comment -

          grr - this patch is wrong.... failed does mean completion in 1.2 spec - I really hate SCORM grading....

          Show
          Dan Marsden added a comment - grr - this patch is wrong.... failed does mean completion in 1.2 spec - I really hate SCORM grading....
          Hide
          Ankit Agarwal added a comment -

          Quoting the extract from scorm 1.2 spec for reference:-
          Format: A set vocabulary phrase. Six possible vocabulary values:

          1. "passed: Necessary number of objectives in the SCO were mastered, or the
            necessary score was achieved. Student is considered to have completed the
            SCO and passed.
          2. "completed": The objective may or may not be passed, but objective was
            experienced by the student. The student is considered to have completed the
            objective. For instance, passing may depend on a certain score known to the
            LMS system. The SCO knows the raw score but not whether that raw score
            was high enough to pass the objective.
          3. "failed": The objective was not passed. The objective may or may not have
            been completed by the student. *The student is considered to have experienced
            the material associated with the objective and failed.*
          4. "incomplete": The material associated with the objective was begun but not
            finished.
          5. "browsed": The student launched the SCO containing the material associated
            with the objective with an LMS mode of Browse on the initial attempt.
          6. "not attempted": Incomplete implies that the student made an attempt to
            perform the SCO, but for some reason was unable to finish it. Not attempted
            means that the student did not even begin the SCO containing the objective.
            Maybe he just read the table of contents, or SCO abstract and decided he was
            not ready. Any algorithm within the SCO may be used to determine when the
            objective moves from "not attempted" to "incomplete".
          Show
          Ankit Agarwal added a comment - Quoting the extract from scorm 1.2 spec for reference:- Format: A set vocabulary phrase. Six possible vocabulary values: "passed: Necessary number of objectives in the SCO were mastered, or the necessary score was achieved. Student is considered to have completed the SCO and passed. "completed": The objective may or may not be passed, but objective was experienced by the student. The student is considered to have completed the objective. For instance, passing may depend on a certain score known to the LMS system. The SCO knows the raw score but not whether that raw score was high enough to pass the objective. "failed": The objective was not passed. The objective may or may not have been completed by the student. *The student is considered to have experienced the material associated with the objective and failed.* "incomplete": The material associated with the objective was begun but not finished. "browsed": The student launched the SCO containing the material associated with the objective with an LMS mode of Browse on the initial attempt. "not attempted": Incomplete implies that the student made an attempt to perform the SCO, but for some reason was unable to finish it. Not attempted means that the student did not even begin the SCO containing the objective. Maybe he just read the table of contents, or SCO abstract and decided he was not ready. Any algorithm within the SCO may be used to determine when the objective moves from "not attempted" to "incomplete".
          Hide
          Dan Marsden added a comment -

          ok - updated the "handling of multiple attempts" in moodledocs 20/22/23/24 to be more explicit around failed attempts and have an updated help string in the commit above that is more explicit on how the grade is calculated.

          Show
          Dan Marsden added a comment - ok - updated the "handling of multiple attempts" in moodledocs 20/22/23/24 to be more explicit around failed attempts and have an updated help string in the commit above that is more explicit on how the grade is calculated.
          Hide
          Helen Foster added a comment -

          Dan, thanks for improving the documentation http://docs.moodle.org/24/en/SCORM_FAQ#Handling_of_Multiple_Attempts and also the help string. It all looks good to me.

          Show
          Helen Foster added a comment - Dan, thanks for improving the documentation http://docs.moodle.org/24/en/SCORM_FAQ#Handling_of_Multiple_Attempts and also the help string. It all looks good to me.
          Hide
          Aparup Banerjee added a comment -

          hm this seems back portable to me - y/n?

          Show
          Aparup Banerjee added a comment - hm this seems back portable to me - y/n?
          Hide
          Adrian Greeve added a comment -

          I don't really agree with just changing the documentation to suit the behaviour.

          Point 2 of the 1.2 spec, in my opinion, says that the objective is "completed" regardless of whether it was passed or not passed.

          If we are going to follow with the idea that a fail doesn't count as an attempt, then the behaviour behind the grading method "First attempt" Should also be changed to follow the same checks. Currently with first attempt grading, failed attempts are counted as "completed" and the first grade, regardless of whether is was a pass or fail is the grade reported.

          I understand that no one has reported this as a problem, and that users may have made an allowance for the current behaviour, and changing it may cause problems. At the very least I think that an issue should be created to include an improvement that follows the specifications properly.

          Show
          Adrian Greeve added a comment - I don't really agree with just changing the documentation to suit the behaviour. Point 2 of the 1.2 spec, in my opinion, says that the objective is "completed" regardless of whether it was passed or not passed. If we are going to follow with the idea that a fail doesn't count as an attempt, then the behaviour behind the grading method "First attempt" Should also be changed to follow the same checks. Currently with first attempt grading, failed attempts are counted as "completed" and the first grade, regardless of whether is was a pass or fail is the grade reported. I understand that no one has reported this as a problem, and that users may have made an allowance for the current behaviour, and changing it may cause problems. At the very least I think that an issue should be created to include an improvement that follows the specifications properly.
          Hide
          Dan Marsden added a comment -

          Aparup - happy for you to backport - I wasn't sure if we had to do anything special for AMOS when backporting lang string updates.

          Adrian - you're welcome to create an issue if you want to work on it yourself and add a new grading method to work the way you think it should. I'm not going to spend any volunteer time on improvements that havn't been requested from real users - I have plenty of other bugs to keep me busy. you get paid for this stuff - so feel free.

          Show
          Dan Marsden added a comment - Aparup - happy for you to backport - I wasn't sure if we had to do anything special for AMOS when backporting lang string updates. Adrian - you're welcome to create an issue if you want to work on it yourself and add a new grading method to work the way you think it should. I'm not going to spend any volunteer time on improvements that havn't been requested from real users - I have plenty of other bugs to keep me busy. you get paid for this stuff - so feel free.
          Hide
          Aparup Banerjee added a comment -

          The purpose that i see mainly is that this allows learners to maximise their experience and playing/gaming with the lesson.
          Perhaps grading methods 'First attempt' and 'Last attempt' could be renamed to 'First passing attempt' and 'Last passing attempt' to make this clearer?

          Show
          Aparup Banerjee added a comment - The purpose that i see mainly is that this allows learners to maximise their experience and playing/gaming with the lesson. Perhaps grading methods 'First attempt' and 'Last attempt' could be renamed to 'First passing attempt' and 'Last passing attempt' to make this clearer?
          Hide
          Dan Marsden added a comment - - edited

          first attempt and last attempt should work as expected - they don't take into account the SCORM lesson_status -
          the grading method "last completed attempt" looks for an attempt with a passed or completed SCORM lesson_status value but ignores "failed" attempts. hope that makes more sense with the updates to docs.

          Show
          Dan Marsden added a comment - - edited first attempt and last attempt should work as expected - they don't take into account the SCORM lesson_status - the grading method "last completed attempt" looks for an attempt with a passed or completed SCORM lesson_status value but ignores "failed" attempts. hope that makes more sense with the updates to docs.
          Hide
          Michael de Raadt added a comment -

          From my experience, students will make reattempts until the pass. Any attempts beyond that would be for revision only. So I'm happy with the current behaviour if the label is updated to explain it. If it includes both "passed" and "completed" attempts, perhaps the label should be "Last successful attempt".

          Show
          Michael de Raadt added a comment - From my experience, students will make reattempts until the pass. Any attempts beyond that would be for revision only. So I'm happy with the current behaviour if the label is updated to explain it. If it includes both "passed" and "completed" attempts, perhaps the label should be "Last successful attempt".
          Hide
          Aparup Banerjee added a comment -

          Thanks Dan,
          I've spoken to Adrian and we've agreed to integrate this to close this issue.

          we sort of agreed not enough user confusion has occurred and this was a lot of developer confusion.

          Show
          Aparup Banerjee added a comment - Thanks Dan, I've spoken to Adrian and we've agreed to integrate this to close this issue. we sort of agreed not enough user confusion has occurred and this was a lot of developer confusion.
          Hide
          Dan Marsden added a comment -

          another pain is that SCORM uses the word "completed" in 2 different places with a different meaning.

          "completed" is a value that can be given to lesson_status

          "The objective may or may not have been completed by the student." - meaning that a new attempt can be generated.

          we could change the existing grading method name to "last completed/passed lesson_status" - but that seems too long and obscure to me and I think the clarification in the docs is sufficient.

          Show
          Dan Marsden added a comment - another pain is that SCORM uses the word "completed" in 2 different places with a different meaning. "completed" is a value that can be given to lesson_status "The objective may or may not have been completed by the student." - meaning that a new attempt can be generated. we could change the existing grading method name to "last completed/passed lesson_status" - but that seems too long and obscure to me and I think the clarification in the docs is sufficient.
          Hide
          Adrian Greeve added a comment -

          I've been convinced. Sorry to hold everyone up and special thanks to Dan for tolerating my objections.

          Show
          Adrian Greeve added a comment - I've been convinced. Sorry to hold everyone up and special thanks to Dan for tolerating my objections.
          Hide
          Aparup Banerjee added a comment -

          Thanks all, this has been integrated now into 22, 23 and master.

          Show
          Aparup Banerjee added a comment - Thanks all, this has been integrated now into 22, 23 and master.
          Hide
          Aparup Banerjee added a comment -

          passing - string change.

          Show
          Aparup Banerjee added a comment - passing - string change.
          Hide
          Dan Marsden added a comment -

          heh - no worries - it takes a special kind of person to tolerate using SCORM in the first place - thanks for the help testing - it was a useful discussion!

          Show
          Dan Marsden added a comment - heh - no worries - it takes a special kind of person to tolerate using SCORM in the first place - thanks for the help testing - it was a useful discussion!
          Hide
          Aparup Banerjee added a comment -

          oh and about AMOS and strings Dan, thankfully theres no issues at all on simple string updates.

          ref: http://docs.moodle.org/dev/Commit_cheat_sheet#Include_AMOS_script_in_the_commit_if_needed
          "If you change the identifier of a string or split a string into two forks, provide a script for AMOS in the commit message"

          Show
          Aparup Banerjee added a comment - oh and about AMOS and strings Dan, thankfully theres no issues at all on simple string updates. ref: http://docs.moodle.org/dev/Commit_cheat_sheet#Include_AMOS_script_in_the_commit_if_needed "If you change the identifier of a string or split a string into two forks, provide a script for AMOS in the commit message"
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Y E S !

          Closing as fixed, many thanks!

          Show
          Eloy Lafuente (stronk7) added a comment - Y E S ! Closing as fixed, many thanks!

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: