In Moodle 1.9, it was "Questions per page" with options

Unlimited
1
2
...

I think that is pretty good. The only change I would make is to replace 'Unlimited' with 'Manual page breaks'.

Also the wording for repaginating needs to be considered, and communicating the difference between these two functionalities.

I tried to break this down in a separate docs document. Tim, please comment on that in case I have missed something. http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz

"It is an error to try to select both 'shuffle questions' and 'manual page breaks' at the same time."

Why should this be an error? It seems to me a perfectly legitimate situation to first have two questions on first page and then four questions on the second page, even if you don't know which questions are on which page. Even if it does not seem like a probable use case, I do not see a reason to artificially prohibit it?

"When making this change, all existing quizzes will be analysed for where the page breaks actually are. If the page breaks are uniformly spread, then they will be set to that automatic pagination value. If the page breaks are irregular, they will be set to manual pagination."

This seems dangerous to me, that is, a heuristic that is both dangerous/unexpected to the user and I am not sure why it would be useful. Even if the questions happen to be uniformly paginated at the moment of upgrading to Moodle 2.0 (which is the time I understood the quizzes would be analyzed), you cannot deduct the user's workflow from that - it might be that the the user wants to have custom number of questions per page, and his other quizzes might still do. If the user is accustomed to being capable of managing his paging manually, why would you want to decide against that all of the sudden?

Also this is problematic, since it makes quiz's behaviour inconsistent internally. That is, you are letting a mode affect the behaviour of the application in a way that is not communicated by the setting itself, and if a user is introduced to a quiz with this setting at "one question per page", they will be confused whether or not quizzes have a concept of a page separate of the concept of a page or not.

1. You know from the start you want your quiz to have one-question per page, so you set that on the quiz settings screen when you create the quiz.
2. Then you add 10 questions to the quiz from the question bank.

At this point, you should be finished, you should not have to think about paging again..

Well, I know, there are other important use cases too. But from what you have written, I can see that you have thought about this a lot more deeply than me. I don't have anything to add to what you have written, and I trust your judgement on this. If we can get rid of Mode C, while still supporting the above use-case, then I think that is a worthwhile simplification.

When was this changed? Has the functionality always been like this, regardless of the wording - I'm thinking, since the current functionality contradicts itself, would it be best to start from the way it might have been before and make any desired changes (related to the above use case for example) on that?

See also: http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz#Storing_the_order_when_paginating_is_fixed

If you remove functionality C altogether, that means not supporting the use case the UI currently supports when shuffle questions is off: "In 3, the user needs to set the layout of their questions manually. But if the number of questions to add is big, it is useful to be capable of automatically page the new questions while adding them, requiring feature C. ". Was this functionality supported already in Moodle 1.9? If it was, users may have come to expect it.

Updated my thoughts about the solution. Please comment on all of the above.
http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz#Options_for_a_solution

http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz#Tim.27s_thoughts

I realising what I am proposing might be a lot of work. If you think my suggestion is the right one, but that it is more work that you can do, please feel free to assign it to me.

Also feel free to disagree with me.

shufflequestions on: all manual paging disabled.
shufflequestions off: manual paging. automatic paginating feature disabled. (+ forced pagination when adding questions, or also an option to just add new questions all to the same page?)

I am not claiming for or against whether the real user needs are as you suggest. I do not know (and to be frank, I have my doubts about whether you do). To me, shufflequestions seems like an entirely separate feature and though whether the user wants automatic paginating might in some cases have something to do with shufflequestions, it does not seem reason enough. But even though I think making such decisions on the behalf of the users is unjustified, this is not the main issue.

Although it may seem simpler to make a user interface support less features, it is a different matter to do so by introducing another mode to the UI - in this case, the switch of which is shufflequestions. In this case, you are are actually introducing additional cognitive burden on the user, since s/he not only has to understand what the options are with the current selection, but also how to persuade the application to give him/her the options s/he needs.

The other side of this coin is: how do we communicate this sort of modality in the UI - currently we aren't communicating it, making the UI just behave unpredictably. So although moving the settings closer to where the effects can actually be seen - to the edit page itself - might help, you are not addressing the issue itself.

My concern is making the UI as predictable as possible. We want to make the UI seem to the user like a tool, not like a living thing that (seemingly) changes on a whim.

Let's assume the model you presented (keeping the modality triggered by shufflequestions), and try to make it predictable. This, to me, would mean that the modality in the UI needs be clearly communicated to the user so s/he can make an informed decision about what to select in the UI. At this point, the only way I can think of how to do this would be adding help texts that the user could not miss. Perhaps I am misunderstanding something below, so please correct me if so.

shufflequestions on: Somewhere near the repaginate... button in the edit UI you would need to have an explanation layout editing controls are disabled because shufflequestions is enabled - perhaps similar to the current state of the UI where the same warning is given if "questions per page" is set on the quiz settings page.

shufflequestions off: On the quiz settings page, "Keep the entire quiz paginated with [ v] questions per page" option would be disabled, with a hard-to-miss explanation saying "Since shuffle questions is set to 'No', you cannot use the automatic paginating feature. [The developers of moodle think it's bad for you.] Please set the shuffle questions feature to 'yes' to use the automatic paginating feature. [Also, you really need to set how many questions per page to paginate new questions with.]"

The above is so complicated that unless the user is lucky and the options s/he needs is already enabled, it creates another burden to just try to understand the options and then remember them while making the decision. To me, it just seems a lot simpler to allow all the combinations between the four functionalities (shuffleq's,manual repaginate,automatic repaginate,paginate while adding) by decoupling the modality/dependence on shufflequestions. This way, you do not have to do any explaining on how the user should behave in order to be capable to successfully select some options in the UI, and you avoid the risk of frustrating users by disabling features users might need after all.

To put it otherwise, I really do not see the harm in enabling in all situations the features you already have in the application anyway, since disabling some of them in some cases only makes things more complicated to the user, and not simpler.

(Although: the user might easily miss that "New page" changes in response to changing shuffle questions while the page reload. The only way to have it clearly communicate the relationship is to use ajax to quickly change the form right away so the user should notice the change, and by highlighting the changed element. This is, by the way, the same issue as usability testing issue 11 here: http://docs.moodle.org/en/Development:Quiz_UI_redesign/usability_testing_of_August_2008/Issues#Question_bank_.2F_question_adding_controls_visibility)

However, I still think the model is too complex.

That is, even though I am starting to see the logic that you probably do not want to set any paginating that varies page-by-page for questions, the order of which is shuffled, this is still an artificial limitation. When the teacher notices that as a result of changing shuffle questions, the whole logic of "New page" changes, they might need to try to decipher the logic of the UI - just as we have done during this discussion - to understand what is actually possible, and what is not. Alas, the application easily becomes a nontrivial entity which no longer just serves the needs of the user but becomes something the user tries to persuade to fill their needs.

Shuffle questions: [Yes |v] | New page: [every N questions |v]
or
Shuffle questions: [No |v] | New page: [Never |v] | (Repaginate ...)

The only way to make sure teachers don't have to try to figure out the logic of the UI, in my opinion, is to purge the dependency between the options altogether, and to let teachers use what they find most naturally in the UI (http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz#Option_1). There is no harm since:
a) even if all the combinations of the features are allowed, they can be expressed simply enough
b) none of the features discussed are destructive anyway.

Would it simplify things if we changed what happens if we changed what happens when you start a quiz with Shuffle questions: Yes?

At the moment, what we do is:

1. Start with a layout string like 0,1,0,2,3,0
2. Strip the pages to get an array of question numbers 1,2,3
3. Shuffle this: 3,1,2
4. Repaginate to a fixed page size: 3,1,0,2,0

What if, instead we just changed the last step to

4b. Put back the original page breaks: 0,3,0,1,2,0

That would make the behaviour with shufflequestions on and off more similar. Is that a good idea? and is it enough to avoid any modality?

Hm, I am not sure in which way you are not sure about what I am saying. My point is simply that each setting on the settings page should do exactly what is states, and nothing else. Just to make sure I added a simple table (openoffice.org mediawiki export rocks :

http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz#Background

And yes, that decouples part of the modality within the settings. This goes a long way to remove the metamodality. After this all you are missing is a consistently communicated way of handling questions that are newly added to the quiz.

That is, still missing from the teacher is understanding of what will happen when s/he adds, say fifty questions at once from the question bank, to a quiz that already has five pages of questions, but pages 3-5 have more questions per page than the first two introductory pages.

Some further background for your reading pleasure...

The 'normal' challenge with such settings as shuffle questions and permanent paging is that they are modes. The fact that they are permanent circumstances that affect the quiz is their nature: the challenge comes when trying to make sure users can see the connection between the behaviour they see and the settings, so that even though the program is doing something for them, they have the power to tell the program to no longer do so.

In order to be capable of making this decision, teachers must sufficiently understand the needs of his/her students and the pedagogical parameters he has set for the course. These together define what kind of an exam makes sense for the course in question. Then with all this in mind, the teacher goes to the Quiz module to determine what is needed. S/he looks at the current state of the quiz and more or less consciously tries to determine ways to purge the cognitive dissonance between what is the current state of his/her quiz and what s/he wants the state to be. If he can go through the settings screen one setting at a time, comparing each setting to the ideal scene s/he has in his/her mind, s/he does not have to remember any extra information, as s/he can forget about the previous setting when she moves on to the next one.

Shuffle questions: on/off
Paginate: on/on for only new ones/off

But if the modality has a (meta-)modality, dependency between different options (and if the user understands this), the user has to interrupt the process of going through the settings one by one to remember two at a time, in order to understand a hierarchy, to determine which leaf of the hierarchy s/he wants to go to and which combination of options is required for this:

Shuffle questions: off
-- paginate: on: result 1
-- paginate: off: result 2
Shuffle questions: on
-- paginate: on: result 3
-- paginate: off: result 4

Peaceful christmas, hope you get to rest. I have gotten today some christmas packages from Finland and am quite happy with my current existence lacking much events.

It seems to me that the current idea would be to decouple shuffling and paginating settings, that is:

• If manual paging is on, regardless of shuffle paging, quiz obeys that manual paging as you suggest above,
• if automatic paging is on, that is obeyed regardless of shuffle paging

But the decision about what to do with the setting of paginating questions when adding them is unclear. This is a more marginal use case it seems, anyway. One option would be to not make it a mode at all but an option to choose when actually adding the question, but it is hard to see where such an option would go.

On the quiz settings form:

1. Change the shuffle question field to
Question order [ As shown on the edit screen / Shuffled randomly ]

2. Change the questions per page field to
New page every [ All questions on one page / N questions ]

3. and - if we are editing an existing quiz, rather than creating a new quiz - immediately next to that dropdown add a checkbox
([ ] Repaginate now)
Using JavaScript, we will ensure that

• if Question order is random, then this check-box will be disabled.
• if question order is not random, then, the check-box will be enabled, and whenever New page every ... is changed, we will automatically check this checkbox.

On the quiz editing screen:

4. With question order not random, this use the interface as it is now, when set to manual page breaks.
With random order, we will change the interface as follows:

• disable the repaginate dropdown. (Disable, or remove completely? What do you think?);
• always display the list of questions with N questions per page;
• remove the add question buttons from everywhere except the last page of the quiz; and
• don't display question numbers.
  (We will still show the up and down arrows. It may help the teacher to check they have all the right questions if they can reorder them. However, we need to fix it so that when you click the down arrow on the last question on the page, it reorder the questions, rather than just moving the question to the top of the next page.)

5. The status line will look like one of:

• Question order: Random, N questions per page (perhaps use the random question dice icon next to the word Random?)
• Question order: As shown, New page every N questions.

I think this is good because it

• minimises the number of Mode we need in the UI;
• keeps the current (Moodle 1.9) functionality that I like; and
• is easy to understand.

I thought I would have implemented this by now, but I keep not getting around to writing any code, so who knows. I am generally trying to get all get the quiz editing and question bank changes finished as the next thing on my todo list, but I will leave this one for a bit, or just try to prototype it and attach a patch.

This is getting too abstract. I will now try to draw what you are suggesting, in Balsamiq. Although it keeps telling me to buy it every five minutes.

Please, please, just do what's described in point 12. here:
http://www.joelonsoftware.com/articles/fog0000000043.html
Cos' this is so intricate in any case that it really needs this little bit of attention. It'll be no time at all .

http://www.balsamiq.com/products/mockups/jira

The whole point of your reasoning seems to be disabling tmber of Mode we need in the UI;
he upper right corner of the table at http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz#Background, "Keep as many questions on pages as edit view suggests, but shuffle order". Although this might be a rare use case, it seems to me that adding this modality and disabling this choice confuses more than it helps.

Pardon me, really, if I have forgotten something already and am not seeing this clearly after the break in trying to see all aspects of this.

• minimises the number of Mode we need in the UI;

It does not minimise the number of modes. It seems to me it is still what you have been saying all the time, and it has not only the modes of randomizing and automatic paging, but also the meta-mode of coupling those two features with each other. This is one mode too many in my opinion.

To answer your question, do not drop off buttons in response to mode changes, but disable them, to increase the predictability. The user still does not know /why/ the button is disabled (that is the exact problem of modes) but at least s/he won't be left wondering whether s/he is "seeing things".

What do you mean by the things you like about 1.9?

There are people I could grab for a hallway usability test at the office, but I don't know what I could say to them to get them to focus on the option we are talking about, without saying to much and influencing the results of the test. Any advice?

And i have just re-read all the comments on this bug, and the wiki page http://docs.moodle.org/en/Development:Quiz_Usability_portal/Different_paginating_features_in_quiz.

I am guessing, but I am quite sure that by far the most common case I Shuffle questions: No. (Actually, I might be able to get some data about that.)

With Shuffle questions: No, and manual page breaks. Then the current interface is very nice. You can easily see the current layout of the quiz, and you can easily edit it. Since there is an easily accessible 'Repaginate ...' action, I don't see any value in a 'keep my quiz paginated with N questions per page' mode. That is, assuming that we keep what I like about Moodle 1.9:

Here is a user story:
1. Mary creates a new quiz.
2. She adds 5 questions (it does not matter how).
3. She repaginates the quiz with 2 questions per page.
4. She selects 5 more questions in the question bank, and clicks 'Add to quiz'.
In Moodle 1.9, those 5 questions would be added in a way that keeps the quiz paginated with 2 questions per page.

However, that is not a 2 questions per page mode, Because you could vary the story by inserting the step
3a. After repaginating, she adds a description to the start of page 1.
Then, before step 4, the layout would be Page 1, 3 items. Page 2, 2 items, Page 3, 1 items;
and, after step 4, the layout would be Page 1, 3 items. Page 2, 2 items, Page 3, 2 items Page 4, 2 items Page 5, 2 items.

That is, it does not overwrite your existing manual edits to the quiz layout, but it still tries to follow your intentions about how many questions per page when adding new questions. That probably saves time for teachers, but if it is not what they want, they can immediately see what has happened and edit the layout.

Hmm. One option that is not easy in the current user interface is to remove a page break. You have to move all the questions to the other page, then delete the empty page. Is it work adding a [Merge pages] button next to the [Add page here] button? (This is really a separate issue.)

So, I think that with Shuffle questions: No, which is the common case, we don't need modality in the UI.

Shuffle questions: Yes is less common. We need to allow it, but let us not mess up the common case while doing so.

With Shuffle questions: Yes, we must make it clear on the quiz editing screen that is the separate issue MDL-17454 that you filed, and I agree with.

The real problem is two inter-linked settings on the quiz editing page. Ah, so perhaps the solution is simple, have a single option:

Question order:

• As shown on the edit screen
• Randomly shuffled all on one page
• Randomly shuffled 1 question per page
• Randomly shuffled 2 questions per page
• Randomly shuffled 3 questions per page

I did this as a patch, rather than a sketch in Balsamiq, becuase that seemed easier to me, and I wanted to make sure that all the details worked out.

Can you try out a patch easily, or do you want me to attach screenshots?

I have not actually changed the quiz settings form. I figure that is quite easy to do if we agree on how the editing UI looks in the two modes. The two modes, to my mind, being Shuffle on and off.

I would like to add the repaginate button to the edit tab. Perhaps it could go in the same row as 'Maximum grade:'

And I still quite like the idea of Merge pages buttons between each pair of pages.

The reason the status area had a yellow background is that it is quite a convention to communicate non-critical FYI-type data to users with that color. Now it seems it isn't really distinguishable as a separate element with the grey background. Or is it due to the buggy patching I tried?

It does not seem like we're talking on the same planet. Gonna try now to decipher all the different issues here, again.

• ask users to do a quiz in a manner they would usually make a quiz in the way they use quizzes (assuming you can find someone who uses quizzes otherwise), and see if they can achieve that naturally. Risk: it might take time to find enough variation between users in the way they use paging. I still think this is the lightest way of doing it with little additional bias.
• think of different tasks that make users use different kinds of pagings. This risks affecting the user to do something they would not naturally do and might introduce major bias in the way the tasks are built.

Anyway, the main point is just to listen, to see users use the functionality, to try to not say too much that directs their behaviour. Even the above details are not that important in the end.

I wanted to ask you about the yellow convention, because I am not so sure I have seen this convention elsewhere. The only one I can think of is the firefox pop-up blocker, but that is more an event notification than a status bar. However, this may just be my lack of knowledge, and anyway that was one of the more speculative changes and is easy to undo, of course.

I have been wondering whether it would be better for us to get together on a Skype chat to discuss this, rather than trying to do it Asynchronously in bug reports. I can see you are online now, but I am tired, so I would rather do this some other time. Can you suggest some times that would be convenient to you?

Well, if you insist that people should not be capable of having automatic paging without shuffling questions, then what you are suggesting indeed removes the metamodality and simplifies the choice.

Then you just have to take responsibility for removing the option to to have automatic paging enabled when shuffling questions are turned off. Simplicity is indeed a useful aim, but it is still a risk to remove the option to do something that was there in a previous version. Though I admit that the loss seems minor, and not having two different things that essentially do the same thing (the mode for paging and the explicit command) seems beneficial in terms of simplicity.

Although I guess what you propose works for most use cases and I admire its simplicity, I still think it's simpler, just keeping both options and decoupling them so that they are not interdependent.

This seems to avoid the risk of removing options, (and, like your solution, the complexity of metamodality). There is one option more than in what you suggest, but if both are worded so that they reflect the actual behaviour, it seems it is still the simplest possible scenario, and easiest to understand: while matching his needs with the options available, the user only has to consider one option at a time.

That is: Even though your suggestion seems like just one selection, users still need to try to match the multiple outcomes of that one selection, with the goal they have in mind, when trying to understand the different options.

If you hallway tested just those two different alternatives that we now consider the simplest ones, we might see which one is in practice easier for the users to process. You simply give users quiz usage scenarios or even explain what you want the outcome to be, and see what happens when they try to make a quiz with either of the UIs. As said, even if you are biased and tell users too much, don't worry, you will still probably learn something about how they perceive the UI, if you get them to think out loud (if you don't, you can always as what they are thinking).

What I am trying to say that it is more valuable than worrying too much about being biased, to just get out there and see what happens, both to see how the test itself works and to see how the UI works.

Also, if we choose your design, if a quiz has automatic paging selected when migrating to moodle 2.0, you should repaginate all the quizzes with that number of questionsperpage, and then turn that setting off for the quizzes that have had that treatment.

http://moodle.org/mod/forum/discuss.php?d=118371

The way I am proposing it should work is the way it has always worked. I am just trying to make it understandable.

And I agree with you about "remove page break".

But what people have perceived to be supposed to work is the reality that non-shuffled questions can be paged, too. If it doesn't behave that way, the assumption is I guess that it is broken, and not that it is a built-in feature.