Add an option to show the correct answer with a reveal button.

[drgrice1]

Technically this does not add an option, but changes how the existing showCorrectAnswers option in the PG environment works. Previously it was purely boolean. Now it is numeric.

If it is zero, then correct answers are not shown (as before).

If it is 1, then correct answers are shown but hidden, and a "Reveal" button is shown at first. If that button is clicked, then the answer is shown.

If set to 2, then correct answers are shown immediately.

This is used for new features in a corresponding webwork2 pull request (openwebwork/webwork2#2275).

[Alex-Jordan]

One thing I see about this is that when a user first comes to a problem's page, it's not clear how to see the correct answers now. The user needs to know to use the Check/Submit button first, so that the general feedback buttons appear, and then access the Reveal button within that feedback popover. It's not more steps than the way things are now, but the way things are now, there is at least something on the page that indicates how to proceed to see the correct answers. I don't know what to do about this though. Maybe there should be a "Show Correct Answers" button like there is with the problem editor?

[drgrice1]

One thing I see about this is that when a user first comes to a problem's page, it's not clear how to see the correct answers now. The user needs to know to use the Check/Submit button first, so that the general feedback buttons appear, and then access the Reveal button within that feedback popover. It's not more steps than the way things are now, but the way things are now, there is at least something on the page that indicates how to proceed to see the correct answers. I don't know what to do about this though. Maybe there should be a "Show Correct Answers" button like there is with the problem editor?

You are missing something here. Only instructors need to know to click the check/submit button to show answers (and that was required before). Students do not. After the due date, if the automatic feedback option is enabled, then the feedback is there without even needing to click the Check/Submit buttons.

[drgrice1]

Thinking about this more, I am not sure that this pull request changes the issue of the user needing to know how to see correct answers much at all. Even with the checkbox there, after you check it and click check answers, you see the same thing that you see with this pull request after clicking check answers. With the new feedback approach, the correct answers are not immediately visible on the page in either case. The user needs to click on the feedback button to see the correct answers.

[drgrice1]

The same would be true of a "Show Correct Answers" button. That or the current checkbox may be an indicator that the correct answers should be shown, but yet, they are not immediately shown as they were before with the old attempts table.

[Alex-Jordan]

Only instructors need to know to click the check/submit button to show answers

I have instructors in mind. In particular, instructors who only use WeBWorK occasionally or are newbies.

and that was required before

But that's not my point. My point is that currently if they go to the problem page, they can read "Show: [checkbox] Correct Answers". So my point is that there is something on the page that communicates "this is how you can see the correct answers". Then they check it and nothing happens, and it becomes clear they need to also push a button.

With the change here, to the inexperienced, there is nothing directing a user that "this is how you can see the correct answers". You can see the buttons, but based on their labels, you do not expect they will eventually reveal the correct answers to you. Your eyes keep scanning the page wondering "how the heck can I just see the correct answers to this problem?"

The same would be true of a "Show Correct Answers" button. That or the current checkbox may be an indicator that the correct answers should be shown, but yet, they are not immediately shown as they were before with the old attempts table.

I'm not commenting on how using these things (the old checkbox or the would-be "Show Correct Answers" button) does not immediately show the answers. I'm commenting on how nothing on the page gives an inexperienced person guidance as to how they can see the correct answers.

[drgrice1]

But that is exactly my point. With the develop branch, you might have the checkbox that gives an indication of how to show correct answers, but then you check it and click "Check Answers" (clicking a button after checking the checkbox is already something that a webwork newbie would not know to do), and after clicking the button the correct answers are still not shown. You scan the page and say where are they?

To be quite frank, by the time that the new webwork user is looking for correct answers, it is more than likely that the correct answer with the reveal button will have been found. I think your point is much less of an issue than you believe.

[drgrice1]

The point is that the first thing a new user would do is try out a problem. They would enter an answer, and click "Check Answers". At this point the feedback buttons appear. Say at this point, the user does not realize the buttons are buttons, and doesn't click on any of them. Then the user sees the "Correct Answers" checkbox, and decides to try that out. The user checks the box. Nothing happens. Maybe the user then tries to click "Check Answers". Nothing changes. The page is exactly the same as before.

I think that a note should be added to the bottom with the notes about hints and solutions, that states something like, "the correct answer shown when a feedback button is clicked is an instructor preview that will only be shown to students after the answer date."

[somiaj]

Overall I like this change. Though I don't see why if automatic feedback is enabled that instructors need to then click check answers to see the correct answer (before the due date). This might address @Alex-Jordan concerns, and although you answered about its current behavior I may have missed your logic as to why not just show the correct answer with the reveal button in this case as well?

[drgrice1]

The point is to only show feedback when feedback is requested. Showing the feedback always for instructors before the answer date is not desirable.

[drgrice1]

Note that if an instructor has submitted an answer and the automatic feedback option is true, then feedback is shown for an instructor with clicking check answers just as it is for students. However, we certainly do not want the feedback shown even for an instructor when there are no answered questions.

[somiaj]

I agree with that, feedback should only be shown if requested, my question was why not always include the correct answer in the feedback (with the reveal button). Currently in my testing with autofeedback enabled, going to a problem does give feedback only in the cases you described, but the correct answer isn't available until after I check/submit answers.

[drgrice1]

I see what you mean. I made it so that the correct answer is always shown for instructors when feedback is shown except when it is a preview.

[somiaj]

One thing I notice testing this (though not a part of this pull request, this was behavior before this), is on problems with multiple answer boxes, if one feedback popup is open, and you click the button to open another, the two feedback popups can overlap, and sometimes the new feedback popup is placed behind the one that is already open (in the screenshot below, I opened the bottom one first, then the second to last one second, and it was put behind the first, and the top one last).

Would it be useful to close all other feedback popups before opening a new one to avoid this? Or ensure the one that is just opened is on top?

[drgrice1]

Which one is on top is simply the order in the DOM, and there is nothing that can be done about that.

We could make other popovers close when a new one is opened.

[Alex-Jordan]

This is all good, and perhaps after it is merged if I think of a way to address that concern I brought up, I'll try later. For now I don't have an idea that I love. I'll agree with you that it's a non-issue for usability, but I still do feel it's an issue for "first impressions". My head is in the marketing department.

Which one is on top is simply the order in the DOM, and there is nothing that can be done about that.

What about using z-index? You could maintain a global js array variable with all the z-indices of the ones that have opened so far. When a new one opens, use a higher z-index and add it to the array. When one closes, remove that z-index from the array.

We could make other popovers close when a new one is opened.

I can imagine situations where you want two open at the same time, to carefully compare the two displayed correct answers.

[pstaabp]

I think this is a good direction with the table-less results. I see @Alex-Jordan 's point about discoverability, but I don't see it any worse than the way it way.

One way would be to change the "Check Answers" button to "Check and Reveal Answers", but I'm not crazy about that. It seems like new instructors will figure it out quite quickly.

Also, I like the automatic revealing of the answers when needed (after the due date).

[drgrice1]

Using the z-index to try and order the popups is not a good idea. There currently is a very small range of z-indices available to work with, and generally maintaining the order of the z-indices would be a nightmare.

It is debatable if it is really that valuable to have two popups open at the same time for answer comparison for different questions.

As to finding the correct answers, I have to say that I really think that is a non-issue. Once an instructor clicks "Check Answers" or "Submit Answers" for the first time on a problem, the correct answers are there in the feedback. An answer doesn't even need to be entered to get this. It is really quite obvious. Furthermore, once answers have been submitted by clicking "Submit Answers" (even with empty answers), when an instructor comes back to the problem after leaving it, the feedback is there with the correct answers in it without having to click any buttons.

I generally think that adding more buttons or making the label on the button something excessively long like "Check and Reveal Answers" is a bad idea also. That will become a layout mess. Note that if the "Show Me Another" feature is enabled we already have four submit buttons for instructors, and this already wraps on narrow screens. That doesn't particularly look the best.

By the way this entire discussion about finding correct answers does not belong here. That is only pertinent to what is done in openwebwork/webwork2#2275.

[drgrice1]

I have added a tweak to this that makes the last opened popover in front of all of the rest. Furthermore, if a popover is clicked on or keyboard focus moves into it, then it is moved to the front. The other popovers are all shown in the order they appear in the DOM. So this partially does what @Alex-Jordan suggested with z-index management, except it only used to z-indices (17 and 18). It does not maintain an order for all popovers, but just maintains a "currently active" popover.

[drgrice1]

I made one additional tweak to this. Each time a feedback popover is closed and the opened again, the correct answer "Reveal" button is back again. So I made it so that once the correct answer has been revealed once, each subsequent time the popover is opened it is automatically revealed without needing to click the reveal button again. Note that a page reload will reset this behavior.

[pstaabp]

Seems like this is ready to merge and use it for a while. We can tweak if needed.