Note: The document you're reading was used for the first 6 participants in our study. Midway through our study, we revised the order of tasks to try to determine if our research protocol is impacting people's ability to find the authenticated menu. The sessions for P7, P8, and P12 used this conversation guide. We extended our study and decided to remove the task to find letters, since that could be in so many various parts of the site, and we wanted to focus remaining sessions on the auth menu. Sessions for P13-P16 used this version of the guide.
- Update participants name in dynamic data file
- Update participants name in Auth mural board
- Post in #feedback-backchannel
- Send out the observer instructions to your observers (Slackbot will do this for you if you type "observer instructions" into #feedback-backchannel).
- Check your [screensharing setup] and share permissions for participants.
Thanks for joining us today! My name is Liz and [NOTETAKER NAME] is taking notes. I also have a couple other colleagues in the Zoom waiting room who would like to observe and take notes. But before we get to them...
Today we're going to talk about navigating a VA website when you’re signed in. Before we start, I have a few things that I want to go over with you:
-
This entire session should take about 45 minutes/60 minutes. I want to be sure not to keep you much longer, so I may occasionally prompt you with the next question or topic.
-
In this session, we want to hear your honest opinions. There are no right or wrong answers; we are not testing your knowledge or abilities. I welcome all of your feedback, and I won't be offended by any opinions you express.
-
Your participation in this interview is completely voluntary. If you would like to stop the session or do not want to answer a question for any reason at any time, that's no problem. Please let me know. Whether you choose to participate or not will not affect your benefits in any way with the VA.
-
You'll be interacting with a prototype today. This is a demo tool that may not function exactly the way you expect. I have prepopulated the prototype with your name, just so it feels a little more realistic. It's mostly populated with dummy data. Some areas of the prototype will be clickable, and some will not. Since it's a demo, none of your actions will affect your VA information or benefits.
-
I know the folks who invited you to participate in this interview asked if it was alright to allow people to observe this interview, but I want to double-check with you before we get going. Are you okay with a couple of additional observers during this session?
- If no: Use Slack to let observers know
- If yes: Admit observers.
- Then, ask the notetaker and observers to give a brief, camera-on introduction, like their first name and position with VA.gov.
- Once complete, ask the notetaker and observers to turn off their cameras for the remainder of the session to save bandwidth
-
Are you comfortable if I record my screen and audio as we talk today? We use the recordings to confirm that we have captured your opinions accurately. After we complete the analysis, the recordings are destroyed, and none of your comments will be attributed to you directly.
-
Would you be comfortable if we decided to use a clip of our session with you to help our partners and colleagues understand Veterans’ experiences? In these video clips, we blur out any information that could possibly be used to identify you, like your onscreen image. These clips will not be shared outside of the VA. At the end of the interview, I'll also confirm that this is okay with you, so you will get a chance to veto this once you know what we've discussed.
- If yes: Once I start recording, I am going to confirm both of these things once more. Start recording.
-
I have started recording. I'd like to confirm: Are you comfortable if I record my screen and the audio as we talk today?
-
Are you comfortable if we decide to use a recorded clip from our discussion today to share with other VA partners if we remove any information that can be used to identify you?
Before we look at anything, I’d like to learn a little more about your experience with VA.
- According to my notes, you have [benefits from VA]. Is that correct?
- Do you ever log into any VA websites?
- If yes: Which website? Tell me about how you use that once you’re logged in.
- If no: What websites do you sign into? How do you use those websites?
- Moderator goal: gather background info that might be helpful later in the session
- If they have a rated disability: I'm going to update the prototype with your disability rating, just so it feels a little more realistic. Would you mind telling me your percent rating?
Update dynamic data file: https://codepen.io/team/VA-gov-Applications/pen/gOEWmPo
Now I’d like to get your feedback on a prototype and have you share your screen. Do you know how to share your screen in zoom?
Guide participant through screen sharing if needed
Great, I can see your screen. Next, I’ll have you pull up the prototype. Would you prefer I send you the link via Zoom chat or tell you the URL verbally so you can enter it in?
Prototype link: https://cdpn.io/pen/debug/JjzNWbR Guide participant to prototype. Prototype will start on the unauthenticated version of the life insurance page.
Excellent, I can see the prototype on your screen.
Just as a reminder, since this is a prototype, not everything is clickable or fully functional. We’re using a tool called CodePen for the prototype, so you might encounter that word as you go through the screens.
It’ll be helpful if you can think out loud as much as possible so I can understand how this prototype works for you.
First, I’d like for you to show me how you’d use this prototype to see whether or not you have any rated disabilities on file with VA.
Goal: get participant to View Disability Rating page
Once participant has found it, ask:
- How was that experience for you?
- How did it line up with what you expected?
- Did the participant expect to have to log in?
- How easily does the participant understand how to log in?
- What path did the participant take to get to the disability rating page? Auth menu, main menu, search, etc?
- Were any usability hurdles encountered?
Next, I’d like for you to imagine you want to apply for property tax relief, and you need a document that shows both your service information and benefits you get from VA. Show me how you’d find that kind of document using this prototype.
Goal: get participant to Letters page
Once participant has found it, ask:
- How was that experience for you?
- How did it line up with what you expected?
- What path did the participant take to get to the letters page? Auth menu, main menu, search, etc?
- If they used the auth menu previously, how easily were they able to get back to it?
- Were any usability hurdles encountered?
As a last task, I’d like for you to show me where you would go to update your phone number on file with VA.
Goal: get participant to Profile hub page. There is not an actual page for updating phone numbers, but the contact information link will click to a dummy page.
Once participant has landed on dummy page Ok, thank you! We didn’t set this page up but it was helpful to see how you’d go about getting here.
- How was that experience for you?
- How did it line up with what you expected?
- What path did the participant take to get to the profile page? Auth menu, main menu, search, etc?
- If they used the auth menu previously, how easily were they able to get back to it?
- Were any usability hurdles encountered?
Note: this task is only for participants who didn’t use their auth menu at all during the session I’d like for you to check out one part of the prototype we haven’t looked at yet. On desktop: Did you notice your name in the heading of the website? One mobile: Can you click on the menu button please? Had you noticed your name here before?
- What do you think will happen if you click on it?
- Go ahead and click on it. What are your thoughts about what happened when you clicked?
Thank you so much for all that feedback! This has been really helpful. I want to get your thoughts on a few more things before we wrap up.
Authenticated menu should be open, and ask the questions below.
I’m going to have you stop screen sharing now, and I’ll share my screen. I created a quick visual aid for this next part of our conversation. Open Mural board. Link: Authenticated Menu Co-Design
Ok, here’s a rough version of the menu we just looked at. I thought this would be interesting to look at with you, so if there are changes you wish you could make, we can make them happen.
- How do the options in this menu line up with what you expected?
- Where do you think each link goes? What makes you think so?
- How would you order these links? Why?
- What would you like to change or add? Why?
- Anything else you’d like me to know about what you’d want from a menu like this?
- Note to moderator, may need to offer context if My VA has not been reviewed or discussed by this point: I’m curious, how would you describe the difference between My VA and Profile to another Veteran
Great, thank you for going through that with me. I’ll stop screen sharing now.
- Overall, about everything you've seen in this session...What worked well for you?
- What was unclear or didn't work well for you?
- Is there anything else that we haven't talked about that you think I should know?
I want to circle back and re-confirm that you are comfortable if we were to share a video clip of this session with our VA partners? Just to remind you, we will remove all information that could be used to identify you before sharing and will only share the videos with VA partners with the purpose of improving VA.gov and other digital experiences.
- If yes: Is there any part of our discussion you would NOT want shared in a video clip?
Well we really appreciate you taking the time to share your thoughts with us today. Your feedback is so helpful to us as we continue to work on the site and make sure it really works for Veterans.
Thanks! Lastly, Perigean will be sending you a thank you note with a little blurb that you can pass along to other Veterans you may know to provide them the chance to participate in future research studies.
Thank you so much again, and enjoy the rest of your day!
[In the rare case of an emergency during a session, be prepared with an emergency exit strategy. Plan to include here at least one option, so its easy to find during a session. Take some time to practice so it feels more comfortable.]