{Information for the tester is in curly brackets {}}
{We'll be testing our prototypes A and B on 8 or 12 people. The probands are identified uniquely as #i in (1, ..., 8/12) }
{Setup for the tester
- Connect iPad to the Mac
cd study./runStudyand enter participant's id (without #)- In QuickTime Player, choose "File" > "New Movie Recording" (not "New Screen Recording"), tap the arrow to choose the iPad and record it's screen }
{Link for questionnaire: https://docs.google.com/forms/d/e/1FAIpQLSc8zU0_1yKfc3mgu-SK4-0RwdJAbGEq76R5gKkZa2THE9VrEw/viewform?usp=sharing}
Hi [#i's name]. Today we are looking for ways to improve the user experience of the responsivity editor in our drawing-app, Flex Studio. It allows artists to define how their layers should move when their containing panel is being resized. So, let's set the mood.
Imagine you're an up-and-coming webcomic artist that has just drawn the last panel of their biweekly release. What's left to do is to tell Flex Studio how to adapt it to different aspect ratios. Your readers on Tapas and Instagram will love this work, no matter if its shown in a wide format or at 1:1. But more on that in our example.
Remember: This is a test of the component; not your creative abilities. If you find something difficult to use, chances are that others will as well, so your feedback helps everyone.
The study will take about 20 minutes. We'll answer any questions you have in the end of the study. Oh, and one last organizational thing: We would like to make a video recording of the screen for evaluation purposse. Would you be so kind and give consent in this form?
{pass paper consent form with name (NOT ID!) and signature.} {in the meantime, begin logging by confirming prompt in command line with "y". You'll see a dialogue with the current date and IV settings to be tested.}
Do you have any questions? Otherwise let's get started.
We'll run two rounds of tests, one for every interface variant. In every round, we'll first introduce you to it by providing a simple demo, and will grant you some time to play around with it in your own drawing. Once you're familiar with it, we'll ask you to make an artist-provided panel responsive according to a reference. Everything clear?
Great, let's begin with variant 1: I'll demo it to you first. {Show and example of the direct/indirect tool with a drawing of your own. Sun pinned to top right, tree pinned to center left, person pinned to bottom center }
{Take away Apple Pencil!}
Now that you're familiar with the tool, let's get to work! Here's what you are tasked to recreate. {Hand out paper reference, reset app state, select correct responsivity variant in debug tool, press run test.}
We'll start our stopwatch once you press this button, and will stay silent until you press the stop button. Note that it is possible to exactly recreate the reference behavior but not required. Press stop when you have achieved a reasonably accurate result. Any questions? Ok. Ready when you are. {Let participant do their thing.}
Wonderful! Please fill out this section of the form now. {Hand form.}
Thanks! Let's proceed with variant 2: Again, I'll demo it to you first. {Show and example of the direct/indirect tool with a drawing of your own. Sun pinned to top right, tree pinned to center left, person pinned to bottom center }
{Take away Apple Pencil!} {Reset app state, select correct responsivity variant in debug tool, press run test.}
By now, you know what's coming: It's time for you to repeat the task. Again, note that it is possible to exactly recreate the reference behavior but not required. Press start when you begin and stop when you have achieved a reasonably accurate result. Any questions? Ok. Ready when you are. {Let participant do their thing.}
Great! Please fill out this section of the form now. {Hand form.}
With this we're at the end of the practical part of this study. Congratulations! You're a Flex Studio pioneer ;)
To wrap the study up, please fill out this last section of the form. {Hand form.}
Thank you very much for participating.
(V1, V2)1:
introduction to the specific tool: (~5min)
To give you some impression of how to use the tool I'll give an example now.
{Show and example of the direct/indirect tool with a drawing of your own. Sun pinned to top right, tree pinned to center left, person pinned to bottom center }
open task 1: (4 min)
Here's our first version of the drawing app. Make sure to try out the reference mode by clicking on this button. You're given 4 minutes to try the app out, beginning now.
main task 2:
(~10 min)
As you can see on the IPad screen, you're given a preset. On the first layer there is [...], on the second [...]...
Now you're shown a video about how the different layers in the drawing should behave in the end when you're resizing the panel. You're asked to create the same behaviour when resizing the panel. You should try to be as accurate as possible, while also being fast. Before you begin with the task, push the button in the top right to start. If you're finished press the button on the top right again to stop.
(show video.)
...
Well done. Now you can go on filling out the questionnaire.
Questionnaire 1
Q1)
Mental Demand: How much mental and perceptual activity was required (e.g. thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving?
Q2)
Physical Demand: How much physical activity was required (e.g. tapping, dragging, activating, etc.)? Was the task easy or demanding, slow or fast, restful or laborious, slack or strenous?
Q3)
Performance: How successful/unsuccessful do you think you were in accomplishing the goals of the task set by us, i.e. matching our cartoon in different aspect ratios? How unsatisfied/satisfied were with your performance in accomplishing these goals?
Q4)
Effort:
How hard did you have to work (mentally or physically) to accomplish your level of performance?
Q5)
Frustration Level:
How insecure, discouraged, irritated, stressed or annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?
Q6)
Restrictivity:
Considering the 5 minutes you were given at the beginning, and the task you just completed, how restrictive is the tool, or how much do you think the tool allows you to express your creativity? Is it restrictive or extensive, enabling or disabling?
Q7)
Complexity:
Again, considering your time with the tool sofar, how hard/easy is it to use? How simple, effortless, straighforward versus complex, difficult, unintuitive was the use of the tool when accomplishing your goal.
(V1, V2)2:
Well done, now it's time for our second version:
introduction to the specific tool: (~3min)
To give you some impression of how to use the tool I'll give an example now.
{Show and example of the direct/indirect tool with a drawing of your own.}
open task 1: (4min)
Again you should get used to the responsivity editor and it's new interface. You'll get 5 min from now on.
main task 2: (~10 min)
Again you'll get a preset and see a video of how the layers should behave when you resize the panel.
Before you begin with the task, push the button in the top right to start. If you're finished press the button on the top right again to stop.
(show video.)
...
Well done. Now you can go on filling out the questionnaire.
Questionnaire 2
Q1)
Mental Demand: How much mental and perceptual activity was required (e.g. thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving?
Q2)
Physical Demand: How much physical activity was required (e.g. tapping, dragging, activating, etc.)? Was the task easy or demanding, slow or fast, restful or laborious, slack or strenous?
Q3)
Performance: How successful/unsuccessful do you think you were in accomplishing the goals of the task set by us, i.e. matching our cartoon in different aspect ratios? How unsatisfied/satisfied were with your performance in accomplishing these goals?
Q4)
Effort:
How hard did you have to work (mentally or physically) to accomplish your level of performance?
Q5)
Frustration Level:
How insecure, discouraged, irritated, stressed or annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?
Q6)
Restrictivity:
Considering the 5 minutes you were given at the beginning, and the task you just completed, how restrictive is the tool, or how much do you think the tool allows you to express your creativity? Is it restrictive or extensive, enabling or disabling?
Q7)
Complexity:
Again, considering your time with the tool sofar, how hard/easy is it to use? How simple, effortless, straighforward versus complex, difficult, unintuitive was the use of the tool when accomplishing your goal.
Questionaire 3
Q1)
Comparison:
(Open question)
Having used both methods, which one did you prefer? Which one of the key aspects (i.e. mental/physical demand, performance, etc.) in the questionaire was the main reason for your preference?
Q2)
Keyframe experience:
Do you have any previous experience with Keyframing, as seen in video-editing/animation software?
Q3)
Constraint based experience:
Do you have any previous experiences with constraint based layouting, like in Figma or similar software?
Q4)
Improvements:
Do you have any suggestions for points of improvements?
Thanks a lot for your participation.