This team project was completed for a graduate course in Human-Computer Interaction at the Indiana University School of Informatics and Computing at IUPUI.
We were challenged to evaluate the usability of an existing web application by conducting:
My team decided to evaluate Microsoft SkyDrive (now called OneDrive), a cloud-based file and photo storage service with integrated Office web apps (Word, Excel, PowerPoint, OneNote) for creating, editing, viewing, and sharing files.
I was directly involved in all aspects of this project.
For the heuristic evaluation, we decided to use Jakob Nielsen’s 10 usability heuristics for user interface design to inspect SkyDrive for usability issues because these particular heuristics are well-known and validated by research:
We decided to rank the severity of each usability issue using Nielsen’s severity rating matrix because it takes into account the proportion of users likely to experience the issue and the issue’s impact on the user experience:
|Few Users Likely to
|Many Users Likely to
|Small Impact on
|Large Impact on
Each team member conducted an independent heuristic evaluation of SkyDrive, in order to identify usability issues. We inspected SkyDrive’s interface and interactions related to: navigation, file and folder management, file and photo sharing, and account management. We then shared and discussed our results to compile a final set of consolidated findings.
In our findings, we described each usability issue with a chart providing the following information:
|Issue||Inconsistent Menu: Options displayed in top menu bar change based on location selected in left navigation pane. Certain menu options are displayed (or not) depending on whether location has files (or folders) and whether any of these items are selected. This may be confusing to users who are used to static menu bars in other Microsoft Office applications.|
|Screen||Default menu options for Files location if no folders or files are selected:Additional menu options appear if item selected (using hidden checkbox that appears on hover):Certain locations in left navigation display no menu options if the location doesn’t have files:|
|Violation||Visibility of System Status, Consistency and Standards|
|Recommendation||Present a consistent menu bar, and disable any menu options that are not currently applicable|
For the cognitive walkthrough, my team needed to determine the target user profiles for SkyDrive and then create scenarios for these users to perform relevant tasks using SkyDrive.
Based on SkyDrive’s integration of Office web apps, my team decided that two key groups of SkyDrive users would be:
We also decided that SkyDrive users could be categorized based on functional role (which might vary from task to task):
Our team used these user profiles to create these task scenarios for the cognitive walkthrough:
Each team member conducted an independent cognitive walkthrough of SkyDrive by completing the task scenarios, in order to identify usability issues. We then shared and discussed our results to compile a final set of consolidated findings.
For each task scenario, we described each usability issue with a chart providing the following information:
|Issue||Unclear Navigation: After user has viewed photo, it is unclear how to navigate back to parent folder. User must click “View folder” link near top of right vertical menu. This may be overlooked by users who are accustomed to back navigation link being displayed in top left.|
|Screen||In photo viewer, navigating back to parent folder location requires clicking blue “View folder” link near top of right vertical menu:
|Recommendation||Navigation to return to parent folder should be more discoverable. For example, place link labeled “Back to Folder” (with back arrow icon) in upper menu bar next to SkyDrive logo.|
For the usability testing, my team created a testing script with a sequence of 8 tasks as part of a scenario in which the user is working with photos and documents related to a vacation. Each task was selected to focus on specific functions or features of SkyDrive that were identified as having usability issues during the heuristic evaluation or cognitive walkthrough.
|Task||Task Script for Participants|
|1 - Upload Folder of Photos||You want to add a folder of photos called “My Vacation Photos” from your computer to your cloud drive. (Participants were provided with SkyDrive account login on computer with existing folder of photos.)|
|2 - View Photos||You want to take a look at the vacation photos in the cloud drive. Briefly view a large image of each photo. When you are finished, go back to the main screen where you started.|
|3 - Create New Folder with New Word Document||You want to write about your vacation. Make a folder called “My Vacation Files”. Inside that folder, make a new file called “My Vacation Story” in the cloud drive to write about your vacation. For right now, add the following text as your story: “This is my vacation story.”|
|4 - Share Link to Word Document||You want to allow some friends to read your vacation story. Send a link to these two friends to allow them to view your story but not be able to change it. (Participants were provided email addresses for link sharing.)|
|5 - Create Another New Word Document||Inside your vacation folder, you decide to write another new file called “My Next Vacation”. Add the following text to this file: “This will be my next vacation trip.”|
|6 - Delete File from SkyDrive||You decide to remove the file called “My Next Vacation” from your cloud drive.|
|7 - Edit Existing Word Document and Share Again||You decide to add to your original vacation story. Add this to the end of your story: “My vacation was very enjoyable.” Resend a link to your two friends to let them know you’ve updated your story. Include a message saying “I updated my vacation story.”|
|8 - Recover Deleted File||You decide that you want to recover the file about your next vacation trip. After recovering the file, move the file out of the vacation folder, and place it into the main part of your cloud drive.|
My team recruited a total of 8 participants (5 female, 3 male) for the usability testing. We conducted individual testing sessions with each participant:
After each testing session, my team reviewed and discussed the observation notes. As necessary, we also reviewed the participant recording to confirm or clarify usability issues. We then compared the results from all participants to compile a final set of consolidated findings.
For each task, we described each usability issue with a chart providing the following information:
|Task||Edit Existing Word Document and Share Again|
|Issue||Read-Only View: All 8 users attempted unsuccessfully to edit document while still in read-only view, which is the default view when a document is opened (which is contrary to how Word desktop application works). 3 users were uncertain about difference in options to “Edit in Word” or “Edit in Word Web App.”|
|Screen||In Word web app, document opens in “read-only” view by default. User must select one of two options to edit document.
|Recommendation||If user is document owner (or has editing privileges for a shared file), document should open in edit mode, instead of read-only view, to provide consistency with Word desktop application.
Default mode for editing document should be to edit using Word web app. Offer option in File menu to download file (in order to edit offline using Word desktop application).
My team compared our findings from our heuristic evaluation, cognitive walkthrough, and usability testing to create a final report summarizing the usability issues discovered within SkyDrive with our recommendations for potential fixes.
The heuristic evaluation and cognitive walkthrough discovered 22 usability issues (severity: 6 low, 13 medium, 3 high). Most of the issues that occurred during usability testing had been previously identified, but there were 2 new issues discovered.
In nearly all cases, users were still able to complete their tasks despite the usability issues — though this may be partly due to motivation to complete the scripted task, whereas in a natural context, users might have given up on the task.
The users’ perception of SkyDrive’s usability varied widely as evidenced by the range in SUS scores (low = 27.5, high = 95). The average SUS score was 67. Generally, SUS scores of 70 or higher are considered acceptable.
All three of these usability evaluation methods have advantages and disadvantages relative to each another:
Ideally, usability testing sessions would be incorporated into a design and development schedule on a periodic basis (e.g., bi-weekly or monthly) to test new features. Testing with as few as 3-5 participants can be sufficient to reveal most issues.