Case Study: Usability Evaluation of Microsoft SkyDrive

Problem

This team project was completed for a graduate course in Human-Computer Interaction at the Indiana University School of Informatics and Computing at IUPUI.

We were challenged to evaluate the usability of an existing web application by conducting:

My team decided to evaluate Microsoft SkyDrive (now called OneDrive), a cloud-based file and photo storage service with integrated Office web apps (Word, Excel, PowerPoint, OneNote) for creating, editing, viewing, and sharing files.

I was directly involved in all aspects of this project.

Approach

Heuristic Evaluation

For the heuristic evaluation, we decided to use Jakob Nielsen’s 10 usability heuristics for user interface design to inspect SkyDrive for usability issues because these particular heuristics are well-known and validated by research:

We decided to rank the severity of each usability issue using Nielsen’s severity rating matrix because it takes into account the proportion of users likely to experience the issue and the issue’s impact on the user experience:

SEVERITY
RATING
Few Users Likely to
Experience Issue
Many Users Likely to
Experience Issue
Small Impact on
User Experience
Low Medium
Large Impact on
User Experience
Medium High

Each team member conducted an independent heuristic evaluation of SkyDrive, in order to identify usability issues. We inspected SkyDrive’s interface and interactions related to: navigation, file and folder management, file and photo sharing, and account management. We then shared and discussed our results to compile a final set of consolidated findings.

In our findings, we described each usability issue with a chart providing the following information:

Example Finding from Heuristic Evaluation
Issue Inconsistent Menu: Options displayed in top menu bar change based on location selected in left navigation pane. Certain menu options are displayed (or not) depending on whether location has files (or folders) and whether any of these items are selected. This may be confusing to users who are used to static menu bars in other Microsoft Office applications.
Screen Default menu options for Files location if no folders or files are selected:SkyDrive Menu Bar View 1Additional menu options appear if item selected (using hidden checkbox that appears on hover):SkyDrive Menu Bar View 2Certain locations in left navigation display no menu options if the location doesn’t have files:SkyDrive Menu Bar View 3
Violation Visibility of System Status, Consistency and Standards
Severity Medium
Recommendation Present a consistent menu bar, and disable any menu options that are not currently applicable

Cognitive Walkthrough

For the cognitive walkthrough, my team needed to determine the target user profiles for SkyDrive and then create scenarios for these users to perform relevant tasks using SkyDrive.

Based on SkyDrive’s integration of Office web apps, my team decided that two key groups of SkyDrive users would be:

We also decided that SkyDrive users could be categorized based on functional role (which might vary from task to task):

Our team used these user profiles to create these task scenarios for the cognitive walkthrough:

Each team member conducted an independent cognitive walkthrough of SkyDrive by completing the task scenarios, in order to identify usability issues. We then shared and discussed our results to compile a final set of consolidated findings.

For each task scenario, we described each usability issue with a chart providing the following information:

Example Finding from Cognitive Walkthrough
Task Viewing Photos
Issue Unclear Navigation: After user has viewed photo, it is unclear how to navigate back to parent folder. User must click “View folder” link near top of right vertical menu. This may be overlooked by users who are accustomed to back navigation link being displayed in top left.
Screen In photo viewer, navigating back to parent folder location requires clicking blue “View folder” link near top of right vertical menu:
SkyDrive Photo Viewer
Severity Medium
Recommendation Navigation to return to parent folder should be more discoverable. For example, place link labeled “Back to Folder” (with back arrow icon) in upper menu bar next to SkyDrive logo.

Usability Testing

For the usability testing, my team created a testing script with a sequence of 8 tasks as part of a scenario in which the user is working with photos and documents related to a vacation. Each task was selected to focus on specific functions or features of SkyDrive that were identified as having usability issues during the heuristic evaluation or cognitive walkthrough.

Tasks for Usability Testing
Task Task Script for Participants
1 - Upload Folder of Photos You want to add a folder of photos called “My Vacation Photos” from your computer to your cloud drive. (Participants were provided with SkyDrive account login on computer with existing folder of photos.)
2 - View Photos You want to take a look at the vacation photos in the cloud drive. Briefly view a large image of each photo. When you are finished, go back to the main screen where you started.
3 - Create New Folder with New Word Document You want to write about your vacation. Make a folder called “My Vacation Files”. Inside that folder, make a new file called “My Vacation Story” in the cloud drive to write about your vacation. For right now, add the following text as your story: “This is my vacation story.”
4 - Share Link to Word Document You want to allow some friends to read your vacation story. Send a link to these two friends to allow them to view your story but not be able to change it. (Participants were provided email addresses for link sharing.)
5 - Create Another New Word Document Inside your vacation folder, you decide to write another new file called “My Next Vacation”. Add the following text to this file: “This will be my next vacation trip.”
6 - Delete File from SkyDrive You decide to remove the file called “My Next Vacation” from your cloud drive.
7 - Edit Existing Word Document and Share Again You decide to add to your original vacation story. Add this to the end of your story: “My vacation was very enjoyable.” Resend a link to your two friends to let them know you’ve updated your story. Include a message saying “I updated my vacation story.”
8 - Recover Deleted File You decide that you want to recover the file about your next vacation trip. After recovering the file, move the file out of the vacation folder, and place it into the main part of your cloud drive.

My team recruited a total of 8 participants (5 female, 3 male) for the usability testing. We conducted individual testing sessions with each participant:

After each testing session, my team reviewed and discussed the observation notes. As necessary, we also reviewed the participant recording to confirm or clarify usability issues. We then compared the results from all participants to compile a final set of consolidated findings.

For each task, we described each usability issue with a chart providing the following information:

Example Finding from Usability Testing
Task Edit Existing Word Document and Share Again
Issue Read-Only View: All 8 users attempted unsuccessfully to edit document while still in read-only view, which is the default view when a document is opened (which is contrary to how Word desktop application works). 3 users were uncertain about difference in options to “Edit in Word” or “Edit in Word Web App.”
Screen In Word web app, document opens in “read-only” view by default. User must select one of two options to edit document.
Severity Medium
Recommendation If user is document owner (or has editing privileges for a shared file), document should open in edit mode, instead of read-only view, to provide consistency with Word desktop application.
Default mode for editing document should be to edit using Word web app. Offer option in File menu to download file (in order to edit offline using Word desktop application).

Solution

My team compared our findings from our heuristic evaluation, cognitive walkthrough, and usability testing to create a final report summarizing the usability issues discovered within SkyDrive with our recommendations for potential fixes.

The heuristic evaluation and cognitive walkthrough discovered 22 usability issues (severity: 6 low, 13 medium, 3 high). Most of the issues that occurred during usability testing had been previously identified, but there were 2 new issues discovered.

In nearly all cases, users were still able to complete their tasks despite the usability issues — though this may be partly due to motivation to complete the scripted task, whereas in a natural context, users might have given up on the task.

The users’ perception of SkyDrive’s usability varied widely as evidenced by the range in SUS scores (low = 27.5, high = 95). The average SUS score was 67. Generally, SUS scores of 70 or higher are considered acceptable.

Reflection

All three of these usability evaluation methods have advantages and disadvantages relative to each another:

Ideally, usability testing sessions would be incorporated into a design and development schedule on a periodic basis (e.g., bi-weekly or monthly) to test new features. Testing with as few as 3-5 participants can be sufficient to reveal most issues.

NEXT: Human-Centered Design of Art Fair App

Next Case Study