Case Study: Usability Evaluation of Microsoft SkyDrive
Problem
This team project was completed for a graduate course in Human-Computer Interaction at the Indiana University School of Informatics and Computing at IUPUI.
We were challenged to evaluate the usability of an existing web application by conducting:
- Heuristic Evaluation ‒ expert inspection against a set of usability principles
- Cognitive Walkthrough ‒ expert review by completing a set of task scenarios
- Usability Testing ‒ observation of target users completing a set of task scenarios
My team decided to evaluate Microsoft SkyDrive (now called OneDrive), a cloud-based file and photo storage service with integrated Office web apps (Word, Excel, PowerPoint, OneNote) for creating, editing, viewing, and sharing files.
I was directly involved in all aspects of this project.
Approach
Heuristic Evaluation
For the heuristic evaluation, we decided to use Jakob Nielsen’s 10 usability heuristics for user interface design to inspect SkyDrive for usability issues because these particular heuristics are well-known and validated by research:
- Visibility of System Status
- Match Between System and Real World
- User Control and Freedom
- Consistency and Standards
- Error Prevention
- Recognition Rather than Recall
- Flexibility and Efficiency of Use
- Aesthetic and Minimalist Design
- Help Users Recognize, Diagnose, and Recover from Errors
- Help and Documentation
We decided to rank the severity of each usability issue using Nielsen’s severity rating matrix because it takes into account the proportion of users likely to experience the issue and the issue’s impact on the user experience:
| SEVERITY RATING |
Few Users Likely to Experience Issue |
Many Users Likely to Experience Issue |
| Small Impact on User Experience |
Low | Medium |
| Large Impact on User Experience |
Medium | High |
Each team member conducted an independent heuristic evaluation of SkyDrive, in order to identify usability issues. We inspected SkyDrive’s interface and interactions related to: navigation, file and folder management, file and photo sharing, and account management. We then shared and discussed our results to compile a final set of consolidated findings.
In our findings, we described each usability issue with a chart providing the following information:
- Issue - description of usability issue
- Screen - screenshot(s) showing issue
- Violation - heuristic(s) violated by issue
- Severity - severity of issue
- Recommendation - possible redesign to fix issue
| Issue | Inconsistent Menu: Options displayed in top menu bar change based on location selected in left navigation pane. Certain menu options are displayed (or not) depending on whether location has files (or folders) and whether any of these items are selected. This may be confusing to users who are used to static menu bars in other Microsoft Office applications. |
| Screen | Default menu options for Files location if no folders or files are selected: Additional menu options appear if item selected (using hidden checkbox that appears on hover): Certain locations in left navigation display no menu options if the location doesn’t have files: |
| Violation | Visibility of System Status, Consistency and Standards |
| Severity | Medium |
| Recommendation | Present a consistent menu bar, and disable any menu options that are not currently applicable |
Cognitive Walkthrough
For the cognitive walkthrough, my team needed to determine the target user profiles for SkyDrive and then create scenarios for these users to perform relevant tasks using SkyDrive.
Based on SkyDrive’s integration of Office web apps, my team decided that two key groups of SkyDrive users would be:
- High school and college students
- Working professionals
We also decided that SkyDrive users could be categorized based on functional role (which might vary from task to task):
- Creator ‒ user that creates, manages, and shares files
- Viewer ‒ user that views files that were shared by another user
- Collaborator ‒ user that views and edits files that were shared by another user
Our team used these user profiles to create these task scenarios for the cognitive walkthrough:
- Task Scenario 1: Ahmed, a college student, needs to create a new SkyDrive account, upload GMAT study preparation files, organize the files into folders, view the files, and take study notes using OneNote.
- Task Scenario 2: Tamika, a high school student, wants to upload a set of photos to SkyDrive, organize the photos by events, create a slideshow of photos, and share an email link to the slideshow with her friends.
- Task Scenario 3: Nandita, a project manager at a mid‐sized company, needs to create a SkyDrive account; create a new Word document and share it with a colleague; view the changes and comments entered in a Word document by a colleague and then modify the file permissions to prevent future changes by others; and upload an existing Excel spreadsheet, rename it, and edit it.
- Task Scenario 4: Jon, a sales representative and colleague of Nandita, needs to update his existing SkyDrive account to change his email, edit a Word document shared with him, notify the original sharer when the file has been edited, and delete a folder containing his personal files.
Each team member conducted an independent cognitive walkthrough of SkyDrive by completing the task scenarios, in order to identify usability issues. We then shared and discussed our results to compile a final set of consolidated findings.
For each task scenario, we described each usability issue with a chart providing the following information:
- Task - task in which issue occurred
- Issue - description of usability issue
- Screen - screenshot(s) showing issue
- Severity - severity of issue
- Recommendation - possible redesign to fix issue
| Task | Viewing Photos |
| Issue | Unclear Navigation: After user has viewed photo, it is unclear how to navigate back to parent folder. User must click “View folder” link near top of right vertical menu. This may be overlooked by users who are accustomed to back navigation link being displayed in top left. |
| Screen | In photo viewer, navigating back to parent folder location requires clicking blue “View folder” link near top of right vertical menu: |
| Severity | Medium |
| Recommendation | Navigation to return to parent folder should be more discoverable. For example, place link labeled “Back to Folder” (with back arrow icon) in upper menu bar next to SkyDrive logo. |
Usability Testing
For the usability testing, my team created a testing script with a sequence of 8 tasks as part of a scenario in which the user is working with photos and documents related to a vacation. Each task was selected to focus on specific functions or features of SkyDrive that were identified as having usability issues during the heuristic evaluation or cognitive walkthrough.
| Task | Task Script for Participants |
|---|---|
| 1 - Upload Folder of Photos | You want to add a folder of photos called “My Vacation Photos” from your computer to your cloud drive. (Participants were provided with SkyDrive account login on computer with existing folder of photos.) |
| 2 - View Photos | You want to take a look at the vacation photos in the cloud drive. Briefly view a large image of each photo. When you are finished, go back to the main screen where you started. |
| 3 - Create New Folder with New Word Document | You want to write about your vacation. Make a folder called “My Vacation Files”. Inside that folder, make a new file called “My Vacation Story” in the cloud drive to write about your vacation. For right now, add the following text as your story: “This is my vacation story.” |
| 4 - Share Link to Word Document | You want to allow some friends to read your vacation story. Send a link to these two friends to allow them to view your story but not be able to change it. (Participants were provided email addresses for link sharing.) |
| 5 - Create Another New Word Document | Inside your vacation folder, you decide to write another new file called “My Next Vacation”. Add the following text to this file: “This will be my next vacation trip.” |
| 6 - Delete File from SkyDrive | You decide to remove the file called “My Next Vacation” from your cloud drive. |
| 7 - Edit Existing Word Document and Share Again | You decide to add to your original vacation story. Add this to the end of your story: “My vacation was very enjoyable.” Resend a link to your two friends to let them know you’ve updated your story. Include a message saying “I updated my vacation story.” |
| 8 - Recover Deleted File | You decide that you want to recover the file about your next vacation trip. After recovering the file, move the file out of the vacation folder, and place it into the main part of your cloud drive. |
My team recruited a total of 8 participants (5 female, 3 male) for the usability testing. We conducted individual testing sessions with each participant:
- Pre-Test: After signing a consent form for the testing session, we asked each participant to complete a pre-test survey with 3 multiple-choice items to assess their level of experience with SkyDrive (none had any prior experience), their level of experience with Google Drive (competing product), and their overall comfort with learning and using software.
- Usability Testing: We asked each participant to follow a Think Aloud protocol as they completed the tasks. One team member acted as the facilitator, while another team member recorded observation notes. We also recorded audio and video of each participant and their computer screen during the testing.
- Post-Test: We administered a post-test interview with 2 open-ended items to ask what the participant liked or disliked about using SkyDrive and to explain why. Then we had the participant complete the System Usability Scale (SUS) assessment based on their experience using SkyDrive. We selected the SUS assessment because it is a commonly-used, validated measure of overall usability.
After each testing session, my team reviewed and discussed the observation notes. As necessary, we also reviewed the participant recording to confirm or clarify usability issues. We then compared the results from all participants to compile a final set of consolidated findings.
For each task, we described each usability issue with a chart providing the following information:
- Task ‒ task in which issue occurred
- Issue ‒ description of usability issue
- Screen ‒ screenshot(s) showing issue
- Severity - severity of issue
- Recommendation - possible redesign to fix issue
| Task | Edit Existing Word Document and Share Again |
| Issue | Read-Only View: All 8 users attempted unsuccessfully to edit document while still in read-only view, which is the default view when a document is opened (which is contrary to how Word desktop application works). 3 users were uncertain about difference in options to “Edit in Word” or “Edit in Word Web App.” |
| Screen | In Word web app, document opens in “read-only” view by default. User must select one of two options to edit document. |
| Severity | Medium |
| Recommendation | If user is document owner (or has editing privileges for a shared file), document should open in edit mode, instead of read-only view, to provide consistency with Word desktop application. Default mode for editing document should be to edit using Word web app. Offer option in File menu to download file (in order to edit offline using Word desktop application). |
Solution
My team compared our findings from our heuristic evaluation, cognitive walkthrough, and usability testing to create a final report summarizing the usability issues discovered within SkyDrive with our recommendations for potential fixes.
The heuristic evaluation and cognitive walkthrough discovered 22 usability issues (severity: 6 low, 13 medium, 3 high). Most of the issues that occurred during usability testing had been previously identified, but there were 2 new issues discovered.
In nearly all cases, users were still able to complete their tasks despite the usability issues — though this may be partly due to motivation to complete the scripted task, whereas in a natural context, users might have given up on the task.
The users’ perception of SkyDrive’s usability varied widely as evidenced by the range in SUS scores (low = 27.5, high = 95). The average SUS score was 67. Generally, SUS scores of 70 or higher are considered acceptable.
Reflection
All three of these usability evaluation methods have advantages and disadvantages relative to each another:
- Heuristic evaluations can reveal a number of potential usability issues across the user interface, but it may not necessarily uncover the most critical issues affecting users.
- Cognitive walkthroughs focus more directly on user tasks, but they are limited by assumptions of how users will perceive and interact with the system.
- Usability testing is the most valid and effective method for discovering issues that impact target users of the system.
Ideally, usability testing sessions would be incorporated into a design and development schedule on a periodic basis (e.g., bi-weekly or monthly) to test new features. Testing with as few as 3-5 participants can be sufficient to reveal most issues.