OVERVIEW
The Cooper Hewitt Smithsonian Design Museum is both an innovator and propagator of change and creativity. It is the only museum in the United States that has dedicated its curation to historic and contemporary design ("Cooper Hewitt, Smithsonian Design Museum," n.d.). It is also one of the first museums to embrace opening up its collection digitally to the public and develop an Application Programming Interface (API). The institution trusts that their API will be used as a design tool to inspire generations to embrace their own creative pursuits and develop new digital experiences and applications.
In doing so, Cooper Hewitt is committed to making whatever iterations necessary to improve its API for developers and to those who are just entering the world of programming. To test the API interface usability and enhance its user experience, Our team of UX Designers/Researchers from Pratt Institute’s Information Experience Design (IxD) program collaborated with the Cooper Hewitt, Smithsonian Design Museum.
CHALLENGE
Our ambition was to create a more feasible API interface that could support the needs of both experienced developers and newly ambitious developers alike. We wanted to determine the API experience for users with varying degrees of programming experience to create harmony between experienced engineers' standards and the lessen the challenges faced for those with less experience.
OUR TEAM
MY ROLE
I was one of four usability students entrusted with testing and analyzing the usability of Cooper Hewitt's API. Within 6 weeks, I:
​
-
participated in client and team meetings
-
scheduled / recruited / moderated remote user tests with participants
-
documented and synthesized qualitative usability data
-
Created a usability report
-
delivered our findings and recommendations to our clients from Cooper Hewitt
OUR PROCESS
DEFINE
KICK- OFF
During the kick-off meeting, we got a chance to learn what an API is, what Cooper Hewitt's API could be used for, and who can use it. This meeting defined our vision and the target users that we would later recruit for our study. During the session, the stakeholders mentioned that the "API is a design project" used by developers and creative technologists alike. However, they also expressed their vision for their API's future and their interest in how it could be used as a learning tool for newly ambitious creatives and developers.
GOAL :
How does Cooper Hewitt's existing API attend to the needs of both experienced and novice programmers, and how can we create a great user experience for both audiences?
PERSONAS
The insights given from the kick-off meeting identified our focus user groups as being someone with a strong familiarity of APIs and using them and someone who may just be starting out on their programming journey and has less understanding of APIs and their use. Two personas were created to represent these groups:
PLAN
Our team created a moderator script for our usability testing sessions to keep the study organized and focused on our goals. The script included pre and post-test questionnaires along with the tasks users would complete during the usability testing.
PRE-TEST QUESTIONNAIRE
The pre-test questionnaire was administered to participants to learn more about their experience with APIs and what they have used them for in the past.
TASKS:
TASK 1:
Please go to the provided link (https://collection.cooperhewitt.org/api/) and explore the website. What are your first impressions?
PURPOSE:
Gain perspective of what a user's first impression with the visual design of Cooper Hewitt’s API is.
TASK 2:
Please get an access token for a new application. If prompted, you may use the following login credentials: Email: chusabilitytest51@gmail.com Password: !Test1234
PURPOSE: Test the ease of the APIs on-boarding
TASK 3:
Imagine that you want to use an API endpoint that will allow you to search the items in the museum’s collections. Find the information you need to use this API endpoint.
PURPOSE: Determine the ease of finding endpoints and method documentation
TASK 4:
For the final task, please use the “cooperhewitt.search.collection” API endpoint and search for items acquired in 1991 that are the color red.
PURPOSE: To see how users use end-point parameters to receive an API call and get impressions on their response of what was returned.
POST-TEST QUESTIONNAIRE
The post-test questionnaire aimed to collect information about the users' experience with the API and the tasks given.
PILOT TEST
Once all testing materials were completed, our team ran a pilot usability test with a senior software engineer to see if we needed to make any changes or alterations to our study and its corresponding tasks.
RECRUIT
PARTICIPANT RECRUITMENT
Our team issued a screening questionnaire to industry software engineers and graduate students within Pratt's school of information. Those interested were screened based on their level of expertise with programming and their familiarity with APIs. Respondents rated their knowledge on a scale of 0-5.
​
-
Those with no experience either with APIs or programming were ineligible.
-
Those who self-assessed their programming skills or familiarity with APIs as a 1 or higher were suitable for this study.
-
8 respondents with various degrees of programming skills and familiarity with APIs were chosen to participate and sent consent forms.
-
50% entry level programmers, 50% experienced programmers
TEST
METHODOLOGY
To conduct the usability tests, we utilized moderated remote user testing methods. Moderated remote user testing is a synchronous form of usability testing and is conducted within a controlled environment. This method consists of recruiting participants and recording their on-screen engagements as they complete designated tasks issued by evaluators. The moderated remote user tests for this study were conducted and recorded through the screen-cast software, UserZoom GO. Each test was moderated by one member of our team and observed by another team member. Participants were encouraged to use the think-aloud method while completing tasks to understand better their decisions and any pain-points surrounding a task, if any.
DATA ANALYSIS:
QUANTITATIVE:
QUALITATIVE:
We selected usability findings that were experienced by the highest number of participants, focusing on findings with 5 or more occurrences.
Usability findings were grouped by similarity then evaluated by severity. We determined the level of severity based on the amount of time that participants needed to recover from a hurdle and the level of frustration they experienced when they encountered that problem.
DISCOVER
COOPER HEWITT MADE A GREAT FIRST IMPRESSION...
"I like the halloween theme"
"I appreciate the sidebar for easy shortcuts"
"Color coding of the page is visually pleasing, straightforward and well presented"
Found playful language like “Make it so” and “play nicely” amusing and funny
THEY EVEN MADE A GOOD POST-IMPRESSION TOO
DISCOVER
USABILITY FINDINGS & RECOMMENDATIONS
FINDING 1: Parameters and their descriptions are ambiguous
1.1
75% of users accidentally skipped over the method documentation page and jumped to "test this method in your browser" link
​
Recommendation:
Remove "test this method in your browser" link from the methods page. This will allow users to read the methods documentation and important notes on endpoint parameters first.
1.2
Users initially viewed the descriptions within the parameter input fields as helpful until they noticed that some descriptions were cut off due to length. The ellipses' presence added frustration to users as they could not view what would have been a helpful context for what to input within these fields.
​
"oh, that's not very helpful" - one participant when they noticed the rest of the text was cut off
​
Recommendation:
Provide succinct and complete descriptions within parameter input fields to subdue user frustration and confusion.
FINDING 2: Method documentation lacks information on response object
Experienced programmers were shocked once they received the response after submitting their API call, while less experienced programmers did not know what to expect.
​
"I didn't know what the response would look like" - participant from study
​
Recommendation 2.1
Provide a sample success and error response object in the method documentation as point of reference for users before they begin working with API methods. We suggest implementing a model like the one provided by the Smithsonian Institution's Public API. As shown below, their API provides a single success response and identifies two error responses that the user may experience.
In addition to displaying the shape of the response object, users may benefit from having a description for headings and titles returned in the response object. We suggest employing descriptions for data types and definitions of the response fields like the one provided by the Art Institute of Chicago.
Recommendation 2.2
Consolidate the method documentation and testing feature on one page to allow users to reference parameter definitions and restrictions quickly. By providing documentation and testing ability on a single page, users could easily modify, update, and iterate their searches. We recommend creating a selecting menu to add input parameters rather than displaying all possible parameters in a list.
​
FINDING 3: Confusion around similar endpoints
7 out of 8 participants expressed sentiments of uncertainty surrounding the closeness and redundancy between endpoints, and as a result, they had trouble distinguishing between them. Endpoints that had terms like "objects in our collection" and "things in our collection" were hard for users to differentiate. As well as between endpoints like 'cooperhewitt.search.collection,' 'cooperhewitt.search.objects,' and cooperhewitt.search.objectsFaceted'
​
"I’m a little confused because this one says ‘things in our collection’ and the other one" - participant from study
​
"I would like to know the distinction between what ‘things’ and ‘objects’ are from their
(the museum’s) perspective, and what faceting them means." - Participant from study
​
Recommendation 3.1
Eliminate or change the name of similar endpoints. In the mock-up below we eliminated the 'cooperhewitt.search.objects' endpoint to reduce redundancy and confusion.
Recommendation 3.2
Include a clear description about the endpoint and what items can be found using them.
​
FINDING 4: Difficulty of experiencing the top level navigation and sidebar
While completing the tasks, more than half of the participants described difficulties distinguishing the functions of the various navigation menus on Cooper Hewitt's interface, namely the sidebar menu, main menu, and API "Quick Reference" menu.
​
Finding 4.1 : Navigation through the sidebar does not provide any feedback
When participants were asked to find more information through the side-bar navigation, they were unable to receive feedback that the page changed to the desired page they clicked on. participants stated that they needed a visible and distinct signal that could quickly tell them what page they are on. One of the participants cited a desire for the current page to be highlighted in a different color in the side-bar. One participant responded:
​
"Introduction to the Cooper Hewitt API’ needs to change a different color”
​
Recommendation 4.1
Add visual feedback on the sidebar so that users can know what page they are on. We suggest highlighting the page title that the user is on in the sidebar menu.
​
​
​
Finding 4.2 : Navigation of the API method "Quick Reference" menu is too long
All users mentioned that they liked the quick reference menu and thought it would help alleviate the exhaustion of scrolling through the entire documentation of possible endpoints. However, those who used the quick reference menu to search for a specific endpoint found it difficult and tedious to see what they were looking for. Novice API users made comments about the quick reference menu saying things like:
“there are too many links”
​
“oh, this is long.”
​
Recommendation 4.2
Adding a search function right under the quick reference header will make the API more accommodating to users who are less experienced using API’s and reduce exhaustive scrolling. With this feature, users may be able to type in keywords for what they are looking for and the search function can return endpoint’s that relate to their search query.
​
​
​
Finding 4.3: User journey to get to the API page is too complex from the Homepage
Very few participants could find the “Collections API” under the “Explore the collection” grey menu bar. From the Cooper Hewitt home page, there is no visible API section in the homepage menu. The users would have to go to "collections" and then click the "explore collection" tab. Once the user does this, they would need to switch to another menu to find “collection API”. It is a pretty complicated user journey to find the API page from homepage.
​
Recommendation 4.3
Add API section within the main menu. This change would help Cooper Hewitt Museum to provide direct information and guide those who desire to learn this new technology in the museum industry
​
​
​
​
Recommendation 4.4
On the API methods page the "Collections API" tab is buried within the menu. We recommend Restructuring the navigation bar and adding a "Collections API" as its own section on the navigation bar.
CLIENT FEEDBACK
On December 9, 2020, my team and I enjoyed delivering our usability findings and recommendations to the Digital Project Manager at Cooper Hewitt and the Museum Experience Consultant, who is working alongside them to transform their API. Our clients were excited to hear our advice and recommendations on how to improve the API interface. They loved our team's custom-made virtual backgrounds on zoom. The clients were very eager to learn more about how the API can be more accomodating to novice programmers and asked questions about their experience with the API. Chad, the Museum Experience Consultant, declared that our report was "super helpful" in showing how the API documentation could be an engaging learning tool for a broader audience.
Cooper Hewitt Usability team after client presentations
NEXT STEPS
To further enhance Cooper Hewitt's API interface's user experience and usability, more testing should be conducted even after changes are implemented. It is essential to test out the new functions on users. If time permitted us to do so, It would have been nice to include more participants who lack experience with API's because we found that a lot of the user data from their tests were especially helpful in determining usability issues and improving the site. Overall I am excited to see how Cooper Hewitt iterates their API in the future.