SMART CHICAGO IS MOVING!!!

Good News!!! The Smart Chicago team is moving and now will be co-located with the City Digital Team at UI Labs. As such, our individual emails will be changing to:

Kyla Williams           

Sonja Marziano       

Denise Linn               

Leslie Durr               

Our new mailing address is 1415 N. Cherry Avenue Chicago, IL 60642 and general phone number is 312.281.6900.

Please check our website at smartchicagocollaborative.org or follow us on twitter @smartchicago for more updates.

We appreciate your patience during this time of transition.

Thoughts on the Knight Community Information Challenge: Design Thinking and Learning Together

Today I was on a panel at the Knight Foundation Media Learning Seminar along with colleagues from three other foundations, along with the Chicago Community Trust, who have formed a cohort doing work around the principles of design thinking. We’ve participated in a number of workshops and we’re sharing our deep dive with others. The hope is that we can move forward the practice of community foundations as they discover and serve the information needs of communities.

Susan Patterson, Program Director at the Knight Foundation, is moderating the panel and she sent along some prompts. This post lays out some thoughts as my primer to the panel.

At Smart Chicago, we’re in a unique position. We’re housed at The Trust, which is the actual partner for the Knight Community Information Challenge (KCIC).  The Trust provides matching funds and has been deeply invested in this work, long before Smart Chicago even existed, centered around the journalism ecosystem in Chicago. You can see the river of work that went into that here. There’s also a complete evaluation of the program: “News that Matters: An Assessment of Chicago’s Information Landscape”.

Smart Chicago has been doing work under the KCIC banner since 2012, starting with our Civic Works program, designed to spur support for civic innovation in Chicago. Christopher Whitaker has led that project for us, and it has been hugely successful, helping projects like Roll With Me (accessible transit directions in Chicago), mRelief, (a text-based way to check your eligibility for benefits in Chicago & Illinois), and. We’ve used it to give seats to innovators at 1871 and support collaboration between local government and emerging companies like Textizen.

Our current project under KCIC is the Deep Dive. There are four components; two to expand existing programs for engagement and two that are brand-new:

  • Support for the 2015 On The Table— a community-wide conversation involving more than 20,00o people in a single day
  • Expansion of the Civic User Testing Group,  a set of regular Chicago residents who get paid to test out civic apps. This has allowed us to add hundreds of testers, expand the program to all of Cook County, and conduct more tests
  • Our project on Experimental Modes of Civic Engagement in Civic Tech,led by Laurenellen McCann, which includes deep research and explication of community-driven processes for creating technology with real people and real communities
  • Lastly, our Un-Summits, which are mass neighborhood convenings around digital skills and data

One of the coolest things about this deep dive cohort is the support and structure that the Knight Foundation has fostered. There are a number of components.

We’ve gotten together, as a group, three times now to learn specific methods for design thinking— formal methods problem-solving. The best thing I’ve learned on this is interview tactics— how to find hyper-users, how to design a questionnaire, and how to conduct interviews that yield actionable information.

It’s the specific skills— not aphorisms, anecdotes, or notions— that I really value.

We’ve also done some online collaboration, which frankly hasn’t worked all that well. I’ve learned that choosing and imposing new software or processes on people never really works. It’s the genuine expressions of an organization that really make an impact. Again, that’s why I value specific modes, specific actions, that are genuine.

Lastly, I really appreciate the work of ORS Impact— the formal evaluators of our program. They’ve created a theory of change that really makes sense and pulls together a set of threads into a coherent narrative.

We look forward to continuing our work in this cohort. In fact, we’re hosting our next meeting here in Chicago in September.

KCIC Inspiration Workshop in San Mateo, CA

KCIC Inspiration Workshop in San Mateo, CA

Here’s the panel in full:

KCIC Deep Dive Presentations; Design Thinking and Learning Together
Moderators: Susan Patterson, co-director, KCIC, Knight Foundation
Panelists: Daniel X. O’Neil, Chicago Community Trust/Smart Chicago Collaborative; Kelly Ryan, CEO, Incourage Community Foundation; Emmett D. Carson, Ph.D., CEO, Silicon Valley Community Foundation; Chris Daggett, president & CEO, Geraldine R. Dodge Foundation

CUTGroup #12 – Roll with Me

For our twelfth Civic User Testing Group session, we tested the Roll with Me, a website that helps residents find accessible transit directions in Chicago. This test had two components— a week-long remote test with an in-person discussion at our offices – The Chicago Community Trust at 225 N. Michigan Ave in the Loop.

Roll-with-Me-Screenshot

Mohammad Ouyoun developed Roll with Me during an internship with Smart Chicago and as part of the Civic Works Project. We sat down with Mohammad to see what he wanted to learn from the test. We focused on individuals who have difficulties with stairs, and Mohammad saw use of this website for people that have temporary needs such as parents with strollers, or individuals using carts or luggage. Here are the things that we really wanted to know:

  • How does Roll with Me work in user’s normal day-to-day? Is it easy to use on different devices?
  • How important are the alerts for users?
  • How does Roll with Me compare to other user’s methods of finding transit routes? What features are users interested in having?

Segmenting

On January 8, we sent out an email to all of our 837 CUTGroup participants. We wanted to know if they would be willing to answer a short remote questionnaire 2 or 3 times during a week, and then be able to come to an office in the Loop for an in-person discussion on January 20, 2015. We asked some screening questions to gather information. We wanted to focus on testers who have difficulties with stairs, and ended up with 7 testers who said they did. Since we were looking for more testers, we also reached out to an additional 8 testers who were heavy users of public transit to provide feedback about this app in comparison to other transit apps and websites.

We ended up having 14 testers participate in this test from different neighborhoods in Chicago. Here is a look at the neighborhoods distribution of the testers:

Screening Questions

We heard form 95 CUTGroup participants through our callout for testers. We received a lot of good information just from the screening questions. Here is a look at what we learned:

  • 67% of CUTGroup respondents use public transit on a daily basis.
  • 18% of respondents use public transit to travel to the suburbs on a regular basis (2 or 3 times a week)
  • 23% of respondents specifically mentioned using CTA tools (such as CTA Bus tracker) as their primary way of getting transit route information. 21% of participants mentioned Google Maps as their primary way to search transit routes.

Test Format

We asked testers to participate in multiple test parts over a week’s time. We were happy with the level of detail respondents provided in the quantitative remote portion of the test .

14 of the 15 testers we invited answered Part I’s questions via an online form. A couple of days later, 11 of those 14 testers also completed Part II questions. Then on January 20, 2015, 10 of the 14 testers were able to make it to the in-person section.

We wanted to have an in-person discussion that was casual and focused on qualitative responses. This was not a traditional focus group. The goal of this discussion was for Mohammad to ask questions directly to testers and get feedback. Through this format, we were able to discuss very specific features that testers were looking for.

Results

How does Roll with Me work in user’s normal day-to-day? Is it easy to use on different devices?

We learned that 71% of testers normally check directions on their mobile device. When talking about how it was to find directions, 50% of testers mentioned that it was simple, easy, or straight-forward.

Only 5 testers tried the “Find Me!” feature. 64% of testers did not try to use it and did not see it during their use of the website. Currently the “Find Me!” feature is not working, but once it is fixed, testers thought it should be larger and more prominent.

3 testers experienced problems with searching for their address due to the fact that their addresses came up in different cities rather than defaulting to “Chicago, IL.”

Also, testers were frustrated that when searching for directions they could not go back and edit the directions, but instead had to retype all of the information again.

How important are the alerts for users?

Roll-with-Me-Alerts-Screenshot

The majority of testers found the alerts to be very useful information, but a few testers thought that they were distracting from the route information. 3 testers thought it would improve the website if the alerts were moved from the top, center location.

Wheelchair CTA rider (#4) says, “I EXTREMELY like that the accessible info is available for the whole CTA system on the same page.”

During the in-person discussion, Writer Rider (#12) explained that she liked having all of the alerts because it brings attention to accessibility needs and the fact that routes might be unavailable due to construction or maintenance.

How does Roll with Me compare to other user’s methods of finding transit routes? What features are users interested in having?

During the first part of the questions, 5 testers (36%) wanted to see a map associated with their directions. 7 testers (50%) wanted to be able to input a custom travel times to get directions. The 15-minute intervals were not sufficient for testers.

For the second part of the test, we asked how important these features were to all of the testers:

How important is it for Roll with Me to display your directions on a map?

5 Very Important         9% (1)
4 Important                 55% (6)
3 Neutral                       27% (3)
2 Slightly Important    0% (0)
1 Not Important           9% (1)

How important is selecting a custom departure time when finding directions?

5 Very Important      55% (6)
4 Important                  36% (4)
3 Neutral                       9% (1)
2 Slightly Important   0%
1 Not Important          0%

How important is selecting a custom arrival time when finding directions?

5 Very Important     55% (6)
4 Important                 27% (3)
3 Neutral                      18% (2)
2 Slightly Important   0%
1 Not Important          0%

Roll with Me for Other Uses

We wanted to also see if Roll with Me could be used for residents who do not have mobile disabilities.

Blue Line + Buses Only (#10) says, “I think the name discourages people who might not be in a wheelchair but might appreciate accessible travel accommodations (ppl. uncomfortable taking stairs, who can’t go far distances on foot, etc.)”

How useful do you think Roll with Me is for people without mobile disabilities? Such as people with strollers, carts, or luggage.

5 Very Useful              27% (3)
4 Useful                       27% (3)
3 Neutral                    36% (4)
2 Slightly Useful         9% (1)
1 Not Useful                0%

Currently, Roll with Me does not specifically target these groups of people, but some testers did see the use of Roll with Me for people without disabilities.

Next Steps

Directions need to be easier to find.

We heard a few suggestions from testers to make finding directions easier:

  • Find me! Feature needs to be fixed and more prominent on the website.
  • Testers tried using cross streets to find transit routes, which either produced incorrect results or did not show any results. One suggestion might be to provide examples of how the address should be written on the homepage.
  • A lot of testers faced the same issue that when they started to type in their address, addresses started to show for other cities before Chicago. Since this app is limited to accessible transit routes in Chicago, testers wanted it to default to Chicago without having to type in city and state info. In addition, there was a bug when testers did not type in the full address.

Adding additional features

  • 90% of testers thought it was “important” or “very important” to have custom arrival and departure times when finding transit times.
  • 63% of testers thought it was “important” or “very important” to have a map to display directions.

We learned that most testers use other websites, such as the CTA Bus Tracker or Google Maps, and are looking for similar features in Roll with Me.

Final Report

Here is a final report of the results with the analysis of the questions we asked, followed by each tester’s responses, and copies of other questions we asked:


The raw test data can be found below with the written answers from every tester.