Category Archives: CUT Group

Results of CUTGroup002: Go2School User Application Test

August 13, 2013Applications, CUT Group, Education, Go2School User ApplicationDaniel X. O'Neil

It’s August in Chicago, and the attention of parents and kids turn back to school. There are lots of changes this year in the Chicago Public School system, with school closings & consolidations and safe passage routes dominating the news and affecting thousands of families.

At Smart Chicago, we don’t play any part in these matters. But since a large part of our mission is to use technology to make lives better in Chicago, we’ve tried to play a small part in making this transition easier. We host and support an app created by local developer Tom Kompare called Go2School, a website that allows you to explore travel options to your Chicago Public School. He came up for the idea for this site while when trying to figure out the best way to get his daughter to school – either by public transportation, car, or walking.

CUTGroup #2/ Go2School, Tester #5

We’ve worked with Tom since the Spring to get server space for his creation, and in late May we tested his app with select members of the CUTGroup, a set of regular Chicago residents who get paid to test civic apps. Following is the results of those test.

Segmenting

In Mid-May we sent two emails to all members of the CUTGroup in order to find the most relevant people with whom to review the site. This complete list of CUTGroup members was 368 people, spread across the city. We segmented on two things:

Must be responsible for taking one or more child(ren) to a Chicago public school this fall
Must be available for testing on either May 28 or May 30

There were 28 people who responded to these requirements. Based on the locations of these people, we decided to hold tests at two libraries: the Kelly Library in the Englewood neighborhood on Tuesday, May 28^th at and at the Uptown Library in the Chicago Uptown neighborhood on Thursday, May 30^th. There were 14 people who expressed interested in each location.

Personas

We conducted eight tests with people. We discovered at test time that one of the participants did not have a child in a Chicago Public School (his daughter did, but she couldn’t make it to the test, so he attended for her. We ended up having a great conversation with him, but his results are not included here). Here’s a look at the general neighborhood locations of the testers, showing wide geographic distribution:

Distribution of Testers for CUTGroup002 / Go2School

The testers were spot-on knowledgable about the topic of getting kids to school on time. In an effort to maintain privacy and to provide the widest possible utility of our testing to civic developers everywhere, we developed some shorthand “personas” that helps in identifying their stance vis a vis the app. We took some video/ screencasts of the tests. Here are some helpful cuts of the feedback. If you click to view the files on Vimeo, you can jump directly to the area of interest.

CUTGroup #2, Tester #2 Father responsible for three children (17, 16, and 10) Samsung Galaxy Kelly Library, Englewood

CUTGroup #2, Tester #2 Father responsible for three children (17, 16, and 10) Samsung Galaxy Kelly Library, Englewood from Daniel X. O’Neil on Vimeo.

0:04: Trouble with Next Button: After a valid address was entered. Would not accept a click, then it did.
0:54: Second Search: Executed with no problems
1:40: Mindful of danger: Schools and families talked about preferred routes
2:31: Jazzing it up: Would be helpful

CUTGroup #2, Tester #3 Public wifi searcher with one child and sometimes one additional child at another location

CUTGroup #2, Tester #3 Public wifi searcher with one child and sometimes one additional child at another location from Daniel X. O’Neil on Vimeo.

0:04: No broadband at home: Heavy user of public wi-fi
1:10: Homepage map confusing: Because it has nothing to do with where she lives.
1:47: Thought there would be more information about the school: Very much likes having custom school info on the site.
2:38: Her usual route is simple. But sometimes she takes a friend’s child to a different school via a more complex route
3:44: Tries that route: Notwithstanding my instinct to do it for her.
4:00: Starting over was a bit of a challenge: Back button shenanigans and back-space/ deleting the entire length of the pre-populated school name.
4:46: The other school she’s interested in is not a CPS school: “So this is useless to my now”
5:30: But she tries a workaround: With the “Explore Map” link.
6:19: Then she found a bug: Noble Street Charter School was geocoded incorrectly in the original data.
6:41: Back to the workaround: In finding a route to a Catholic school.
7:17: Issue with the time picker: She wants to enter the time she should leave in order to get there on time, rather than entering the time she needs to arrive (which is how the system is designed).
7:40: Then somehow the back button was invoked. Don’t really know how. She recovers and again enters the time she thinks she should leave to be there by 8AM.
8:12: Did not know she had to pick a day. And did not readily see the error condition. Also: still wants to enter *departure* time rather than arrival.
9:58: And now the error re: arrival time becomes evident. And that was a bummer.
10:48: This was an excellent user test. Very grateful.

CUTGroup #2, Tester #5 Clock-watching, train-watching mother of three Samsung Galaxy Note II Uptown Library

CUTGroup #2, Tester #5 Clock-watching, train-watching mother of three Samsung Galaxy Note II Uptown Library from Daniel X. O’Neil on Vimeo.

0:04: Error Correction: Chose the wrong school from list; used delete button to erase entire school name and go back to the pre-populated list, where her child’s school popped up again
0:33: Pressing the delete button: Many times, quickly, so as to go back to the pre-populated form with school names
1:06: Missed the “what day” choice: User did not select what day they were traveling and could not move forward in the interface upon clicking “Next” . She chooses “time to arrive” just fine…
1:19: But then clicks “next” repeatedly: Not seeing that there is an error condition (did not choose a day for travel)
2:30: User does not want to see route on map: Because she is so familiar with it. She wanted validation of her routes and clarity on exactly what time she’d get there, and that’s that. She’s happy.
3:12: “It will really help all the parents know what time they should leave the house” — What this website does

Upshots

Beyond this super-specific feedback, there were a number of themes worth calling out:

People seemed to want to have more school-specific information. For instance, they loved the “call school” option featured on the main directions page. People seemed to expect to get custom info about the school they choose in the lookup tool, and were delighted when it was delivered. Many people called this out as a key differentiator from Google Maps or the like
Almost everyone called out the three options— walking, CTA/Metra, or driving— as a positive thing
Lots of people had trouble with the “when do you want to arrive” time / date picker screen. We have some good views of this in Tester #2 at 1:06 and Tester #3 at 7:17, 8:12, and 9:58
At least two of the parents independently called out safety as a consideration in route planning
People seem to really like and use the Chicago Public Schools “Parent Portal” and mentioned it often as a main resource for information

Perhaps the most common of the site’s compliments was the ability to choose which date to leave and which time

All Test Data

Here’s the raw spreadsheet with all of our notes from each of the tests we conducted:

Join us!

Are you interested in joining 500 of your fellow Chicagoans in the CUTGroup? Do it!

[VIDEO] Super-Specific CUTGroup Feedback on Go2School App

We are preparing our full take on CUTGroup tests we did on Tom Kompare‘s Go2School, a website that allows you to explore travel options to your Chicago Public School., but I wanted to share some great cuts we took from one of the testers from Tuesday, May 28^th at the Kelly Library in the Englewood neighborhood. Take a watch— it’s really worthwhile. If you view the video on Vimeo, you can skip direct to each part.

CUTGroup #2, Tester #3 Public wifi searcher with one child and sometimes one additional child at another location

CUTGroup #2, Tester #3 Public wifi searcher with one child and sometimes one additional child at another location from Daniel X. O’Neil on Vimeo.

Volunteer at Smart Chicago: Be a CUTGroup Test Proctor

July 31, 2013CUT Group, VolunteersDaniel X. O'Neil

Chicago Health Atlas Homepage, June 2013 Launch We need volunteers to help us with our next few CUTGroup tests. Here’s the commitment:

Help us out for 2 hours, 6-8pm, sometime during the week at a public computer center somewhere in Chicago
You will be working with CUTGroup testers to review a website about restaurant inspections
You’ll be completing some simple forms to gather the tester input and overall results of the UX test

We need testers from all over the city in both English and Spanish.

There is no compensation for being a CUTGroup Test Proctor, but it’s a great opportunity for developers, designers, and UX specialists to get in front of real Chicago residents using honest-to-goodness civic technology.

Interested? Complete this simple form and we’ll take a look.

Results of CUTGroup001: FreedomPop Router Test

July 26, 2013Broadband, CUT Group, FreedomPop Survey #1Emily Escarra

Introduction

EveryoneOn is program of Connect2Compete that offers low-cost 4G internet connection in select Chicago zip codes. Smart Chicago Collaboration tested the signup, unboxing, and service prior to the EveryoneOn launch event in Chicago, of the Freedom Pop service offerings. Since expanding access to the internet is one of the core missions of Smart Chicago, we decided to come up with a process to use the CUTGroup to test the hardware, customer workflow, and coverage for this product.

Through this project, we were able to more deeply understand the market for at-home Internet access, to provide actual access devices for residents who need it, and get real data on how well the Clear network (the underlying service for FreedomPop) serves Chicago.

Devices

FreedomPop Devices, Chicago

The test was centered around the FreedomPop Hub Burst — a combination 4G modem and wireless router that for both office and home settings. It connects up to 10 wi-fi-enabled devices to the internet with a secure connection. Speeds of up to 8Mpbs download. Signal reaches up to between 100-150 feet, depending on interference. Devices were paid for by EveryoneOn.

Segmenting

At the end of March and beginning of April 2013, two emails were sent to CUTGroup members in the zip codes where the lowest cost EveryoneOn option is available : 60602, 60604, 60608, 60609, 60612, 60615, 60616, 60651, 60619, 60621, 60623, 60624, 60626, 60636, 60637, 60640, 60644, 60649, 60653, and 60660.

We use Mailchimp to manage outbound communication with testers. Here’s the first email: (“Participate in a 4G Internet Access Test and Get a Free Router“) and the second one. We had a total of 94 CUTGroup members in these zip codes residing in these zip codes. Eight qualified CUTGroup members responded to our query and were a part of the test. We then added participants gathered from other relevant sources, including partners in City government and members of our Connect Chicago Meetup group. We tested a total of 19 devices with 19 different people. Here’s a look at the distribution area for testing of the devices:

All of the devices were free to the user, with the knowledge that each participant would be subject to the normal terms of being a FreedomPop customer. 13 testers then completed this form (“4G Internet Access Test Survey“) and we compiled the results. Six testers gave feedback in a less formal way (email, phone, or in-person conversation).

Results

Receiving the device was unboxing it was a breeze.

All participants received the device, packaged and delivered as promised. No one reported difficulty in opening the device.

Setting up the device was easy.

The majority of participants reported little difficulty setting up the device. The majority of participants reported difficulty when trying to set up the actual service itself. The test was intended and advertised as a free trial, but individuals were incorrectly prompted for credit card information. When entering information, data fields did not populate correctly. Changing browsers did not fix the above issues.

As one tester from Uptown noted,

“Setting up the device was very easy. Setting up an account was difficult.”

Connecting the router to the Internet was easy.

For the majority of users, the connection to the internet was immediate.

Internet speeds were comparable to other Internet connection products.

Every participant able to connect reported equal quality of service to other devices used in the past.

All participants used a laptop as a primary device and most used a smartphone as a secondary device.

All participants used a laptop as their primary source to connect to the internet (example models: Macbook Pro and Macbook Pro Air, Toshiba Satellite C655, IBM T410, and Dell Inspiron).

The majority of participants report using their smart phones and tablet (models include: HTC Inspire, iPhone 4s, and the iPhone 5)As a back-up method of connecting to the internet.

All testers had a positive experience with the CUTGroup itself

100% of participants had complimentary feedback on the CUTGroup program and process in general. All either would participate or consider participating in the future.

This was our first CUTGroup test and our first time presenting the results in public. We’re devoted to sharing our process, methods, and results with the civic innovation community so that we can all learn how to make better tools and create better experiences on the Web.