Technologies for Creative Expression

Originally written for the National Endowment of the Arts Special Taskforce on Arts and Human Development
NEA Art Works blog post | Webcast archive of my public presentation hosted by the NEA

When I was 7 years old there were a lot of art materials around the house. Blocks, clay, paint, paper—I loved to make things. My grandmother was an artist and my grandfather an inventor, and I had caught the bug. I would squeeze and sculpt Sculpey, a polymer clay my grandfather had created, draw new creations in my notebook, and learn math by creating geometric patterns with my Cuisinaire rods.

I grew up and kept playing with the materials and tools that I loved. At Yale college I studied sculpture and conceptual art, and through an artist internship helped create a new children’s toy called ZOOB, which is the first construction toy based on things that grow. Children could create with ZOOB and reflexively learn about patterns and processes from nature, like how proteins fold and how skeletons move. They could bring their creations to life in their imaginations as they discovered what a century of life scientists had documented in papers and books. But while kids loved ZOOB, I noticed something was changing. It was the late 1990’s and more and more of children’s learning and play was being directed by electronic toys and media which followed predictable scripts–the buzzing of a horn, the sound of an alphabet–and didn’t allow for children to tell their own stories.

So I went back to school to learn how to create interactive media that support people’s creative expression. At the MIT Media Lab I joined the tangible media group and invented a series of interactive media and tools. Topobo is a constructive assembly system with kinetic memory. A construction toy in the spirit of ZOOB, but one that children can literally bring to life with a twist of their wrist. A child can make a dog, move its body in their hands, and watch the dog dance or walk, mimicking the motion they have taught it to do. Topobo helps kids learn new things. The same way stacking blocks helps kids learn how buildings stand up, Topobo helps them learn how animals walk, and how form is connected to motion. Kids know about this when they are little – they remember learning to crawl and walk – but usually we don’t give them tools to think about it until they are in college. With new toys like Topobo, they can start playing with these ideas when they are 4 or 5, and discover new patterns as they grow older.

As I was getting Topobo out to schools and museums around the world, my wife and I became proud parents of a newborn daughter. With her came the sad discovery that one cost of being more “mobile” was living far from family and being disconnected from them. Frequent trips from Cambridge, MA to California were tough, so I started making playthings for my baby which helped her get to know her far-away grandparents. Stuffed animals held familiar people’s songs; a photo album of her grandparents had a phone number they could call to leave stories, songs and messages. My daughter just had to touch their photos to hear them replay. But I didn’t know how to bring a deeper sense of connection to lots of people.

I moved to Nokia Research in Palo Alto, CA to make “Family Communication” the focus of my work. In 2008 we joined with Sesame Workshop and created a series of tools for families with young children to have stronger relationships even when they are apart. Storyvisit was one such tool, a free web site that coupled children’s story books and video chat. Families could hear and see each other, and also had something fun and understandable to do together: read a book! Children, sitting with their parents, could read along with a far-away parent, grandparent, aunt or uncle and get to know each other while they enjoyed some good books. Reading books was something everyone knew how to do, and something that could spark conversations. Tips for grown-ups would help them use the book to launch into conversations with the children. This is great both for kids’ learning and for family togetherness. Our feeling was that there is no one better to learn from than the family who loves you the most, and we could use technology and media to bring families together and help them to tell their own stories to each other. It was a success: we documented hundreds of happy families whose conversations went from 2-3 minutes on an ordinary video call to 15-20 minutes with StoryVisit. Ten times more time with the three year old you loved meant that much more time to know and love each other. Since then video chat technologies have gotten better and a few startups like Kindoma are working to bring this idea to families everywhere. 

As I was wrapping up this work I got a call from Google[x], at the time an almost unknown department of Google that a friend described to me as “the closest thing to Willy Wonka’s Chocolate Factory in Silicon Valley.” Curious, of course, I went and met Astro Teller (“captain of moonshots”) and heard about the vision to create technologies with lasting cultural impact. I knew little about the details, but decided to take a chance and join the team. The small series of projects included the self driving car, and a project in wearable computing which has become Google Glass. The project challenged me, because it was so very intimate and personal in its nature, but also opened an opportunity for empathy through the wearer’s ability to share their point of view with people who are far away. The magic of a high quality camera that was ready-at-hand spoke to the artist in me, so I dedicated myself to making it a tool that people could use to tell their own stories.

For me, Glass is a platform for empathy and creative expression. Some of my favorite stories are from activists, parents, artists, and teachers who are using this new form of photography to share their experiences with people they care about. There are journalists using Glass to support democracy, such as when Tim Poole did live broadcasts of protesters in the streets of Instanbul. (Tim has been doing journalism for years – the difference with Glass is that when he talks to people they are talking to him and not to his camera.) There are artists like David Daytona using Glass to embed stories into their traditional artworks and give audiences a new point of view into his artistic intent. There are mothers using Glass to help far-away grandparents get a mom’s-eye-view of their newest granddaughter learning to lift her chin and smile at someone she loves.

Technology is changing so fast, and it’s hard to know exactly where it will lead. But I’ve always been inspired by Alan Kay’s quote that “the best way to predict the future is to invent it.” I hope to continue to bring an artist’s perspective to its evolution, so that the technologies we create do in fact become platforms for empathy, creative expression, and human connections.

Here is a public presentation of these ideas, presented by the National Endowment of the Arts.

Future of Play

As a researcher At the MIT media lab I have been pursuing my vision to create interactive toys that people bring to life through play and learning. My expertise is in designing products that navigate the boundary between the physical and digital worlds, building on over ten years inventing and bringing innovative designs to market. My work involves conceiving new ideas for interactive media, and helping manage prototyping, testing and marketing of those ideas. My goal is to reinvent children’s interactive technologies to reflect children’s passions, creativity and encourage collaboration.

Topobo moose

Topobo is a 3D constructive assembly system with kinetic memory, the ability to record and playback physical motion. You can build a dog with Topobo, wiggle its body around with your hands and teach it to walk. The dog will then repeat your motions repeatedly. The same way kids can learn about static structures playing with blocks, they can learn about dynamic structures playing with Topobo. In this video interview with the Science Channel I explain the system in more detail.

Topobo integrates all of the things I love about design – a clean and emotionally engaging concept, strong aesthetics, clear interaction and enough flexibility for the user to put their own personality into the product. Furthermore, designing Topobo has been an amazing collaboration, and I have been able to learn skills from product designers, graphic designers, electrical and computer engineers, educators and kids. We have tested Topobo in classrooms with kids ages 5-14 throughout the design cycle. These interactions were enlightening for us, giving us insight how others made meaning of the system, seeing people build and animate creations we’d never imagined, and getting feedback about how to improve Topobo in future iterations. To our delight, both 2nd and 8th graders told us they thought Topobo was designed for them!

Our work led to four peer-reviewed conference papers, an ID Magazine Design Distinction, a Prix Ars Electronica Honorable Mention for interactive art, and numerous international shows in art, design and education. The system has also had a mature design cycle. In 2004 I received a Microsoft iCampus grant to conduct longitudinal research studies, and pursued a complete two-year redesign of the system including production manufacturing in Hong Kong. My experience conducting design work at the Chinese factory has given me a deeper understanding of the full design and manufacturing cycle of interactive media products.
Kids playing with Topobo

Jabberstamp - girls drawing+recording

Jabberstamp is the the first tool that allows children to synthesize their drawings and voices. To use Jabberstamp, children create drawings, collages or paintings on normal paper. They press a special rubber stamp onto the page to record sounds into their drawings. When children touch the marks of the stamp with a small trumpet, they can hear the sounds playback, retelling the stories they have created.

Children ages 4+ use Jabberstamp to embed names, narratives, characters’ voices and environmental sound effects in their original drawings. Children’s compositions help them communicate their stories with peers and adults, and allow them to record and situate stories in personally meaningful contexts before they have mastered writing.

I originally stumbled upon toy invention through an artist internship that helped shape my undergraduate sculpture studies at Yale. One summer I helped artist Michael Joaquin Grey conceive and design a new building toy called ZOOB, a haptic modeling system that allows people to reflexively learn about biological structures like bones and molecules. I later helped Michael start up a design and manufacturing company to bring ZOOB to market.  ZOOB was a great success—we did over $3M in our first year, and won a number of awards including Dr. Toy Best Toy, Toy of the Year and ID Magazine Design Distinction for consumer products. Over the next several years I directed the company’s internal design department, overseeing all of the company’s print and product design.

After ZOOB, my independent art and design work led me to the MIT Media Lab, where I joined the Tangible Media Group to develop my technical and conceptual skills developing electronic media. My early work at MIT was driven by an ambition to create new product designs that engage people’s kinesthetic sense and love of play to support communication, entertainment and learning.

Super Cilia Skin
Super Cilia Skin is a literal membrane separating a computer from its environment. Like our skin, it is haptic I/O membrane that can sense and simulate movement and wind flow. Our intention is to have it be universally applied to sheath any surface.  As a display, it can mimic another person’s gesture over a distance via a form of tangible telepresence. A hand-sized interface covered with Super Cilia Skin would produce subtle changes in surface texture that feel much like a telepresent "butterfly kiss."  

Our extensive design studies spanned dozens of physical and electronic prototypes and led to a technical paper in CHI and a full journal paper in "Textile: The Journal of Cloth and Culture" (Berg 2004). The investigation focused on the haptics and scale: as a product, Super Cilia Skin is a "touch telephone." As wallpaper or carpet, we considered the potential for actuated, telepresent interior design. At the tectonic scale, sheathing tall buildings, the interface assumes dual roles as billboard size display or as wind-driven electromagnetic power generators.

The You’re In Control system uses computation to enhance the act of urination. Sensors in the back of a urinal detect the position of impact of a stream of urine, enabling the user to play interactive games on a screen mounted above the urinal.

In order to allow men and women to publicly interact with YIC, we built a customized game controller that can be strapped around the waist and pressurized to create a stream of water. Players can toggle between the "Flying Hamsters" and "You’re A Nation" games by flushing the urinal. It really is a pissing contest!

You’re In Control (Urine Control) interactive gaming system

About fifty years ago my grandfather invented a polymer clay called Sculpey and reinvented how children can create and express themselves. His life as an inventor and entrepreneur inspires me to imagine how electronic toys can be more than flashy gadgets or media conduits, and actually amplify children’s creativity, expression and playful learning.

+Hayes Raffle

{ contact | cv | portfolio }