Challenge & Research

Need

Children are drawn towards the emergence of new media and technology. They are eager and savvy to explore new modes of tools and toys. New media has enabled them to communicate and get connected with both peer (siblings, cousins, friends) and mentor (parents, grandparents, teachers). There is a drastic change of physical (body, arms, hand) to virtual play (screen based, device based) in the preoperational age group (2-7yr), where children explore creativity by playing, expressing, learning, exploring, sharing using virtual media.

The project PUBBY (puppet buddy) is a social media platform for children to connect with peer and mentor’s through role playing physical and virtual puppets.

Child Behavior

Going beyond this traditional teach-learn pattern and developing digital technologies that enable children to express themselves together through storytelling, communicating and inventing new ways in effect to find their own voice. The project initial study started with understanding pattern of children learning at home, with friends, by themselves and through technological aid.

We know that digital technologies can enable children to become more active & independent learners, taking charge of their own learning through direct exploration, expression, and experience. The focus shifts from “being taught” to “learning.” For letting children collaborate globaly, we require new multicultural, multilingual, and multi-modal approaches to learning. In order to understand child’s environment I created mind maps

Findings from User Studies

Shadowing - At Day schools

Collected insights for children (2-8 year old) of behavior, grasping, pattern, dislikes, peer interactions and developing soft skills etc. Interaction with teachers leads me to understand, vitally important aspects of traditional ways of engaging into play and get them to learn.

Focus interview - Interaction with parent

Opened up concern and insecurity of parents while they are away. I understood parents consciously want take control of choice making. I found that with working parents spend extra time and effort with their children playing indoor games regularly to bond.

Field study and intercepts - From Outdoor play

In outdoor games children have biased play according to gender and age. In times of confusion they invent their own game play and rules with group approvals. They devise fair play amongst themselves.

Participatory study - group play date

I organized a group play date which was framed to involve there cognitive, physical and emotional senses through story telling. This was conducted to understand mental model to verify elements which would engage a child in the play.

Insights & Observations

Children at the pre-operational age enjoy playing with physical toys. They role-play, building imaginary environment, building scenarios etc is the way to bring their toys to life. This lead us to understand that tangibility tickles children to express and explore. My focus was on supporting children exploration of emotional expression through their narrative and physical play.

When children are involved through tangible toy, it makes the process of imagination easier and believable for them. This physical interactivity engages and involves the child physically and emotionally.

Puppetry is a medium, which is already proven to evoke children to engage, story tell and role-play. Using this rhetorical knowledge of puppets, we used them as input devices to trigger storytelling platform using social media to story tell.

Concept & UX

Building case for concept

Why Pretend play

The solution should enhance pretend play skills in different aspects

Children see the world from their own egocentric point of view, but through maturation and cooperative play, the child begins to understand the feelings of others.

Pretend play helps child understand the power of language. In addition, by pretend playing with others, he learns to reenact a story or organize play. This process helps in building connection between spoken and written language — a skill that will later help him learn to read.

Pretend play provides child with a variety of problems to solve which helps in building cognitive play.

Why virtual platform

It is an intuitive platform, to understand “life skills” in a remote situation using storytelling and role playing as content and puppet as a medium to interact. As the attention span of a child in preoperational child is very less. I tried to strengthen it by involving child’s hand (physical - puppets) and heart (emotional - stories) both. These stories allow the child to take decisions and face consequence in the play.

Understanding and analyzing child’s environment and willingness to utilize the new media, which gave us opportunity to emotionally, cognitively and physically engage the child to play and learn.

Why Puppets

The real hand puppets have a virtual representation in the virtual environment as a character. The actions the child makes the hand puppet does, the same is imitated by virtual character on the screen. This gives a sense of achievement to the child, as he is able to see the result of his role playing activity. This extends to audio feedback; character on screen behaves like a puppet of voice input, if the child gives a voice command the character will follow that as well.

Concept

A storytelling platform where children & peer can use puppets to enter a virtual, play & learning environment. The glove puppet control a virtual character on screen which can be shared remotely with peer through PUBBY app. For children in separate or remote locations because of after-school situations, security, issues, vacations, both the parents working etc.

Content

When a child listens to a story they place them in the story and get fully immersed into it. Every feeling and emotion influences child’s mind to react or feel. I designed the stories have the framework of problem solving situations in imaginary world. To explain them consequences, situations stories have been used. If a child is put in a situation he or she will find out solution to where there problem.

The narratives are designed in such a way that it questions the child’s intention or action instead of giving or teaching them the right answers. The story is a modular tree format where there is no right or wrong direction; instead there is always a choice to be made, every action resulting in a react ion. The child understands and grasps the concept easily as it has a direct implementation in the real world. The child has to face certain situation and certain questions, which challenges cognitively to react.

Interface

• Usability (easy to use and adapt), interaction (simple actions, everyday play like)
• Adaptation (can be used in any spatial reconfiguration to support knowledge transfer and adaptable thinking)
• Reflection (every action has a reaction)
• Personal (sense of achievement)

Design Research & Development

According to the Tech study none of the existing products in market identified puppet detection.

• Designing of Puppet gestures.
• Designing of variation of puppets and exploration of material.
• Designing animated character.
• Deploying gestures and animation together on an engine.
• Further working on the prototype.

Building Working prototype & Testing

I used Arduino, processing, and sensors to make this prototype.I started with making a basic blob detection to conduct a quick evaluation with children if they enjoy puppet with a virtual character. Blob Detection is image processing where it detects two or more than two color blobs on screen from the camera environment; I later stitched the color blobs to the mouth of the character on processing with basic mouth opening code. Resulting in that the mouse character could open close the mouth.

I later used a sensor for better detection as the blob attempt was picking a lot of background color. So I made a real hand puppet with LDR sensor for making the virtual character move. This resulted in much more accurate and responsive result. I took help to try out image processing from IR LED’s which was not a success either. We also made the Please MIT media lab glove which gave accurate serial on depth(accelerometer), gyro sensor and a flex sensor. This gave us extremely accurate results. Later we decided that we do not want such an accurate hardware and want to make it Frugal so that anyone can use it.

Learnings

As an experience designer, I learned a lot in the process of developing PUBBY. Apart from usual design process of solving problem, I had to hit corners where I was not skilled. Which made me learn extensively. There were many aspects through which I could help complete the eco-cycle of PUBBY.

I realized that one person could do limited (even if there is capability) therefore my mentor appointed very talented creative coder (Kalindu Priyadarshana, NTU Singapore) and animator (Pranisha Shrestha IITB) on the team, with whom I worked for a year as a researcher after finishing my Master’s.

We were a team based in Singapore and India. There were many smaller projects which sprouted out while developing Puppet Buddy. The project is still in progress and will soon be launched.

Here is a trailer for it: