The Multiplayer Storybook

Research

Reading on paper and screens

Digital natives and new media literacy

There’s nothing novel about tablets and smartphones any longer. Children are “likely to be exposed to technology at times even before they are exposed to print or traditional books”1 and “digital natives” has become a household term. Some parents report that sharing a paper book has become a pleasant novelty.2

Educators are starting to worry less about digital devices negatively impacting learning and development, instead widening the young field of “new literacies” (participatory media like sharing photos and videos, social media, online shopping etc.).

[...] we regard literacy learning as social in origin and mediated through action and interaction using cultural artefacts. These artefacts evolve over time as societies develop, and in the current era, we argue that literate activity is characterised by the use of both print and digital media. Particularly when using digital devices, meanings can be expressed through multiple modes of symbolic representation, such as combinations of spoken and written language, images, icons, sounds, layout and animation.

Flewitt et al., 2015

Trans-literacy

Trans-literacy is the idea that skills gained from reading in one medium, translates to other media.

Reading e-books promotes traditional literacy skills and is particularly supportive in the area of vocabulary development, and young children’s interaction with enhanced digital books also advances their facility to communicate and comprehend across modes and platforms, sometimes called trans-literacy development.

Cahill & McGill-Franzen, 2013

This would suggest that if you want children to become book-readers, there are advantages for digital narratives to stay close to a book-like format. Trans-literacy is not only promoted by similarities between printed and digital reading environments, it is also important for children to learn the differences between them.1

Why digital devices are bad

It seems almost intuitive, at least for those of us who were introduced to tablets and smartphones as grown-ups, to view digital devices as potentially harmful. One study found mothers to have a strong “preference for reading on paper, whether this was reading themselves or for their child”, while their children don’t really care that much. It also found that the mothers reading paper books made more story-relevant comments with a higher “interaction warmth” than when reading digital books.2

Several studies point out how interactive features, though engaging, are often distracting. This usually means it affects story recall negatively, i. e. the child is too busy clicking buttons to remember what the story was about later. Interactive features can even cause “cognitive overload” in young children when they have to switch between listening to a story, getting words explained, and playing games – all within a few moments.3 Some educators also worry apps and games lead to addiction and over-stimulation, as well as having “negative consequences for the kinds of patient and persevering learning dispositions needed for the occasionally arduous process of learning to read and write.”4

Children touch and press everything, and features that were designed to be in the background can suddenly become foregrounded.2 Glitches, bugs and interface quirks can become fun toys on their own. For instance, a cool page turn animation can end up being more interesting than what’s on the page itself. Marsh et al. argues that this is a new type of play that only happens on digital devices. They call it transgressive play, “in which children contest, resist and/or transgress expected norms, rules and perceived restrictions in both digital and non-digital contexts”.5

Why digital devices are good

The pairing of text narration and multimedia elements strengthens verbal comprehension and story recall by reducing cognitive load (younger children aside).3, 6 This is true for both single and shared reading, and taken advantage of in pre-schools, schools and at home.1

Teachers report that iPads can help children with motivation, consentration, and classroom communication. It also facilitates “collaborative and independent learning in playful and creative ways.”4

[...] well planned literacy-related iPad activities stimulated children’s motivation and concentration, and offered rich opportunities for communications, collaborative interaction, independent learning and enthusiastic learning dispositions. [...] immediate feedback, along with tangible and satisfying end products, motivated children to engage deeply with iPad-based literacy activities, which as one practitioner commented, attracted their attention like ‘bees to a honeypot’

Flewitt et al., 2015

Animations are better for explaining difficult words and concepts than still illustrations – some words, especially verbs, really benefit from some motion. Music and sound effects can also be important when depicting more abstract expressions or emotions and feelings3, for instance a sad-sounding minor harmony.

Parent-child interaction

Shared reading

As mentioned earlier, shared reading is about the social connection and interaction between parent and child. In broad terms, this interaction consists of parents’ “scaffolding” (guidance) and discussions around story, illustrations and interactive elements.

Several studies7, 8 suggest that illustrations generally make parents and children interact more with each other, both verbally and non-verbally. (In some cases, the lack of illustrations make parents compensate by reading with more emotion.) Illustrations are very effective at capturing children’s attention to the stories being read to them, and help “familiarize children with situations that they might otherwise reject”. When the dyads have illustrations to talk about, it’s easier to keep the children interested and focused, leading to strengthened story recall. (Surprisingly, anthropomorphic illustrations and language are the exception, leading to lower levels of learning.7)

The parent and child’s body language and their positions relative to each other is called “dyadic posture”.

Fig. 3.1 | Shared reading and dyadic postures.

(Fig. 3.1) illustrates some typical dyadic postures during shared reading. The first is the “curled-up” position, where the dyad sit close to each other while looking at the same page. This occurs whether the parent (fig. 3.1, a) or child (fig. 3.1, b) is reading out loud. When the child is reading, the parent may participate by “scaffolding”, i. e. guiding, providing help and information.

The second posture (fig. 3.1, c) occurs when there’s nothing but text on the page (or illustrations are non-essential), and comfort matters more than proximity. Typically when a parent reads a storybook to the child:

[...] when a mother read from paper, she often held the book between herself and the child, with the child very close to her, either tucked under her arm to facilitate visual sharing or in a very relaxed posture with audio sharing, but little sight of the book.

Yuill & Martin, 2016, p. 6

On-screen shared reading on a tablet device (fig. 3.1, d) is known to present some challenges, related to children’s perceived “ownership”9 of the tablet as a medium, and their closeness to the screen obstructing their parents’ view of what’s happening on the display.

When children read from a screen, they tended to hold the tablet in a ‘head-down’ posture typical of solo uses [...] leading [the mothers] to curl round behind the child in order to ‘shoulder-surf’ the screen, rather than adopting the ‘curled-up’ position common when reading the paper book.

Yuill & Martin, 2016, p. 6

Games and play

There have been several broad classifications of the ways children play. A couple of famous ones, Roger Caillois’ definitions from 1961 and Corinne Hutt’s categories from 1979, more or less agree5 that the main groups are:

  • Games for learning, or epistemic play
  • A variety of role play, or ludic play, where children draw from their experiences – from their own life, fantasy and drama (TV, movies etc.)
  • Games with rules, play based on skills or chance, often competitive

Hughes’ taxonomy of play from 2002 expand these into to 16 play types that can be used to describe most activities children are engaged in.5 The study “Digital Play: a new classification” (Marsh et al., 2016) updates Hughes’ taxonomy for “on-screen” play, and adds a 17th play type that “occurred when children used features of the apps that were not part of the design, thus transgressing the app producers’ intentions”. (Something to look out for!)

Fig. 3.2 | Some of the play types from Hughes’ taxonomy

The authors argue that there is no play type that cannot be replicated digitally, with detailed examples of iPhone-games turning into life-and-death situations. Comparing it to traditional play:

[...] contemporary digital cultures provide rich opportunities for the promotion of play that is rooted in children’s everyday experiences. This is not [...] an inferior form of play; rather, it sits alongside more traditional play activities and is important for creative development

Marsh et al., 2016

The list goes a long way to explain the popularity of games like Minecraft, in which most of the 16 types are catered to, from mastery of environments to various kinds of role play. I found Hughes’ taxonomy and Marsh’ new classification to be very handy both for reference and inspiration, as I could generate ideas by going through the play types and think of ways to apply each one to my own storybook.

State of the Art

Looking at interactive children’s media, the products range from books to games – from game-like paper books to digital games inspired by children’s picture books – with a whole spectrum of categories in between.

Printed books are alive and well, with no decline in sales over the last 20 years.10 According to one author I spoke to11, a normal first edition is printed in 2500 copies, with expected sales around 200 books a year.

It’s hard to gauge how well children’s apps sell based on the app store charts, but throughout this project, they have taken up 80–90% of the “paid app” bestseller lists. (Poio, a constant chart-topper, has reportedly “helped 30.000 kids learn how to read”12, but since their software is used in schools, that doesn’t translate to sales figures.) Google’s Play Store distinguishes between sales numbers and highest grossing apps, but Apple’s algorithms are unknown.

The charts are localized, so we can make one of two assumptions about the Norwegian market:

  1. We spend a lot of money on children’s apps.
  2. We buy very few apps, but when we do, they are for our children.
    I suspect the latter is more plausible. (As a sidenote, I saw an obscure and quite expensive iPad app shoot right up into the top 25 “best-selling” paid apps only hours after I had purchased it.)

Paper books

Though their themes and topics are as diverse as adult books, traditional children’s books are normally categorized by age, ranging from picture books to “middle-grade” books. (I do not include “young adult” (ages 12–18) literature in this study.)

3–5 years, Picture books

Books teaching fundamental language and concepts (for instance: vocabulary, numbers, emotions, civic services) through big letters and colorful pictures. Often “interactive” features using paper in novel ways, like the holes eaten by “The Very Hungy Caterpillar” (Eric Carle, 1969) or the paper flaps in Eric Hill’s “Spot” series.

6–8 years, Storybooks

More focus on text than on pictures, though most often illustrated. Classic books by Roald Dahl & Quentin Blake (“The BFG”, “Matilda”), Astrid Lindgren ("Pippi Longstocking") and Ole Lund Kirkegaard ("Gummi-Tarzan").

9–12, “Middle grade” books

Novels, pretty much, where pictures are mostly reserved for their covers. Lots of fantasy and comedy, often serialized. The fact books for this age group are more advanced and diverse, both related to school and children’s interests. Current best-selling series about “Captain Underpants” (Dav Pilkey),  “Diary of a Wimpy Kid” (Jeff Kinney), “Harry Potter” (J. K. Rowling).

E-Books and “enhanced“ digital storybooks

In its simplest form, an e-book is the screen version of its print sibling, with static text and illustrations. Cahill & McGill-Franzen (2013) mention four minimum criteria for classifying an e-book:

  1. Text presented visually on a screen
  2. Book-like configuration (table of contents, pages, etc.)
  3. An organizing subject matter or topic
  4. Multimedia enhancements

Several Norwegian publishers have their own “bookshelf” apps for purchasing and viewing their children’s books. (You’d think the publishers could work together on this one, but I’ve yet to see a bookshelf-app that covers several publishers’ children’s books –  it seems troublesome to have a large library of digital children’s books when you have to remember on which publisher’s bookshelf to find a given book.)

“Enhanced” storybooks make more use of their interactive platform, and usually contain animated elements, interactive “hotspots”, music and sound effects, and voice-acted narration. These seem primarily made for solo use, where the children can explore the books on their own.2

Audiovisual and interactive features can even be effective stand-ins for parental “scaffolding”:

[...] contemporary digital cultures provide rich opportunities for the promotion of play that is rooted in children’s everyday experiences. This is not [...] an inferior form of play; rather, it sits alongside more traditional play activities and is important for creative development

Takacs et al., 2014

Adaptations of the classics (Dr. Seuss’ books, for instance) are well represented on the iPad, but stories specifically written for the format still seem to be rare.

Educational apps and games

Apps with a focus on teaching: often ABCs and math, but also about topics like space and dinosaurs. The iPad has become a common teaching tool in Norwegian primary education, and there’s a whole ecosystem of software designed for classroom use and as support for school textbooks, not always found on the app stores. Some of these apps stay close to a book-like presentation, while others are closer to games.

Poio (fig. 3.3, top) is part game and part storybook with a unique approach to teaching 4–9-year-olds how to read. The children control a character that collects alphabet letters, and can then spell words with these letters and use those words in a story. Getting rave reviews, Poio has topped the paid app charts13 for the entirety of this project, even at a quite high price point (kr 199,-). It’s compatible with the curriculum for Norwegian first graders and used as a teaching tool in schools.12

[Update, 2019: Poio is now on both iOS and Android and has versions for Swedish and English readers.]

Dragonbox (fig. 3.3, bottom) is a series of mathematics games for children of all ages that teach basic and advanced maths in fun ways. Developed by WeWantToKnow AS in collaboration with the publisher Aschehoug, it comes in a school edition where the app is bundled with textbooks and manipulatives (i.e. building blocks).14

Fig. 3.3 | Top: Poio (Source: Poio press kit) | Bottom: DragonBox BIG Numbers (Source: DragonBox press kit)

Pretend play

Games that have more in common with toys like LEGO or Barbie than books. These “sandbox” games have no set goals and leave the storytelling up to the player via interactive play sets. These are entertainment products, but valuable for social interaction and fun to play with others. (Not “multiplayer”, but playing together with/on the same device.) Some apps have features that let the players record and share their stories.

Fig. 3.4 | Pretend play in Toca City

This is the best-selling category on the paid app charts, with series like “Toca Life(fig. 3.4), “Sago Mini” and “Dr. Panda” filling several of the top spots.13

Computer/video games

I’ve chosen not to examine the whole field of video games in detail, as it’s huge on its own – broadly speaking, every game with a story can be said to be an interactive narrative. Still, there are some genres to take note of (and inspiration from), mainly stemming from the adventure games of the 90s.

Some adventure games back then tried to play with live-action video (“Under a Killing Moon”, “Voyeur”, “Phantasmagoria”), earning the moniker “interactive movies”. With adventure games enjoying somewhat of a renaissance lately, with games exploring novel takes on interactive storytelling, we have come full circle back to movie-like games like the “Uncharted” series and this year’s “A Way Out”. They exist alongside games that are visually removed from Hollywood blockbusters, but first and foremost tell a story.

Fig. 3.5 | 1: Grim Fandango Remastered | 2: Kentucky Route Zero | 3: Puzzle Agent | 4: Burly Men at Sea

Adventure games

Adventure games (Fig. 3.5, 1) had their heyday in the 90s and early 2000s, with titles by Lucasfilm/Lucasarts (“Monkey Island” series, “Grim Fandango”) and Sierra On-Line (“King’s Quest” and “Leisure Suit Larry” series) now regarded as classics. The genre has had somewhat of a revival in later years, with a lot of new games and “remasters” of old ones.

These “point-and-click” adventure games tell semi-linear stories and requires the player to solve puzzles and problems. Progression is typically made through exploring environments, talking to NPCs and collecting items in an inventory. Inventory items can then be used with environments and NPCs to solve puzzles.

I enjoy games in which the pace is slow and the reward is for thinking and figuring, rather than quick reflexes. The element that brings adventure games to life for me is the stories around which they are woven. When done right, it is a form of storytelling that can be engrossing in a way that only interaction can bring. [...] If any type of game is going to bridge the gap between games and storytelling, it is most likely going to be adventure games. They will become less puzzle solving and more story telling, it is the blueprint the future will be made from.

“Stories you play”

A collective term for games that focus solely on telling a story, with various degrees of interactivity. Usually quite linear stories with little problem solving, the gameplay is closer to reading a book or watching a movie.

I have nothing against stories you play, though they’re currently shelved with video games only because we privilege unreliable terms like “interactivity.” [...] in all cases they involved me watching some things happen, clicking a button, then watching some more things happen. [...] That’s put the burden of invention on audiovisual novelty.

These are nothing new – laserdisc games like “Dragon’s Lair” were popular in the 80s, and CD-ROM brought on a wave of “Full Motion Video”-games (“Voyeur”, “Phantasmagoria”) – but there’s been a lot of interesting releases over the last few years. Advances in graphics might have made “interactive movies” viable in new ways (see: “Uncharted” series, “A Way Out”), but it’s could also be seen as a sign of video games maturing as a medium. Not everything is about action, horror and adventures anymore: “Kentucky Route Zero(Fig. 3.5, 2) is a surreal and poetic story about loss and death, “Firewatch” is a romantic thriller and “Night in the Woods” is a drama comedy about returning to a the dying small town you grew up in.

More to the point: just like “Robinson Crusoe”-like adventure novels a became genre popular with children, these story-focused games are starting to become more children and family friendly. Brain&Brain’s “Burly Men at Sea” (Fig. 3.5, 4) from 2016  is a “story you play” presented like a children’s book. It was very well received, with reviewers bringing up shared reading:

It’s a gentle seafaring tale I’m looking forward to playing through with a child when I next see my smaller family members but which I’m more than happy to play for my own enjoyment as well. I think I’m on my sixth distinct playthrough at the moment and still discovering new things.

It’s the dreaded “product you discover during the diploma that you wish you had thought of first”, but it’s at least comforting to see how far you can take interactive children’s books and that there seems to be a big interest in them.

Case studies:

I took a closer look at a few enhanced e-books, thinking they were the closest to my own project.

I was surprised (and nostalgic) to find “Arthur’s Teacher Trouble” (Marc Brown, 1986) on the App Store. This is part of Brøderbund’s catalogue of “Living Books”, a popular CD-ROM series from the mid 90s, now ported to the iPad as “Wanderful Interactive Storybooks”. It is literally, pixel by pixel, the exact same digital book from my childhood, with the same voiceover. I had a lot of fun with this book when I was a child, and it was instrumental in teaching me English.

There are two modes, with and without voiceover. Every word on the page can be clicked to hear it out loud, and almost every little element in the background illustrations can be interacted with, triggering sounds and animations. The pictures are pixellated and the voiceover is compressed into a crunch, but the title is more than 20 years old. (I’d say it’s still a charming book, but then again, I’m wearing rose-colored glasses for this one.)

Jakten på Teddy is the only storybook-app I found that I know was specifically written for iPad. It was “Jakten på Teddy” tells the story of Theo, whose teddy bear gets stolen by the evil witch Ipswitch. Theo gets transported into a magical realm where all his toys have come to life, and has to search for Ipswitch to rescue Teddy.

Theo travels through three locations before confronting the Ipswitch in her castle. In each location, he meets a living version of one of his toys, all broken in different ways by the witch. The locations all have interactive hotspots that reveal short animations. Theo and the toys have a short talk, the reader plays a little minigame to mend the broken toy, and it’s off to the next location. When confronting the witch, the toys you have mended come to your help in defeating her.

“Kubbe lager skyggeteater” is the least “enhanced” of the case study e-books. It tells the story Kubbe, a piece of wood, who is spending a day with his grandmother in the forest. Grandma tells Kubbe about the art of shadow theatre, inspiring him to make his own play.

“Kubbe...” has beautiful illustrations and excellent voice acting, even a long animation about shadow theatre, but interactive hotspots are few and far between. There’s very little to do but listen to the story and turn the pages.

I’ve included Telltale Games’ “Puzzle Agent 1 & 2” (Fig. 3.5, 3) because it’s presented like a children’s book and has a lot of minigames. It tells linear, humorous mystery stories interspersed with brain-teaser puzzles, with full voice-acting and illustrations by Graham "Grickle" Annable. The player can travel between locales and talk to NPCs, but progression is made by solving puzzles with increasing difficulty. The minigames are well suited to touch controls, and both Puzzle Agent games have been ported to the iPad.

Findings

These enhanced e-books are well executed products, but not taking full advantage of the all the opportunities of interactive storytelling.

Dynamic stories

In the case of “Jakten på Teddy” the story branches, but only slightly. The app is inspired by “Choose Your Own Adventure”-books and its marketing really plays up how the story is different each time you play, but the truth is that there is very little variation between each playthrough. There are six different locations to visit, and four different orders in which to complete the story, but the climax is always the same. Player choices have very little impact, it doesn’t matter what choices you’ve made or which toys come to your help in the end, the ending plays out exactly the same.

“Arthur...” has a ton of interactive hotspots, but the story is the same every time. “Kubbe...”, though a charming e-book, might as well have been a movie. “Puzzle Agent” has a linear story, but progresses like an adventure game, making it feel like it’s based on player actions.

If the story is completely linear, it’s a good idea to break it up with events that require player input (beyond turning a page) to progress. Branching stories must be linear enough to write, but to offer any real replay value to anyone but the youngest children, they need to be dynamic enough to offer up some real variations based on player input.

Interactive elements

In “Jakten...”, the interactive hotspots are telegraphed with a distracting sparkle animation, yet offer little in return for pressing them. Neither “Kubbe...” nor “Arthur...” gives any special focus to hotspots, but where almost every element on the screen (including words) offer a reward when clicked, almost none of Kubbe’s do. “Kubbe...” makes it worse by having very detailed illustrations, at times with tons of elements that you want to interact with, but very few that do anything.

It might be hard to predict what the user thinks, but everything that looks like an interactive element, should be an interactive element. At the same time, a hotspot should manage to look like one without drawing too much attention to itself, for example using odd-looking borders or animations. Both “Arthur...” and “Puzzle Agent” gets this right.

Interactive elements and minigames should not be active or playable while the story is being read, but become available during the break between pages. This is easy to solve when using a pre-recorded voiceover, but is a tricky problem for “live” shared reading.

Conclusions

  • Children aren’t awed by iPads any more.
  • All kinds of play types can be recreated digitally
  • There is nothing indicating that iPads are inherently bad for children, children’s books or shared reading, but one needs to be aware of distracting elements and decide on when to teach and when to entertain. Younger children are especially sensitive to distractions.
  • Interactive media for children has its uses in the classroom.
  • If the app’s interface is too fun to use, it will be used for transgressive play. Keeping close to a paper book format lets the children develop trans-literacy skills.
  • The biggest challenges with shared reading on iPads, are children obstructing the parent’s view and children’s perceived “ownership” of the medium.
  • Digital children’s books are seeing some interesting developments, but the evolution comes from interactive narratives circling back from games, not “growing up” from e-books. The genre of “stories you play” is starting to come into its own, with a lot of novel takes on interactive storytelling.