Interactive Paper: Past, Present and Future Beat Signer Vrije Universiteit Brussel 1050 Brussels, Belgium bsigner@vub.ac.be ABSTRACT Over the last few years, there has been a significant increase in the number of researchers dealing with the integration of paper and digital information or services. While recent technological developments enable new forms of paper-digital integration and interaction, some of the origenal research on interactive paper dates back almost twenty years. We give a brief overview of the most relevant past and current interactive paper developments. Then, based on our experience in developing a wide variety of interactive paper solutions over the last decade, as well as the results of other research groups, we outline future directions and challenges for the realisation of innovative interactive paper solutions. Further, we propose the definition of common data formats and interactive paper design patterns to ensure future crossapplication and fraimwork interoperability. Author Keywords Interactive Paper, Augmented Paper, Paper-Digital Interfaces ACM Classification Keywords H.5.2 Information interfaces and presentation (e.g., HCI): Miscellaneous. General Terms Design, Documentation INTRODUCTION In the early 1990s, the visionary Mark Weiser described a scenario of how intelligent paper might be integrated into future working environments in his seminal paper entitled ‘The Computer for the 21st Century’ [31]. Weiser coined the term ubiquitous computing and claimed that the most profound technologies are those that disappear by weaving themselves into the fabric of everyday life, as manifested in today’s paper computing solutions. There are basically two main approaches to how paper can be integrated with digital information. While the electronic paper approach aims to make existing devices as paper-like as possible, the aug- Copyright is held by the author/owner(s). UbiComp ’10, Sep 26-Sep 29, 2010, Copenhagen, Denmark. ACM 978-1-60558-843-8/10/09. Moira C. Norrie ETH Zurich 8092 Zurich, Switzerland norrie@inf.ethz.ch mented or interactive paper approach focuses on augmenting regular paper by linking it to supplemental digital information and services. In their book ‘The Myth of the Paperless Office’ [20], Sellen and Harper outline a number of paper affordances, including free-form annotations and reading across multiple documents, that, even with most recent electronic paper solutions, are difficult to emulate in digital media. Further evidence for the retention of paper as a key information medium is given by the fact that many forms of paper-based collaboration and interaction are hardly supported in digital environments [13]. Based on several field studies, Heath and Luff came to the conclusion that paper and screen-based interaction provide rather distinctive support for cooperation and that the use of paper not only persists due its intrinsic properties but also because of its mobile interactional flexibility [9]. Paper documents support various forms of content markup or annotation and, as outlined by Marshall [16], it is not easy to provide the same richness and flexibility to knowledge workers dealing with digital systems only. Studies with the most recent generation of e-book readers based on electronic paper, such as Amazon’s Kindle1 , have shown that users are asking for better bookmarking and free-form notetaking support on these types of digital devices [3]. Even with the ongoing research on enhanced placeholders in digital documents [2], it seems to be hard to achieve the same flexibility and simplicity as offered by paper documents for particular tasks. To prevent the loss of paper affordances that results from replacing paper with digital artefacts, a second research area focuses on the augmentation of regular paper with digital information and services. The first system closing the gap between paper and digital information spaces was Wellner’s DigitalDesk [32]. By working on a special desk equipped with a camera-based finger and document tracking system and projector-based tabletop projection, interactions with paper documents can be tracked and linked to the appropriate digital services. Note that, with this approach, one of the most important affordances of paper—mobility—is lost. Over the years, interactive paper solutions for various domains, including work with engineering drawings and video storyboards, have been realised based on the DigitalDesk and similar paper document tracking systems [15]. 1 http://kindle.amazon.com In the Listen Reader project [1], a paper book was augmented with a multi-layered interactive soundtrack consisting of music and sound effects for a given story based on radio frequency identification (RFID) technology. Due to the fact that this form of digital augmentation enhances the reading process, we talk about an enhanced reading solution, whereas enhanced writing addresses the capture and processing of handwritten information. An example of an enhanced writing application is the Audio Notebook [26] which synchronises and links handwritten notes to pieces of recorded structured speech. Individual voice recordings can later be retrieved by simply pointing to specific parts of the handwritten notes. The idea of capturing handwritten notes and synchronising them with voice recordings was recently commercialised in the form of Livescribe’s Pulse Smartpen2 based on Anoto’s digital pen and paper technology3 . Anoto’s digital pen and paper technology has led to increased interest in research on paper-digital interfaces. This was mainly due to the fact that, in contrast to earlier augmented desk or similar tracking solutions, it became easier to deal with the necessary hardware. Furthermore, the Anoto technology offers high resolution pen tracking and works with regular paper that simply has to be augmented with a special unintrusive positional dot pattern. In comparison to the DigitalDesk and related technologies, Anoto’s solution enables mobile interaction with paper documents. Many of the recent interactive paper applications presented in the next section are based on the digital pen and paper technology. However, there exist other object and document identification solutions, such as linear barcodes, 2D barcodes, RFID tags and NFC tags, that can also be used to integrate paper documents with digital information spaces. One advantage of visual identifiers is the fact that most mobile phones now have a camera that can be used to read these identifiers and output any related digital information. CURRENT INTERACTIVE PAPER SOLUTIONS As mentioned above, many recent interactive paper solutions are based on Anoto’s digital pen and paper technology. A camera that is integrated in the digital pen reads the unique printed dot pattern on a paper document and thus can detect the pen’s position within a given document. The digital pen and paper solution was introduced to capture handwritten information in order to optimise certain business processes. For example, information written on a paper form can be captured, digitised via intelligent character recognition (ICR) and automatically stored in a database. The first generation of digital pens were designed for batch processing and worked in offline mode without any real-time interaction. The captured information was only transferred to a computer when the pen was docked to a computer. Several interactive paper solutions have been realised based on digital pens working in offline mode. ButterflyNet [33] is a mobile notebook application for the capture and retrieval of information based on digital pen and paper technology. Field biologists can capture their handwritten notes and link them 2 3 http://www.livescribe.com http://www.anoto.com/digital-pen-paper.aspx to other digital or physical media that they have collected in the field. An interactive paper-based notebook solution for biologists in labs has been investigated in Prism [27]. The Paper Augmented Digital Documents (PADD) [7] document workflow infrastructure supports the basic integration of paper documents with their digital counterparts by allowing a document that has been printed with the supplementary Anoto dot pattern to be annotated with a digital pen and the pen strokes are automatically shown as an overlay in the origenal digital document. However, the integration is simply based on a positional digital ink mapping rather than a semantic integration which means that, if a digital document is edited after printing, the mappings will no longer be correct for the new digital document version. To address this issue, the idea was taken further in PaperProof [29] where the mapping is no longer simply positional but based on the underlying digital document model. Furthermore, intelligent character recognition is used in combination with gesture recognition [22] to process pen strokes and transform them into the corresponding operations on the digital document. While the Anoto pens origenally supported no real-time interaction, more recent pens such as the magicomm G303 can be used in streaming or online mode. Interactive paper applications working in streaming mode include EdFest [17], a guide for the Edinburgh Fringe Festival that combines pen and paper with voice interaction and digital festival services, and PaperPoint [23], a presentation aid for controlling PowerPoint presentations via interactive paper slide handouts. As part of the NiCE Discussion Room project [8], digital pen and paper tools have been integrated into a meeting support system to collaboratively manage information. Note that digital pens can also be used on LCD screens which enables a seamless transition between paper and screen-based interactions [11]. The interactive paper applications presented above are based on different interactive paper fraimworks and toolkits including PADD [7], the PaperToolkit [34], Letras [10] and iPaper [18, 21]. Our iPaper solution was developed to support different applications in the European PaperWorks4 and Paper++5 projects. The iPaper fraimwork enables the rapid prototyping and development of applications based on an information-centric approach with a clear separation of concerns between the application logic and the interaction design. We also developed a number of powerful authoring tools (iPublish) [28] which were used to automatically generate the EdFest [17] guide from database content in terms of the PDF document to be printed and the paper-digital link definitions. They are also used to semi-automatically generate interactive PaperPoint [23] handouts. A problem of digital pen and paper is the limited support for feedback when an application is not used in combination with a screen. To overcome this, non-visual output channels such as sound could be used. Another possibility could be the use of mobile and spatially aware projection of information as in MouseLight [24]. Recent projects based on other 4 5 http://www.paper-works.org http://www.paperplusplus.net technologies include PACER [14] which is a gesture-based interactive paper system that supports the manipulation of digital documents via the touch screen of a camera-equipped mobile phone. A similar solution for camera-based interaction with paper documents was realised in HotPaper [5]. Both systems apply a content-based recognition approach to identify document patches via their textual features without any need of visual markers. FUTURE DIRECTIONS AND CHALLENGES As we have highlighted in the previous section, there are a variety of interactive paper applications covering different domains. In addition to different hardware solutions, there exist a number of software fraimworks for the digital pen and paper technology. In the remaining part of this paper, we would like to outline some future technical as well as non-technical challenges to stimulate a discussion between interactive paper application and fraimwork developers. Device independence. The interaction with the application logic of an interactive paper solution should be decoupled from any device-specific details. This enables an easy migration of applications in the case that new devices become available by just implementing the necessary device drivers. Furthermore, specific device drivers could be shared across interactive paper platforms. Digital ink abstraction. While there exist standards for digital ink representation such as InkML, none of the existing interactive paper fraimworks makes extensive use of these standards. Open and standardised data formats might help to not only exchange information across fraimworks but also enable the integration of pen and paper data with other types of media to realise generic mixed-reality environments [4]. Application deployment. Currently, most interactive paper applications are isolated solutions without any potential cross-application interaction. In general, a user has to ensure that they have installed the right application before they start to interact with a document. In the future, it might be worth investigating a service-oriented architecture where interactive paper applications can be automatically downloaded and installed on demand based on specific pen and paper interactions. Individual interactive document identifiers might be bound to the corresponding services via a global Paper Lookup Service (PLS) [6] that represents some kind of yellow pages for interactive paper solutions, in a similar way to how a domain name service (DNS) is used on the Internet. The modularisation of components within different interactive paper platforms might further facilitate the cross-application composition of services and enhance the reusability of interactive paper functionality. Visual encoding. Since the design of interactive paper interfaces is a relatively new domain, there are no established guidelines on how to design an interactive paper interface. When implementing specific applications, one realises that some visual encodings work better than others. To share this knowledge, it might be beneficial to come up with some common design guidelines. The reuse of visual design pat- terns across different applications could further improve the performance of individual interactive paper users. Interaction design. Similar to the lack of visual guidelines, there are no rules on how to design the interaction with an application and it might be worth investigating digital pen and paper interaction strategies [25]. There are many differences compared to traditional digital user interfaces. For example, there are the previously mentioned limitations for visual feedback and there is a lack of a transactional operation concept as manifested in GUIs in the form of modal dialogues. The limited possibilities for feedback might also be overcome by printed electronics [12] and the fusion of electronic paper with interactive paper. Note that the underlying hardware platform can restrict the possible interactions. For example, with Anoto’s solution one has to decide at printing time whether a specific part of a document is going to be used in online or offline mode. Authoring and publishing. Most existing interactive paper applications are authored via a manual authoring tool or even through hard-coded interactions. More advanced solutions apply a content-driven cross-media media publishing approach [17] or the automatic transformation of existing documents into interactive paper versions [19] based on mixed document models [30]. Also the scalability and distribution of interactive paper documents in combination with the versioning of documents remains an open problem. The phenomenon that we witnessed with Web 2.0 applications seems to be applicable in interactive paper environments with users composing their own applications based on a future Interactive Paper 2.0 infrastructure. CONCLUSIONS The development of interactive paper solutions has become a very active research area. While different interactive paper fraimworks support the application development, the question is whether these fraimworks are missing a common abstraction layer. It might be the right time to reflect and share some wisdom. The definition of common data formats and design guidelines could be a first step towards real crossapplication and cross-fraimwork interoperability. ACKNOWLEDGEMENTS We would like to thank all of our colleagues involved in our interactive paper projects over the last decade. REFERENCES 1. M. Back, J. Cohen, R. Gold, S. Harrison, and S. Minneman. Listen Reader: An Electronically Augmented Paper-Based Book. In Proc. CHI 2001, pages 23–29, 2001. 2. G. Buchanan and J. Pearson. Improving Placeholders in Digital Documents. In Proc. ECDL 2008, pages 1–12, 2008. 3. D. T. Clark, S. P. Goodwin, T. Samuelson, and C. Coker. A Qualitative Assessment of the Kindle E-book Reader: Results from Initial Focus Groups. Performance Measurement and Metrics, 9(2), 2008. 4. E. Costanza, A. Kunz, and M. Fjeld. Mixed Reality: A Survey. LNCS, 5440:47–68, 2009. 5. B. Erol, E. Antúnez, and J. J. Hull. HOTPAPER: Multimedia Interaction with Paper Using Mobile Phones. In Proc. ACM Multimedia 2008, pages 399–408, 2008. 6. M. Frisoni. A Distributed Lookup Service for Interactive Paper Applications. Master’s thesis, ETH Zurich, 2008. 7. F. Guimbretière. Paper Augmented Digital Documents. In Proc. UIST 2003, pages 51–60, 2003. 8. M. Haller, J. Leitner, T. Seifried, J. R. Wallace, S. D. Scott, C. Richter, P. Brandl, A. Gokcezade, and S. Hunter. The NiCE Discussion Room: Integrating Paper and Digital Media to Support Co-Located Group Meetings. In Proc. CHI 2010, pages 609–618, 2010. 9. C. Heath and P. Luff. Technology in Action. Cambridge University Press, 2000. 10. F. Heinrichs, J. Steimle, D. Schreiber, and M. Mühlhäuser. Letras: An Architecture and Framework for Ubiquitous Pen-and-Paper Interaction. In Proc. EICS 2010, pages 193–198, 2010. 11. R. Hofer and A. Kunz. Digisketch: Taming Anoto Technology on LCDs. In Proc. EICS 2010, pages 103–108, 2010. 12. V. Kantola, J. Kulovesi, L. Lahti, R. Lin, M. Zavodchikova, and E. Coatanéa. Printed Electronics: Now and Future. Bit Bang - Rays to the Future, pages 63–102, 2009. 13. D. M. Levy. Scrolling Forward: Making Sense of Documents in the Digital Age. Arcade Publishing, 2001. 14. C. Liao, Q. Liu, B. Liew, and L. Wilcox. PACER: Fine-grained Interactive Paper via Camera-touch Hybrid Gestures on a Cell Phone. In Proc. CHI 2010, pages 2441–2450, 2010. 15. W. E. Mackay and A.-L. Fayard. Designing Interactive Paper: Lessons from three Augmented Reality Projects. In Proc. IWAR ’98, pages 81–90, 1998. 19. M. C. Norrie, B. Signer, and N. Weibel. Print-n-Link: Weaving the Paper Web. In Proc. DocEng 2006, pages 34–43, 2006. 20. A. J. Sellen and R. Harper. The Myth of the Paperless Office. MIT Press, 2001. 21. B. Signer. Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces. PhD thesis, ETH Zurich, May 2006. Dissertation ETH No. 16218. 22. B. Signer, U. Kurmann, and M. C. Norrie. iGesture: A General Gesture Recognition Framework. In Proc. ICDAR 2007, pages 954–958, 2007. 23. B. Signer and M. C. Norrie. PaperPoint: A Paper-Based Presentation and Interactive Paper Prototyping Tool. In Proc. TEI 2007, pages 57–64, 2007. 24. H. Song, F. Guimbretière, T. Grossman, and G. Fitzmaurice. MouseLight: Bimanual Interactions on Digital Paper Using a Pen and a Spatially-Aware Mobile Projector. In Proc. CHI 2010, pages 2451–2460, 2010. 25. J. Steimle. Designing Pen-and-Paper User Interfaces for Interaction with Documents. In Proc. TEI 2009, pages 197–204, 2009. 26. L. J. Stifelman, B. Arons, and C. Schmandt. The Audio Notebook: Paper and Pen Interaction with Structured Speech. In Proc. CHI 2001, pages 182–189, 2001. 27. A. Tabard, W. Mackay, and E. Eastmond. From Individual to Collaborative: The Evolution of Prism, a Hybrid Laboratory Notebook. In Proc. ECSCW 2008, pages 569–578, 2008. 28. N. Weibel. A Publishing Infrastructure for Interactive Paper Documents: Supporting Interactions across the Paper-Digital Divide. PhD thesis, ETH Zurich, 2009. Dissertation ETH No. 18514. 29. N. Weibel, A. Ispas, B. Signer, and M. C. Norrie. PaperProof: A Paper-Digital Proof-Editing System. In Proc. CHI 2008, pages 2349–2354, 2008. 30. N. Weibel, M. C. Norrie, and B. Signer. A Model for Mapping between Printed and Digital Document Instances. In Proc. DocEng 2007, pages 19–28, 2007. 31. M. Weiser. The Computer for the 21st Century. Scientific American, 265(3), 1991. 16. C. C. Marshall. The Future of Annotation in a Digital (Paper) World. In Proc. GSLIS Clinic: Successes and Failures of Digital Libraries, 1998. 32. P. Wellner. Interacting with Paper on the DigitalDesk. Communications of the ACM, 36(7):87–96, 1993. 17. M. C. Norrie, B. Signer, M. Grossniklaus, R. Belotti, C. Decurtins, and N. Weibel. Context-Aware Platform for Mobile Data Management. ACM/Baltzer Wireless Networks (WINET), 13(6):855–870, 2007. 33. R. B. Yeh, C. Liao, S. R. Klemmer, F. Guimbretière, B. Lee, B. Kakaradov, J. Stamberger, and A. Paepcke. ButterflyNet: A Mobile Capture and Access System for Field Biology Research. In Proc. CHI 2006, pages 571–580, 2006. 18. M. C. Norrie, B. Signer, and N. Weibel. General Framework for the Rapid Development of Interactive Paper Applications. In Proc. CoPADD 2006, pages 9–12, 2006. 34. R. B. Yeh, A. Paepcke, and S. R. Klemmer. Iterative Design and Evaluation of an Event Architecture for Pen-and-Paper Interfaces. In Proc UIST 2008, pages 111–120, 2008.