Documenting and Experiencing Travel in the Era of Augmented Reality.

Documentar y experimentar viajes en la era de la Realidad Aumentada

Recibido:09 de junio 2019 Aprobado: 24 de junio 2019

Dr. Gerda Cammaer22 Dr. Gerda Cammaer is Associate Professor in the School of Image Arts (Ryerson University) where she teaches in the BFA Film Program and in the MFA Program in Documentary Media. She is co-director of the Documentary Media Research Centre (DMRC) and co-editor of Critical Distance in Documentary Media (Palgrave Macmillan, 2018). Cammaer is a filmmaker, curator and scholar. Her films have screened at various international film and video festivals. Her research focuses on the revival of microdocumentaries, notably how current consumer friendly technology and mobile devices challenge the expectations and definitions of documentary film.
School of Image Arts, Ryerson University (Toronto, Canada).

Resumen

Viewfinders (www.viewfinders.gallery) es un proyecto internacional y colaborativo, creado por la Dr. Gerda Cammaer y el Dr. Max Schleser (Swinburne University, AUS) sobre imágenes de viajes usando el plano de seguimiento. El proyecto está estructurado alrededor de tres exploraciones practicas y teóricas: filmación con teléfonos inteligentes, experiencias de visualización interactivas y no lineales visionamiento y experimentar el contenido en la realidad a través de una aplicación móvil de realidad aumentada (RA). El objetivo de este proyecto, es presentar imágenes al mundo como un viaje no-lineal en aparatos móviles a los que se puede acceder en cualquier lugar y momento, expandiendo tanto la experiencia de viajar como la experiencia del cine documental. Viewfinders ofrece la oportunidad a los participantes de ver cómo otros han experimentado con los espacios cuando viajan y descubren una nueva visión que ven formarse ante ellos.

Abstract

Viewfinders (www.viewfinders.gallery) is an international collaborative research project created by Dr. Gerda Cammaer and Dr. Max Schleser (Swinburne University, AUS) about travel images using traveling shots. It is structured around three practical and theoretical explorations: smartphone filmmaking, interactive and non-linear viewing experiences, and experiencing content in reality via the mobile AR app. The aim of the project is to present images of the world as a non-linear journey on mobile devices that can be accessed anywhere and everywhere, expanding both the experience of travel and the experience of documentary cinema. Viewfinders offers participants the opportunity to see how others have experienced the spaces through which they travel, and discover a new view that they see forming before them.

Palabras claves

Documental, Realidad aumentada, teléfonos inteligentes, viajes.

Keywords

Documentary, Augmented Reality, smartphones, travel.

Documenting and Experiencing Travel in the Era of Augmented Reality.

33 This text is based on the first part of a longer and more elaborate essay titled “Viewfinders: Exploring Travel and Landscapes via Augmented Reality”, which was written in collaboration with Dr. Max Schleser as a chapter for Interactive Documentary in Canada, Michael Baker and Jessica Mulvogue (eds.), McGill-Queen’s University Press, forthcoming 2020.

Documentar y experimentar viajes en la era de la Realidad Aumentada

Dr. Gerda Cammaer,
School of Image Arts, Ryerson University (Toronto, Canada).

Introduction

Viewfinders is an international collaborative creative arts research project that connects travel experiences with mobile creativity. The mobility and functionality of devices like smartphones means that audience engagement with content no longer needs to be a passive viewing experience, but can involve interaction, participation and even location aware and/or responsive content. Viewfinders is an online project that incorporates all these elements to create a navigable augmented landscape composed of places visited, generated by mobile users in the form of brief traveling shots. It aims to recreate and explore traveling, contribute to the study of mobile cinema, compare various forms of moving image media (with a focus on interactive and AR experiences), investigate the possibilities of collaborative and practice-based research, and of working with peer-generated content (Cammaer and Schleser in Brasier et al 2019, 70).

The idea for this project was first presented and submitted for peer review at the 2016 MINA Conference (Mobile Innovation Network Australasia) and RMIT’s Docuverse conference, both in Melbourne, Australia. In 2017, the online curation platform was exhibited at the Invisible Geographies exhibition of FLEFF (Finger Lake Environmental Film Festival) and the Mobile Utopia exhibition at Lancaster University, UK (both online 2018). Finally, the Viewfinders AR prototype was launched at the 2018 i-Docs symposium in Bristol (online 2018) and is discussed in detail in the eBook Docuverse: Approaches to Expanding Documentary (Cammaer and Schleser in Brasier, Hansen et al, 2018).

Augmenting Travel (Film) Experiences

Within a dynamic screen production environment, there are now new possibilities to expand cinematic space and representations of travel that can make them more interactive, inclusive and immersive, to not just present images but to surround them with a world. Viewfinders tries to do this by creating an interactive travel experience for the audience that transcends linear narrative documentary and other cinematic representations of travel. It aims to present moving images of the world as a non-linear voyage on mobile devices that can be accessed anywhere and everywhere, expanding both the experience of travel and the experience of cinema. It moves away from linear story-telling, and it has no “meaning” in the conventional sense. If there is meaning to be derived it comes from the relationship between the viewer, the various film clips and how these are experienced. This is achieved in two stages.

The first part of Viewfinders is an online curation platform that allows participants to provide short (up to one minute) traveling shots captured on smartphones for an immersive documentary experience that both situates travel images into the world and creates a world around them. Viewfinders is built exclusively with travelling shots or tracking shots filmed on mobile devices and while on the move, because they represent best the experience of movement and travel, the freedom to travel, different modes of travel and the trajectory of travel. While film history is full of attempts to explore and expand screen space with various camera movements, thanks to mobile cinema this has become a lot easier. Moreover, the screen itself has become mobile. By uploading one-minute traveling shots shot on mobile - video taken while on the move, for example from moving vehicles - participants also contribute to an ongoing broader creative and conceptual investigation into the content and aesthetics of travel films made on mobile devices. In the meantime, the video database created serves as the basic material for the AR app, and might later also be used in a more “conventional” linear travel essay film.

The second part of Viewfinders, the AR app (which is the focus here), offers participants the opportunity to observe how others have experienced the spaces through which they travel, and discover a new view that they see forming before them. Viewfinders allows the participants to be transported to different places on the spot and in the moment, inspired by the place where they are, and that place’s specific landscape: using their screen as a viewfinder, they themselves become “view finders.” The title, Viewfinders, refers to the technical word for a “viewfinder” on a camera, pointing to the use of new technology such as mobile devices and smartphone filmmaking, which are used as viewfinders themselves, and how these have changed the way we “point and shoot” – see for example how the typical body posture of a smartphone photographer or videographer is completely different from using a traditional photo or video camera, when one has to hold the camera close to the eye as to look into the viewfinder. In this sense, Viewfinders would translate as “Visores” in Spanish. But the title also refers to fact that it requires active participants, people who search for and find landscapes, cityscapes, places visited and who are mobile, on the move. In this sense, the title would translate as “buscadores de vistas” in Spanish. In English, with some imagination, the title “Viewfinders” can actually mean both at the same time. The project itself, embodies both uses and meanings.

Viewfinders builds on the tradition of travel essay films as it offers viewers an experience of space that “redefines the self within a constantly shifting elsewhere” (Corrigan 2011, 105) except that it does this literally: the space is no longer limited to what is presented on the screen (places travelled and filmed by someone else) but includes any place where the viewer is situated when they are using the app, and connects this place to those other places travelled (in so far that they were filmed and submitted). In other words, users/participants can experience the different traveling shots, not in the form of a narrative story created for the screen that replaces reality with a constructed one, but by experiencing it in reality, on their mobile screen, interacting with it in situ and in real time (and with real sounds). As such Viewfinders is part of the next wave of AR applications, those that include context: “Contextual information transforms the AR experience and content, because it now moves from an experience that is the same for every user to one that is specific to you, your location, your interests and your needs.” (Papagiannis 2017, 4) Besides comparing and exploring different perceptions of places from around the world, the AR App also allows the viewer to discover new places, or to see known places in a new perspective. With Viewfinders, we thus hope to contribute to a greater geographical awareness and connectedness.

Viewfinders also has an affinity with a variety of other film genres that have travel as their central theme, and that use traveling shots in expressive ways. The most obvious link is with home movies and amateur travel films that display a similar spontaneous form of “snap-shooting” and “moment collecting” following an “impulse to collect the world and resist, sometimes defiantly, the systematic or the ‘orderly’ as it was being defined in classical cinema” (Orgeron 2006, 77). Home movies implicitly reflect the amateur’s belief in the ability of small film equipment to capture the world. Often, amateur filmmakers show attempts to find more exciting ways to portray travel with an enthusiastic, almost reckless, and definitely unscripted use of travelling shots (Cammaer 2016). A first compilation of travelling shots submitted to Viewfinders provides a good example of this (online 2018).

International Collaboration and Co-Creation

Viewfinders offers the possibility to understand and experience the world as a traveler, through the experience of movement and mobility, as mobile cinema, as well as to connect with other travelers. Viewfinders enables one to make new connections to peers through exchange and co-creation, and possibly to interact with the entire world while being out and about. Rather than providing information a la Lonely Planet, Wikipedia, or other narrative elements, the notion of traveling itself becomes the experience that is shared and interacted with. In the same way as graphical editing works, Viewfinders connects locations with other travelling shots that users, or rather peer-creators, decided to upload. Co-production and co-creation is at the heart of the Viewfinders AR app as it augments the travel experience. While AR in its earlier version was a closed system, Viewfinders opened it up, allowing users not only to be consumers, but also contributors of the content. The project is by creatives for creatives that appreciate travel films and smartphone filmmaking. As a creative arts research project, it explores the intersection of immersive media and travel films, and addresses documentary issues beyond medium specific distinctions. Moreover, as creatives/researchers we are working within the liminal space known as ‘practiced-based-research’ or ‘research-creation’ as it is referred to in Canada.” (Cammaer, Fitzpatrick, Lessard 2018, 6) The project is thus collaborative on different levels and in multiple ways: the research team is international (Canada, Australia, New Zealand/Aotearoa) and interdisciplinary (documentary studies, mobile media, filmmaking and interactive design); we worked together with peers, fellow mobile filmmakers, who were asked to contribute travelling shots and were asked for feedback and suggestions; and lastly, once the project gets launched on a larger scale, basically anyone with a smartphone will be able contribute and help to further develop the project as co-authors, editors and observers of the work.

Beyond Story and Towards Relational Assemblage

The basic premise of Viewfinders as a work of non-fiction that is about traveling (and) landscapes, particularly how it relates smartphones, technology and form, make it one of these works “that do not concern themselves so much with representing life-worlds as with issuing forth novel reverberations.” (Vannini 2015, 12) In the context of AR, Viewfinders blurs the notion of digital and spatial travel experiences as a form of expanded cinema and puts to the test the possibility of making an interactive doc that is not about storytelling, but about offering an opportunity to experience traveling landscapes in a way that allows us to be immersed in the real world, and not in a (pre-constructed) story about the real world. Interactive documentary can do something different, particularly when one takes into account that new media, the technical substrate for interactive documentary, is not linear and sequential like film or video. As Manovich argued over seventeen years ago: “many new media objects do not tell stories; they do not have a beginning or end; in fact, they do not have any development, thematically, formally, or otherwise that would organize their elements into a sequence. Instead, they are collections of individual items, with every item possessing the same significance as any other” (2001, 218).

Since multilinear and interactive media can do other things in relation to nonfiction that addresses the world, with Viewfinders we are exploring the possibilities to create a “relational assemblage” of landscapes traveled and what this can do to create a real-life experience for the viewers. For example, what does it do to someone filming Pukurua Bay in New Zealand from a driving car, to have the AR App connect this view to a similar shot of the coast of Lichnos Greece, followed by Saint-Girons beach in France, also shot from a driving car, and even a shot of Lake Ontario in winter shot from a train window? What if that same shot of Pukurua Bay filmed by a traveller is linked by choosing specific geographical entry points with a shot of the same bay filmed at a different place and time, or even a shot of entering Wellington station by train? (The examples described here can be viewed as part of the promo video for Viewfinders on Vimeo; online 2018). The App allows viewers to alter the relationships between the different shots by choosing different tags (e.g. day/night, mode of transport, sky, water, busy/calm etc.) and obviously also by what they are filming and what the App recognizes in the shot (thanks to the image recognition software Google Cloud Vision API). The App thus “assembles” different relationships between the various traveling shots, and creates an “on the spot” montage that is not meant to “explain” reality, but see it as a combination and interaction of elements - in this case, landscapes - or as Corrigan would call it, “a constantly shifting elsewhere.”

Roadblocks: Dreams and Reality in the Era of AR

Viewfinders explored the opportunities of the digital domain and tried to use these for its creative practice of collaborative filmmaking. Despite the celebratory rhetoric associated with participatory culture there is still only a comparatively small proportion of online users that are actively engaged in audio-visual production, that could be termed ‘practitioners’. Coded video-graphic practices are shaping, generating and reinforcing specific software literacies and the expectations of users, and these tools together, with a host of technologies supporting online platforms such as YouTube, are already shaping the popular imagination over what exactly ‘video’ is nowadays, and how it can be employed. The Viewfinders filmmaking process was actually more like developing software than being a ‘traditional’ film project. A lot of effort went into the exploration of the quickly evolving field of mobile technology and how location-aware technologies allow physical spaces themselves to be inscribed with layers of digital meaning. MAR (Mobile Augmented Reality) interfaces can exploit a full suite of hardware to create new interactions between everyday surroundings and the digital media environment.

Although this was all exiting and appealing to us as the creative team behind Viewfinders, all this also came with a whole series of specific technical challenges, and unexpected restrictions, that slowed the development of the project down and somewhat limited it. Web and Mobile technology changes constantly, and while developing our project we were confronted with the cancellation of TouTube Editor (which we were using for our online curation platform to combine the different travelling shots), various updates by Google and particularly the Geo-Locative Technology are using, as well as various Web browsers (that make the App dysfunctional). Moreover, because we were using online (open source) software, each phase or part of the project quickly risked becoming dysfunctional due to upgrades and/or other constricting moves by the major players in the field such as YouTube, Google and Apple.

Note that the field of AR is still rapidly evolving and continued disruptions are likely to accompany future commercial releases. We also discovered that dedicated video recognition services are only in their infancy, and increasingly specialised functions are still in full development. Some of the other technical applications we needed, such as shot isolation, image tracking, stabilisation and motion detection, have many practical implications for algorithmic editing and video database navigation that further complicated the development of the project. And while continued advances in machine learning are enabling ongoing development of new tools for moving image analysis, at this point, dedicated video recognition services are in their infancy, making it necessary to use and generate video stills of the footage, instead of the actual moving images. These are just some of the technical implications that required a lot of creativity and technical ingenuity, which had an impact on the overall development of the project and its final form.

While we had to deal with various technical limitations and set-backs, our designer Phillip Ruberry was actually able to do a lot with very limited coding skills and technical computer knowledge and INMÓVIL. Año 2019, Vol. 5, N. 2 septiembre 2019. ISSN 2528-7990 we are very happy with how the online curation platform and the prototype for the AR App turned out. The biggest drawback is without any doubt that at this point the Viewfinders App is only designed for Android. Despite that Apple as a company has the image to be there for creatives, it is actually much more restricting once you try to develop something on your own. Also, initially Apple took a long time to jump on the bandwagon of VR and AR, while companies like Samsung embraced these new technologies from the very first beginnings. At this stage, we hope that for the still developing field of MAR (Mobile Augmented Reality) there will be new opportunities for all kinds of creative projects, and that these will be offered by all smartphone companies, Apple included. At the time of writing this article, the Viewfinders project has been paused while it is actually in urgent need of several updates. A major reason for this is that Phillip Ruberry has finished his Masters of Design and has moved on. Dr. Max Schleser and myself are considering various options to continue, expand and revive the project, but it seems almost impossible to do so without support from a commercial developer. As creative researchers, that is a big step we are not ready to make yet. In other words, the future for Viewfinders is uncertain, but open.

Conclusion: Creative and Futuristic Visions

Mobile filmmaking explores the creative fringe, often conveying personal narratives in a unique and innovative manner. Within a dynamic screen production environment, there are now new possibilities to expand cinematic space and representations of travel that can make them more interactive, inclusive and immersive, to not just present images but to surround them with a world. In interactive documentaries, it is through the actions of the participants that the story is moved forward and as they gradually discover new locations, the cinematic space becomes a form of narrative on its own. Viewfinders mimics this aesthetic but also makes the filmmaking process itself part of the viewing experience. Although technology played an important role in the development and creation of the project, and in its limitations, the emphasis of Viewfinders is mostly on creating a meaningful experience for the viewer, and less on being a test case for various new tools. The Viewfinders AR App was developed as blue-sky research to speculate on creative futures and for an engagement with the world based on peer participation, but it is above all a playful experiment in relational assemblage and montage (Cammaer and Schleser 2018). Our ultimate goal with this research-creation project is to create a greater geographical awareness and imagination. The research team comprising Gerda Cammaer, Phillip Rubery and Max Schleser are aware that Viewfinders as a creative arts research project is following a vision that colleagues might consider to be a mobile utopia. But we feel strengthened by some of the current ideas about interactive documentary and AR as a medium that position it no longer primarily as a tool for storytelling, but as a tool for preforming the world critically. Through demonstrating alternative applications of AR and exploring what is possible with imaginative interpretations and augmentations, the team behind Viewfinders hopes to contribute to current developments at the intersection of creative arts, documentary and experimental film, and a more comprehensive understanding of interactive documentary in particular.

References:

VARIOUS LINKS TO THE VIEWFINDERS PROJECT: