Attempts are now underway in various countries to create new user experiences (UX) that meld the real world and the mirror world (the world where everything in the real world is digitized). As one such use case, Nomura Research Institute (NRI) took on the challenge of creating new art experiences using XR technology^* at the 2020 international exhibition of contemporary art, “Yokohama Triennale 2020”. We asked participating members Makoto Irie and Harumi Ishida about their intentions and what they achieved.

• *  

  A collective term for AR (augmented reality), VR (virtual reality), and MR (mixed reality), etc.

Media that can convey three-dimensional information across space and time as a technical unity point

--- Why did you think to try using XR technology in works of art?

Irie: Since the invention of letterpress printing, information has been democratized, eventually leading to the use of interfaces such as paper media, personal computers, and smartphones. However, when expressing things and events in the real world, the information gets degraded because it is dropped into two dimensions. Meanwhile, XR technology, which enables three-dimensional expression, is a medium that can convey realistic experiences and value information that transcend time and space. Some art works have been transmitted over hundreds of years of space-time, and I think they have a high affinity with XR.

With elemental technologies such as 5G mobile communication technology (5G), smart glasses and other XR devices, artificial intelligence (AI), and AI cloud infrastructure, it is now possible to have experiences where the mirror world is overlapped onto the real world through spatial computing, so I made it my goal to create new art experiences that engage other senses, rather than being limited to visuals.

Creating new experience value with animation and voice

--- What kind of work did you do at the Yokohama Triennale, specifically?

Ishida: The exhibition was held at two venues, the Yokohama Museum of Art and Plot 48 in Minatomirai, Yokohama, but there was a little bit of a distance for visitors to come and go between them. Since art works are required to be installed when constructing new buildings in Minatomirai, there is public art peppered throughout the streets. So, we made a plan to allow people to enjoy public art also en route, while guiding them from the nearest station to the venue.

Irie: When the QR code on a postcard is scanned, you’re connected to the app and a map of Minatomirai appears. A number of different ways to get to the venue are displayed, so that even families on their first visit to Minatomirai can enjoy walking around the city on their way to the venue, following routes that pass by works of art owned by private companies like Nissan Motor, Keikyu Corporation, and Shiseido, or that feature colorful art, for example. In addition to providing information about the art, such as the artists, owners, and background, we also included a feature where, by selecting “View in AR”, a faithfully reproduced AR representation will appear on the postcard. We realized that we could provide enjoyment to families with children, and everyone else, through renditions that could not be done in reality, made possible only with XR technology, like playing an animation of an anchor statue falling to the seafloor and adding a metallic sound to impart a sense of realism.

Ishida: Large works can be viewed from above or rotated 360 degrees. It’s interesting that, in addition to enjoying works where they’re actually installed, you can prepare before you go to build up the anticipation, or feel like you’ve brought your favorite art home with you and appreciate it on a postcard. Although the event was at risk of being cancelled amid the coronavirus crisis, the attempt to provide a digital viewing experience was well received by all involved.

Hopes for creating a flagship city for the realm of XR

--- Did the delay in the decision to hold the event have any impact on development?

Irie: After discussions with people from the Yokohama Museum of Art and Yokohama City Hall relating to the Yokohama Triennale 2020, we set up a persona and conducted ethnography and user interviews, for example, to take a human-centered design approach to determine the minimum viable product (MVP), and then made a prototype in advance to involve the concerned parties. Therefore, although we had a rough idea of the direction, we spent only about two months in development for a full-scale production. By adopting agile/scrum as the development method, evaluating the app based on moving objects, and coordinating with concerned parties, it was possible to release the app in a short period of time. However, it was difficult to render works of art in 3D, improve precision, and reproduce them faithfully (photogrammetry). It was also our first time using a mechanism called PWA (Progressive Web App) as a means to reduce the burden on visitors while making it in time for the date of the event.

Ishida: Conventional systems development begins with the needs and problems of the clients, but this time we experienced creating from scratch and seeing through to implementation with the seeds-oriented thinking of “can we do something with XR?” Members with various objectives and motives, such as wanting to create new things with XR technology, to realize collaboration with other companies, to make it an opportunity for on-the-job training for new business creation, or purely for the love of art, came together to make voluntary efforts. This project succeeded because the abilities of diverse talents, including collaboration with outside companies, were brought together, and I came to realize the importance and responsiveness of “co-creation.”

--- How are you considering using this experience going forward?

Irie: It’s expected that XR will be one solution for media that convey three-dimensional messages that feel close to real life. The next Yokohama Triennale will be held three years from. The Yokohama Museum of Art will soon be closed for large-scale renovation work, so I hope that we provide value using XR technology during that blank period. In addition, although the target of this time was 9 works, by expanding it to include works of art throughout the city and creating opportunities to become acquainted with the history and culture, perhaps the charm of the city can be rediscovered and people’s behavior patterns and consciousness will change. I think it would be pretty cool if we could create positive cycles through digital marketing using the media of XR, such as improving the services of commercial facilities by utilizing the data acquired by the app, and have private companies in the Minatomirai area combine their respective strengths to make it a flagship city in the realm of XR.