JiaoPana, LiangLib,HiroshiYamaguchic,KyokoHasegawab, Fadjar I.Thufaild, Brahmantarae, SatoshiTanakab
aGraduate School of Information Science and Engineering, Ritsumeikan University, Japan
bCollege of Information Science and Engineering, Ritsumeikan University, Japan
cNara National Research Institute for Cultural Properties, Japan
dResearch Center for Area Studies, National Research and Innovation Agency, Indonesia
eBorobudur Conservation Office, Indonesia
Received 20 July 2021, Revised 6 November 2021, Accepted 9 November 2021, Available online 6 December 2021, Version of Record 6 December 2021.
Abstract
We propose a deep learning-based reconstruction method for three-dimensional (3D) cultural heritage objects, which are destroyed or disappear and thus are unavailable for 3D scanning to create digital archives. The proposed reconstruction method uses a single monocular photo and predicts its depth values based on a monocular depth estimation network. While most recent studies have focused on scenes with sharp (high-curvature) edges and large depth differences, such as indoor and outdoor scenes, the proposed method is designed to exert its performance on scenes with small depth differences and soft (low-curvature) edges. The key idea is to involve the soft edges as guidance information in the depth estimation network. We demonstrate the effectiveness of the proposed method by applying it to the hidden reliefs of the Borobudur temple, a UNESCO world heritage site. The hidden reliefs were buried behind stone walls and are no longer visible. The 3D information of the reliefs can only be estimated from their old monocular photos. We achieved 97% reconstruction accuracy for validation data. We also integrate the reconstructed hidden reliefs with 3D scanned data of the visible regions and execute see-through visualization for the integrated data. This visualization can simultaneously show us both visible regions and invisible regions of the temple.
1. Introduction
It is increasingly important to establish three-dimensional (3D) digital archives for the preservation and analysis of cultural heritage sites. With 3D digital archives, complete information can be preserved permanently, which enables people to analyze and appreciate cultural heritage sites. Moreover, we can implement valuable, visual, digital, cultural heritage content, such as high-quality 3D imaging systems and interactive virtual reality systems. Recently, 3D scanning technology has been rapidly developing so that we can easily create 3D digital archives of extant cultural heritage objects (Nuttens et al., 2009, Park et al., 2014 (2014)). However, many cultural heritage sites have been partially damaged and lost. In addition, some or all parts of cultural heritage sites are hidden and made invisible for preservation reasons. In such cases, we cannot directly and three-dimensionally measure actual objects. If an adequate number of photos from different directions remain available, 3D models can be reconstructed digitally using multiple-image-based methods (Hermoza and Sipiran, 2018, Kyriakaki et al., 2014, Kersten and Lindstaedt, 2012). However, in many cases, only one single monocular photo of each cultural heritage object is preserved, which is insufficient for 3D restoration. The ancient reliefs buried behind the stone walls of the Borobudur temple in Indonesia are a typical example.
The Borobudur temple in Indonesia is an UNESCO World Heritage Sites that is known as the largest Buddhist temple ruin in the world. The complete ancient relief carved on its stone walls adds to the cultural value of the Borobudur temple. The temple has 2,672 bas-relief panels (sculptural reliefs in which forms extend only slightly from the background), containing 1,460 narrative panels and 1,212 decorative panels (Santiko and Nugrahani, 2012). These reliefs consist of five sections, each of which describes a different Buddhism story. Unfortunately, due to safety concerns, the foot encasement of the temple was reinstalled, and the section carved on the foot encasement named ”Karmawibhangga” was buried behind the stone walls. This section has 160 relief panels, and only four panels at the southeast corner are revealed and made visible for visitors. The other 156 relief panels are invisible and hidden by the stone walls. For each hidden panel, there remains one grayscale photo taken from directly in front of the reliefs in 1890 (refer to Fig. 1).
