Artificial intelligence in art conservation and preservation: Experience from the Ghent Altarpiece


During the complex restoration process of the famous Ghent Altarpiece, some of its mysteries that puzzled art historians for a very long time were unraveled. Alongside the restoration, research projects were carried out that revealed so far unknown and surprising aspects of the painting. Aleksandra Pižurica, professor in Statistical Image Modeling and head of the research group Artificial Intelligence and Sparse Modelling, and her team contributed to Phase I (2012-2016), the treatment of the exterior panels, and Phase II (2016-2020), the lower panels of the interior.

Paint loss detection

In the 1950s, conservators already discovered that parts of the altarpiece had been overpainted, however, they were lacking the technology and time to determine and characterize the overpaints exhaustively. Aleksandra recounts how during the investigation it became apparent that 70% of the surface of the outer panels was overpainted. When these overpaints were removed during the restoration process, a different style of painting became visible. The original work by Hubert and Jan Van Eyk is indeed more dramatic as can be seen, for example, in the elaborately painted draping of some of the garments.

The Ghent Altarpiece - Lam Gods
Left: the face of the lamb before restoration. Right: the face of the lamb after the overpaint layers were removed. ©Aleksandra Pižurica

‘The most dramatic find when revealing the paint layers of the main panel was the expressive face of the lamb. This more humanised face of the lamb attracted a lot of media attention when it was revealed.'

However, the original paint layers also contained more pronounced cracks and paint losses. Restorers asked whether we could develop an automatic method for paint loss detection. Detecting and documenting such deteriorations is important for the decision making process for the actual restoration, but is a laborious manual task that is prone to errors: The semi-automatic tools that are available also require a lot of manual work, and only allow for relatively rough annotation. Aleksandra's work has focused on automating this task, using multiple imaging modalities of which some were recorded before the treatment (digital macrophotography, infrared macrophotography, X-radiography), and some while the treatment was in process (digital macrophotography and infrared reflectography).

For the paint loss detection her team used a set of image sections with the paint losses annotated manually by conservators and then taught a machine learning algorithm to do the same. A machine learning model could then be used for the classification of fragments as paint losses or cracks. Once these were detected and labeled, automatic methods were developed to virtually fill in the gaps and cracks, thus obtaining a virtual restoration of the painting. As the physical paintings were also restored manually, this provided the opportunity for a qualitative comparison with the digital restoration.

John The Evangelist
Details from the panel “John The Evangelist”. Left: Detail of the painting before restoration. Middle: the paint loss detected by the algorithms. Right: Inpainting result on the detected cracks. ©Aleksandra Pižurica

Virtual restoration was for example used to make the text on a painted book more readable and to discover whether a real text was used or a combination of calligraphic characters. After a virtual inpainting of the cracks, the letters became more visible and hence suitable for further investigation.

Phase 3

The third restoration phase will start in 2022. Aleksandra and her team will specifically target the painting of pearls, present in the upper panels of the Ghent Altarpiece. They will develop approaches able to extract some kind of digital signatures of the painted pearls with more recent machine learning methods.

Technical Art History series on Digital Tools for Cultural Heritage

A few weeks ago prof. Aleksandra Pižurica gave a lecture on on this topic. The seminars are part of the Technical Art History Series in Advanced Imaging Technologies for Cultural Heritage organized by the Rijksmuseum, the Computational Imaging group at CWI  Amsterdam, and the Venice Centre for Digital and Public Humanities.

Missed it? You can watch the complete seminar here.

Source: Looking through art - Imaging the Ghent Altarpiece: insights from an interdisciplinary research and restoration project by Erma Hermens.