Author: Artem Reshetnikov, researcher at the Barcelona Supercomputing Center
Andrea Mantegna, was an Italian Quattrocento painter of great talent. He started his artistic career at the age of 10, when he became the apprentice of Paduan painter Francesco Squarcione. With the purpose of having Andrea learn more than he knew, Squarcione had him study plaster copies of ancient statutes and some paintings on canvas, mainly originated from Tuscany and Rome. Mantegna learned a lot in his youth and when he was only seventeen, he painted the main altar of Santa Sofia in Padua, an art piece that seems to be from an expert artist and not from a simple boy.
Between the years 1457 and 1460, Andrea Mantegna painted the triptych of the San Zeno Altarpiece, an art piece divided into three sections: Agony in the Garden, Resurrection and the central panel The Crucifixion. The triptych is located in the Basilica di San Zeno, the main church of Verona, however, the three predellas along with the main picture of the painting have been replaced by copies. The originals are in the Musée des Beaux-Arts in Tours and in the Louvre.
Crucifixion is one of the most important iconographic scenes of Western Art and many researchers of art would say that it is the most complex one. It is always full of different objects which have significant meaning for scene understanding because they bring in an additional layer of symbolic information.
Figure 1. The Crucifixion, central panel from the San Zeno of Verona altarpiece, Andrea Mantegna (1457 – 1460)
Within the framework of Saint George on a Bike, a project that aims to enrich the descriptive annotations of paintings automatically based on AI algorithms, researchers have chosen The Crucifixion as one of the paintings to be analysed and enrich its annotations with automatic descriptions.
In the scene (Figure 1) we can see a classical representation of crucifixion with two crucified persons on sides and Jesus Christ in the center. Algorithms of Saint George on a Bike project mainly focused on the detection of iconographic objects and crucifixion is one of these. We can clearly see that all three crosses were detected with high precision. Based on this information algorithms are able easily to define the main topic of the painting.
However, it is often said that the devil is in the details. Our main goal is to understand scenes based on small iconographic symbols. Such symbolic objects are extremely important. First of all, algorithms can detect the crown of thorns. It is a famous iconographic symbol that allows us to understand which one of three crucified persons is Jesus. On the left side, we can see several skulls detected by algorithms. As a result, skulls have formed a significant part of Christian iconography, at the foot of the Cross in many religious paintings, as a synecdochic reminder of Christ’s death, and by extension as a memento mori for generations of believers and, in later art, as a secular symbol of human frailty.
On the left side, algorithms were able to detect a set of persons, one prayer, and a halo. Halo is an important iconographic symbol in Christian Art as well. Traditionally it is a radiant circle or disk surrounding the head of a holy person, a representation of spiritual character through the symbolism of light. A halo and a prayer indicate that this group of people is Christ’s followers. According to the description of the painting, there is no clear information about who it was but it seems that the four Marys are there: Holy Mary who has fainted, Mary Magdalene, Mary of Clopas, and Mary Salome. It is unclear who the figure in the red(prayer) is, but it could be Nicodemus. On the right side, we see the group of Roman soldiers. Algorithms were easily able to detect sets of persons, lances, helmets, knights, horses, and shields.
However, we have to admit that simple object detection does not allow us to understand the scene. In line with Saint George on a Bike’s goals, identifying individual objects in an image, e.g. of a sketch or a painting, is one crucial step towards complex scene description and objects’ labels semantic disambiguation. Simply detecting a sword and a helmet or halo in a scene is clearly not the same as also detecting a knight or a holy person. An algorithmic (automated) approach to reveal how image’s detected objects connive by their relative positions and interconnected visual cues to produce meaning (whether symbolic or literal) in a complex composition is a topic of intense research in Machine Learning and Artificial Intelligence.
Saint George on a Bike: Training AI to be aware of cultural heritage contexts
Automatic image captioning is a process that allows already trained models running on commodity computers to generate textual descriptions from an image. It is a burgeoning reality in a handful of other areas such as classifying image contents on social media. However, to date, no AI system has been built and trained to help in the description of cultural heritage images, while factoring in the time-period and scene composition rules for sacred iconography from the 14th to the 18th centuries.
As part of the Saint George on a Bike project, researchers at Barcelona Supercomputing Center build and train AI systems to help cultural heritage institutions describe and classify their art pieces automatically. In the end both casual users and cultural heritage professionals will benefit from a better access to collections and also a better experience navigating through collection catalogs. They will owe this to richer artwork annotations, leading to improved image scene indexation and search capabilities, obtained with the help of a specialized AI system.
To learn more about the Saint George on a Bike project, visit https://saintgeorgeonabike.eu/.
