Algorithm discovers hidden connections between paintings at the Metropolitan Museum | Instant News

A team from MIT helped create an image retrieval system to find the closest matches to paintings from different artists and cultures.

The machine learning system developed by the Massachusetts Institute of Technology was inspired by an exhibition at the Rijksmuseum in Amsterdam. The exhibition featured Francisco de Zurbarán’s “San Serapion” (left) and Jan. “Endangered Swan” by Jan Asselijn.

Image courtesy of MIT CSAIL.

People often see art as the greatest journey of the past, consolidating a moment in time and space. Let us temporarily escape from the current beautiful vehicles.

With an endless treasure trove of paintings, the connections between these artworks from different times and spaces are often overlooked. Even the most knowledgeable art critic cannot absorb millions of paintings over thousands of years, and cannot find unexpected similarities in themes, themes and visual styles.

To simplify this process, a team of MIT researchers Computer Science and Artificial Intelligence Laboratory (CSAIL) and Microsoft created an algorithm to discover hidden connections between paintings between the Metropolitan Museum of Art (Metropolitan Museum) and the Rijksmuseum in Amsterdam.

Inspired by the National Museum’s special exhibition “Rembrandt and Velaquez”, the new “MosAIc” system understands the “closeness” of the two images through an in-depth network and discovers paired works from different cultures, artists and media Or “similar works”. In that exhibition, the researchers were inspired by an unlikely but similar pairing: Francisco de Zurbarán’s “San Serapion The Difficulty” and Jan Arthurlin (Jan Asselijn) “The Threatened Swan”, these two works portray a profound altruistic scene with a creepy visual similarity.

CSAIL doctoral student Mark Hamilton said: “These two artists have no correspondence or encounters in their lives, but their paintings suggest that the rich, latent structure is the basis of their work.” Mosaic. “

To find two similar paintings, the team used a new image search algorithm to find the closest match for a specific artist or culture. For example, in response to a query about “which instrument is closest to the painting of this blue and white dress”, the algorithm retrieves an image of a blue and white porcelain violin. These works are not only similar in style and form, but also stem from the extensive cultural exchange of porcelain between the Dutch and the Chinese.

Hamilton said: “The image retrieval system allows users to find images that are semantically similar to the query image. They are the basis of reverse image search engines and many product recommendation engines.” Limiting the image retrieval system to a specific subset of images can be visually relevant. Relationships in the world generate new insights. Our goal is to encourage people and creative artifacts to reach new heights. “

How does this work

For many people, art and science are irreconcilable: one is based on logic, reasoning, and proven truth, the other is based on emotion, aesthetics, and beauty. But recently, artificial intelligence and art have a new flirtation, and this flirtation has become more serious in the past decade.

For example, a large part of this work previously focused on using AI to generate new art. Have Gao Gan Project developed by researchers at MIT, NVIDIA, and the University of California, Berkeley; previous work by Hamilton and others GenStudio Project; even art sold with AI at Sotheby’s $ 51,000.

However, the purpose of MosAIc is not to create new art, but to help explore existing art. A similar tool, Google’s “X resolution“Find the art path connecting the two artworks, but the difference of MosAIc is that it only needs one picture. It does not find the path, but finds the connection in any culture or media that the user is interested in, such as Find the shared art forms “Anthropoides paradisea” and “Seth Kills the Snake, Hibis Temple of Amun”.

Hamilton pointed out that building their algorithm is a difficult task because they hope to find images that are not only similar in color or style, but also similar in meaning and theme. In other words, they want dogs to be close to other dogs, people to be close to other people, and so on. To achieve this goal, they explored the internal “activation” of the deep network for each image in the open access collections of the Metropolitan Museum and the National Museum. How do they judge image similarity, that is, the distance between the “activations” of this deep network commonly called “functions”.

In order to find similar images between different cultures, the team used a new image search data structure called a “conditional KNN tree”, which combines similar images into a tree structure. To find a close match, they start with the “trunk” of the tree and then follow the most promising “branch” until they are sure to find the closest image. The data structure improves on its predecessor by allowing the tree to be quickly “pruned” for a specific culture, artist or collection, thereby quickly responding to new queries.

Hamilton and his colleagues are surprised that this method can also be used to help discover the problems of existing deep networks, which are related to the recent emergence of “deep fakes.” They apply this data structure to find areas where probabilistic models (such as Generative Adversarial Networks (GAN) that are commonly used to create advanced products) crash. They called these problematic areas “blind spots” and pointed out that they gave us a deep understanding of how to bias GANs. This blind spot further shows that even though most fakes can deceive humans, GANs still have difficulty representing specific areas of the data set.

Test MosAIc

The team evaluated the speed of MosAIc and how close it is to human intuition for visual analogies.

For the speed test, they want to ensure that their data structure provides value in simply searching the entire collection through fast, powerful searches.

In order to understand the degree of coordination between the system and human intuition, they produced and released two new data sets for evaluating conditional image retrieval systems. A data set poses a challenge to the algorithm, even after it is “styled” using neural style transfer methods, images with the same content cannot be found. The second data set challenges the algorithm to recover English letters in different fonts. In less than two-thirds of the time, MosAIc was able to recover the correct image from the “haystack” of 5,000 images at a time.

Hamilton said: “Looking forward, we hope this work will inspire others to think about how information retrieval tools can help other fields such as art, humanities, social sciences, and medicine.” These fields are full of information that has never been processed using these technologies. Can bring great inspiration for computer scientists and domain experts. This work can be extended with new data sets, new query types, and new ways to understand the connections between works. “

Hamilton wrote the paper on MosAIc together with Professor Bill Freeman and MIT students Stefanie Fu and Mindren Lu. The MosAIc website was established by MIT, Fu, Lu, Chenbang Chen, Felix Tran, Darius Bopp, Margaret Wang, Marina Rogers and Johnny Bui in the Microsoft Garage winter internship program.

/ University issued. Full view Here.


image source

to request modification Contact us at Here or [email protected]