IAD - Interactive Art Discovery

Featuring Paolo Finoglio Masterpiece

Project Proposal

IAD - Interactive Art Discovery is an interactive installation project that connects artwork and visitors through Projected-AR and touch-less sensors.

IAD narrates the artwork with an accessible and natural language; it allows the enjoyment in an innovative and engaging way.IAD is in the process of researching and prototyping a reproduction of Paolo Finoglio’s painting “The torture of Olindo and Sofronia,” housed in the Pinacoteca Civica of Conversano, in the province of Bari.

FIRST DEMO - Proof of Concept

Digitalizing and new Technologies

The digitization of museums and cultural institutions has become a key strategy for enhancing the accessibility and appreciation of cultural heritage. While initiatives by Italy’s Ministry for Cultural Heritage and the European Commission have advanced digital cataloging through platforms like Cultura Italia and Europeana, there is a growing need to move beyond static databases. Emphasis should shift towards leveraging new technologies to foster deeper connections between museums, territories, and visitors, while also rethinking how digital tools can enhance education and communication.

BACKSTAGE TIMELAPSE (Work in Progress)

Digitizing artistic heritage offers vast potential, such as virtual exhibitions, immersive itineraries, and interdisciplinary collaborations with music and theater. These innovations can transform how people interact with and experience art, enriching both knowledge and engagement. In an era shaped by social media and evolving communication styles, museums must adopt creative, high-quality designs that integrate innovative content and advanced technology to tell compelling stories and remain relevant in the digital age.

Hand Tracking

Through the use of Ultraleap Motion Controller and Microsoft Kinect, it is possible to interact with the reproduction of the pictorial or plastic work,The tracking of the hand is processed in real-time by proprietary software that allows navigation within the work. By highlighting sensitive areas, it is possible to access audiovisual content and storytelling.

Projected Augmented Reality

The work of art, recreated in three dimensions, is projected onto its own reproduction printed on canvas, creating with light and visual effects a unique experience in which the work of art comes to life and tells its story to the visitor.

Purpose

IAD does not create a virtual “double” but widens research on the nature of the work, cultural, environmental and historical, technical and constructive, expressive and communicative, whose range can increase and expand in an interdisciplinary way through new interactive modes, different approaches, and users.

Ways and qualities of interaction conceived and designed to make active and subjective reading of art, a participatory use and therefore passionate, from the youngest to the most experienced user.

Developing the metaphor, let’s imagine the digital contribution of IAD as an all-around lens tool through which our perspective vision can delve into the work in detail and as a whole.

In an era in which “technical replicability” reaches its exponential maximum, it is necessary to avoid the easy virtual double, as a flat and deceptive “fake”, building instead a solid structure that perpendicularly deepens and at the same time expands in an embrace all the surroundings of which the work is the fruit, underlies and contains.

Inclusion and openness. Art comes out of the museum.

IAD is accessible and open to all users (visitors, tourists, teachers, students, and children).

IAD allows for a multi-sensory perception of works of art that become “visible” and “ accessible” not only through the eyes but also through an active role of the user who virtually “touches” the work, illuminates it, thus creating the premise of an empathic and engaging experience. IAD allows living the experience of art as a discovery, leaving to the authentic vision of the original its proper role.

IAD allows you to “export” the model from the museum where the work is kept to other contexts such as museums and related realities, galleries, libraries, fairs, exhibitions, thus creating a wider circuit of disclosure also helpful in promoting cultural tourism.

IAD - CASE STUDY

IAD is being developed as an independent R&D project by Blending Pixels, lead by Roberto Santoro.

The collective is investigating the potential of hand tracking as a tool for no-touch interaction in museum installations.

The primary source of inspiration for the visuals was Peter Greenaway’s work on the 2008 multimedia installation dedicated to Leonardo da Vinci’s The Last Supper. Using the projection mapping technique, Greenway created one of the first experiences of “projected augmented reality” (Projected-AR).

At the same time, the founding members of Blending Pixels participated in the first shows in Italy of Architectural Projection-Mapping, which uses the same projection technique.

Projected-AR, combined with Hand-Tracking interaction, allows making interactive any physical object, painting, or plastic artwork. Thanks to the acquisition of art experts and digital dramatists, it is possible to transform artworks into a compelling story, an experience of significant emotional impact.

La Gerusalemme liberata

Paolo Finoglio

The series, commissioned by Giangirolamo II Acquaviva d’Aragona and his wife Isabella Filomarino dei principi della Rocca, was made by Paolo Finoglio, a painter of Neapolitan origin, between 1640 and 1643, to exalt the heroism of the count and the prestige of the county.

The canvases, the only ones by Paolo Finoglio with a profane character, relegate to the background the theme of war (the first crusade occurred between 1096 and 1099) and highlight the poem, the deeds, the loves, the stories of the individual characters.

The canvases, like the poem, are highly theatrical, the gestures are exaggerated, the atmospheres recall those of Monteverdi’s melodramas. On the backgrounds are painted small but numerous figures similar to those of Micco Spadaro, aka Massimo Gargiulo. The foreground is majestic, looming over the viewer for lack of perspective space.

IAD - WORKFLOW

Digital Acquisition - Printing on Canvas

Starting from the work digitized by photography, our graphic crew did a slight retouching to eliminate reflections. Applying an Artificial Intelligence system based on a machine learning Upscale model, the image was enlarged by 400% without data loss.

The resulting output makes it possible to obtain both the texture for the 3d model and the file for printing on canvas the reproduction in scale for the prototype.

The painting has been analyzed and remodeled in 3d, imagining the depth of the portrayed scene and recreating all the materials with advanced 3D-sculpting and Texturing techniques.

The obtained textures were applied to the 3d model of the painting and used for PBR (Physical Based Rendering) in real time. This type of rendering resulted in high-quality visuals, especially in light response.

STORYTELLING

Analysis of the Artwork ~ Storytelling

Once the interactive elements in the painting were identified, the textual content for the story of the work was produced. In the Supplizio di Olindo e Sofronia case, given the nature of the story told in the painting, we decided to write a dialogue between the characters represented.

The dramaturgy is made up and adapted in the interaction diagram, where the narrative logic is defined. Different story tracks can be created that will be freely explored by the user.

Audio Recording ~ Interactive Music ~ Software Development

The 3D model is imported into the proprietary software, and the audio recordings of the texts are associated with the relevant characters. The music, which is also interactive, plays a crucial role in emphasizing the narrative’s most important moments, making the experience even more exciting.

GAMIFICATION | EDUTAINMENT

The visitor can highlight the figures in the painting, as if with a lantern. If he pauses on an active character at that moment of the story, he can listen to his voice. The narration thus takes place in a fragmented manner, gradually revealing the story from a game perspective while preserving the educational aspect of the installation.