Blurred Lines: Art, Science, and the Race Against Time

 Spanning centuries, crossing international borders, and blurring academic boundaries, researchers from Chicago, the Netherlands, France, and Italy have collaborated under the National Science Foundation's Partnerships for International Research and Education (PIRE) program in order to solve problems related to the conservation and study of cultural heritage objects.

 "CuBISM is focused about trying to make connections between material properties and visual properties," says Oliver Cossairt, co-primary investigator of the Computationally-Based Imaging of Structure in Materials (CuBISM) program and Associate Professor of Computer Science and Electrical Engineering at Northwestern University.

 CuBISM is centered around the implementation of new systems and techniques both to better image and visually display as well as to better understand the internal molecular structure of a material. In this case, the material is art. Why art?

 "Art objects are some of the most challenging examples of things to study ," says Cossairt.

 Through these two main avenues, materials modeling and computational imaging, questions posed by the artworks themselves are studied, responding to the needs of art historians and conservators around the globe.

 According to Aggelos Katsaggelos, co-primary investigator of CuBISM, the secondary objective is to educate the students beyond the scope of this program.

 "They are using art as an example, as a vehicle. They are able to answer scientific questions, develop tools that could be used outside art… and PIRE being this international collaboration grant gives this additional capability of the students to interact with international colleagues," he says.

 This research crosses many academic boundaries, from Chemistry and Materials Science to Computer Science and Engineering Science and Applied Mathematics. Collaboration is key to this project's success, and to the success of science in general.

 "The best way to do science… is to just ask an interesting question and then do whatever it takes to answer that interesting question," says Lionel Fiske, a graduate student in Engineering Sciences and Applied Mathematics working on forward image modeling in the program.

 Researchers at CuBISM have been investigating a number of interesting questions, prompted by art and pursued by science.

 What happens inside a paint layer as a painting dries?

 Rebecca Harmon, a graduate student in Chemical and Biological Engineering, has been investigating the initial curing chemistry of oil paint by computational modeling.

 "I'm using the computer essentially to perform experiments on how it cures and getting a look inside a paint film without having to take a slice," she says.

 Harmon's advisors on the project, Linda Broadbelt from Northwestern University and Piet Iedema from the University of Amsterdam, each approach the experiment with a different model. Broadbelt's kinetic Monte Carlo approach is able to track large numbers of chemical species, providing a high level of molecular detail. However, the kinetic Monte Carlo approach is not as well suited for extension to fully cured states where the polymer network is highly crosslinked, a task better suited to Iedema's deterministic model. Harmon's work is melding together the two approaches, finding ways in overcoming the barriers for each.

 "Oil paint has been around for centuries, and we've been studying it for about a hundred years... but there are still some questions about what actually is going on in there."

 Linseed oil, a common pigment binder used by artists from Jan van Eyck to Pablo Picasso, is favored for its slow-drying nature, meaning that artists could take their time blending each color and section of the painting until it was just right. On the molecular level, however, things get complicated quickly as it dries.

 "The linseed oil molecule is pretty crazy; there are three fatty acids on it," Harmon explains, so they have started by focusing on one smaller molecule, ethyl linoleate, their model monomer, running simulations and models to examine how it cures at the molecular level.

 For Harmon, the work has been particularly satisfying, as the models they study can be used to check the work of prior chemists who didn't have the technology to simulate how chemical reactions play out at this level of detail. On the cultural heritage side, Harmon says her eyes were opened to the possibilities of combining chemistry and art and "using these tools to better protect things and be better stewards for different objects." The PIRE program itself also had a number of different draws.

 "It had art and chemistry and traveling and education and outreach and I was like, 'Sign me up. This is it."

 How do the properties of the paint change as it dries? How do they change in different conditions?

 For Gwen dePolo, a graduate student in Materials Science and Engineering, NU-ACCESS was not her first experience mixing art with science. As an undergraduate at the University of Nevada Reno, she received a Bachelor of Science in Chemistry and a Bachelor of Arts in Music. An internship at the Library of Congress led her to this field.

 "It was a very cool way for me to do science and have a very practical application and that's how I ended up looking for heritage science-based programs," she says.

 Like Harmon, dePolo's work also seeks to understand linseed oil. More specifically, dePolo utilizes the rheometric quartz crystal microbalance (QCM) in order to measure how the oil’s properties change both over time and across different conditions.

"What is the point where the material goes from a brittle/glassy response to a softer/rubbery response?" she asks.

 This is just one of the questions conservators consider as they work on painted surfaces, as the surrounding environment has many influences on the way the material itself will respond to treatments and handling. DePolo is able to test her samples under different conditions, altering temperatures and humidities to measure how they respond.

 "We're able to look at how properties change and create these small samples that we can destroy and subject to really extreme conditions, which we just can't do with the real objects."

 In order to anticipate the needs of the real objects, dePolo considers the international aspect of the PIRE program crucial to informing her work.

 "There has been a large thrust of the international components," she says, "so a lot of the projects that I'm working on--the linseed oil one--comes from my supervisor in Amsterdam, Piet Iedema's group, and that's a very heavily motivated project from his group so that's been very steeped in working in collaboration with the Rijksmuseum and working within that field."

 How does light affect titanium dioxide-based paint?

 A newcomer in the painting world, titanium dioxide as a pigment was only invented and first used a little over a century ago. Unlike the pigments used by the Old Masters that have been studied for hundreds of years, the way titanium dioxide degrades over time is relatively unknown.

 Today, titanium dioxide-based paint is among the most common paints there are, raising questions about how to care for paintings that contain this pigment in the future.

 Thomas Schmitt, a graduate student in Materials Science and Engineering, has been studying how titanium dioxide-based paints react to visible light. It is well known that titanium dioxide absorbs UV light, which is why it's found in many sunscreens. In paintings, however, UV light causes this pigment to degrade.

 "It led to kind of the natural question of, well, is this only UV light or is there more going on?" says Schmitt.

 His first experiment involved leaving the paint out in light and then seeing how it reacted. What he discovered was that there was a reaction happening: the usual long chains and networks of oil molecules were cut shorter by the titanium dioxide acting like "scissors," providing evidence of degradation. Schmitt just needed a way to find out why.

 As part of the international collaboration of the PIRE program, researchers at the University of Perugia completed theoretical experiments to figure out why this was happening. Using computational models, they were able to study how a single oil molecule and a single titanium dioxide molecule reacted.

 "We could never do that in real life, one oil molecule and one titanium dioxide particle," Schmitt says "so that was cool, to be able to then understand what was going on and to find out that they were reacting prior to the light exposure to allow the titanium dioxide to absorb visible light."

 By studying how titanium dioxide-based paints react to visible light, Schmitt hopes that his research can help conservators find ways to prevent light-based degradation in the future. Even though a lot of work in art conservation is trying to protect paintings centuries older than those made with titanium dioxide, there is still plenty of time for further research to take place.

 "I think the flipside of that is we also have a lot of time to prevent the changes happening in these paintings."

 How do you document a hologram?

 Pengxiao Hao, a postdoctoral student in Materials Science and Engineering, studies holograms, an art form that has rarely been probed in the conservation field. Working in collaboration with the MIT Museum, which houses one of the largest collections of holograms, Hao has been able to have access to these unique artworks for study.

 Focusing particularly on two series of works from famed holographer Stephen Benton, a lot of the challenges faced from studying these objects originate from the three-dimensional nature of the material itself.

 "It's not a static object. It's dynamic," she says, "when [people] view holograms they kind of dance in front of it to get different views."

 A hologram is an image generated from the interference of wavefronts that is recorded on a physical medium. This medium, photographic film, contains an immense amount of information in a very small space, a thickness of about 20 microns, making digitizing the film difficult. Hao focuses on using a range of techniques to image the three-dimensional holograms instead, trying to render the art object in as much detail as possible.

 By using DSLR, hyperspectral imaging, Optical Coherence Tomography (OCT) and XRF to image the holograms, different aspects are revealed with each method.

 "The color of the hologram reflects its physical structures, and hyperspectral cameras can very accurately capture that color on a nanometer scale, "she says.

 In addition, XRF and OCT help identify manufacturing and processing techniques,understanding the chemical recipes artists used to make the hologram. Silver halide is one common element involved in the process, but some artists used their own recipes, which remain a mystery. Studying these objects and their materials is especially valuable to places like the MIT Museum and to researchers like Oliver Cossairt, who advised Hao on the project.

 "I was just very interested in how can we conserve this? How do we preserve this for future generations? How can someone in the future enjoy this collection the way that I did?" says Cossairt,

 Hao says, " We hope that we can build something like an online viewer so that people can slide bars and be able to see the details of the holograms."

 By imaging these holograms and making them available to a wider audience, Hao hopes to share the experience of viewing the holograms but in a digital format.

 "What Pengxiao has shown is that you can make a complete recording of the entire hologram," says Cossairt, "You can basically create these virtual experiences of viewing the hologram as though you were really there in the gallery."

 How do you separate the layers of an x-ray image of a painting?

 In some cases, looking beneath the surface can obscure more than it illuminates. For Bingjie "Jenny" Xu, a graduate student in Computer Science, and Lionel Fiske, a graduate student in Engineering Sciences and Applied Mathematics, distinguishing between what's on top and what's just underneath is a complex problem that requires complex solutions.

 Being able to separate the layers of these images can reveal important information about an artist's process. Few artists paint directly on top of a blank canvas, so these imaging techniques often reveal things such as preparatory sketches, and can spot where artists made changes to their compositions. Sometimes, entire paintings were covered up and the canvas reused.

 "Traditionally, the scientists need to take a little piece out of the painting and do the microscope scanning to see what is in the internal layer," Xu says, "but that is destructive and you cannot apply that to the whole painting, so we are trying to develop a new method that can scan the internal materials, the pigments inside the painting, in a noninvasive and nondestructive way."

 The use of  XRF, or X-ray fluorescence, to peer beneath the surface of a work of art is an established technique that involves imaging the work using X-rays, which reveal and identify the chemical elements within, both at the surface level and underneath it.

While the XRF image may reveal every bit of paint and pigment on a canvas, being able to discern which elements belong to which painting or image is the problem.

"If you imagine that there's a couple of layers that have been completely hidden--have been completely painted over--well, you're gonna see those too, and, as a result, we actually see sometimes is a couple of images super positioned on top of each other, kind of squished together," Fiske says.

While there are many potential methods for separating superimposed layers, Xu and Fiske have utilized two separate additional measurement modalities to try and solve this problem. In addition to the XRF, Xu has been looking at Optical Coherence Tomography (OCT) and Fiske has been looking at hyperspectral imaging as other sources of information.

OCT, often used in medical fields, is a noninvasive way to image an object in two and three dimensions, providing information about internal structure, but not the exact material composition.

"We would like to use a machine learning method to combine the information we get from these two [XRF and OCT] techniques and to find a way to somehow combine them and get both the layer structure and the material information at each layer," says Xu.

Fiske hopes to do something similar, using imaging spectroscopy, a much simpler technique than XRF, as the source of additional information. Imaging spectroscopy measures the wavelength of reflected light, taking in information about color across the spectrum.

"What you can do [with imaging spectroscopy] is you can look at just how much of these wavelengths are reflected at each point across that painting, so if you have a blue pigment it's gonna reflect a lot of blue light," Fiske says.

Because reflected blue light suggests the presence of blue pigment, the imaging spectroscopy can be compared with the XRF image, which reveals the chemical composition in more detail. In addition, the imaging spectroscopy is only surface-level, which provides clues as to which parts of the XRF image correspond to surface-level pigments, and which are hidden underneath.

 "This allows us to sort of digitally remove them from the total image."

By removing the surface details, the idea is that the XRF image of the underlying components will become easier to visualize. Fiske hopes that one day this sort of technique will give both art historians and conservators new tools and new techniques to make their jobs easier.

"To ask these really interesting questions about the history of objects and where they come from and these things are really murky when you're just going through books," he says.

 How can XRF scans of paintings be created more efficiently?

In the field of cultural heritage science and conservation, having XRF scans revealing the underlying chemical structure is a critical tool to begin study into an object. However, the typical process of scanning, a raster scan, takes up to an entire day for just one painting. As the scanner trudges along the canvas in parallel lines, it spends the same amount of time dwelling on each point.

 Henry Chopp, a graduate student in Electrical Engineering, has been investigating ways to streamline the scanning process by developing algorithms to identify the parts of the painting that are most critical to scan.

 "The problem is intelligent acquisition of information. You want to build the camera, no matter what the modality is, to look at the scene and you ask the question, which parts of the scene are the most important?" says Katsaggelos, who worked with Chopp to answer this question.

 Having developed these algorithms to determine which parts of a scene to focus on, the next step was telling the scanner how to go about scanning the painting.

 At first, they tried to implement a system where the previous set of points scanned would determine the next set of points for the equipment to scan, focusing more on the important elements of the painting, such as the flower centerpiece, rather than the background. However, this system lacked efficiency as the scanner would dart from one side of the canvas to the other, taking time to travel between each point.

 So, Chopp returned to the raster scan's pattern, but switched up the timing.

 "Instead, we vary our dwell times at each point so maybe in the background we say just do a quick raster scan, but once you get into the flowers then you start to take more time and actually do some better scans."

 This model can be extended beyond XRF, and is applicable to any scanning technology that has a point sensor, such as OCT.

 "You take a picture of the painting, you upload that to our model, and then it'll tell you how long to sample each point. The other benefit is that you can tell the program how long you want to spend on it, so if your scan is supposed to take ten hours and you want a two hour scan you can do that," he says.

 To Chopp, the ultimate benefit of this technology is the customization available, making it possible to scan three or four paintings a day, according to the institution's needs.

 "When you think of institutions like the Art Institute [of Chicago] or any other museum, really they have God knows how many pieces of artwork they'd like to scan, but some paintings are on the waitlist."

 What happens next?

 "The goal is really to measure change," Cossairt says, of what he hopes will become a future endeavor, "what you want to do is just to get some sort of complete visual documentation with as much analytic capability as possible of a work of art and then come back to it later and compare what the measurements that you made now versus the measurements that you made before."

 As the PIRE program comes to a close in 2022, many of the researchers hope that the questions raised under CuBISM become the seeds for future projects, as art objects continue their individual journeys through time and space.