“Remediatizing Visual Media as Sonic Data. Sonification and Cultural History.”

« Réencoder les médias visuels en données sonores. Sonification et histoire culturelle »

Abstracts

Like Jonathan Sterne before him, American historian Michael J. Kramer considers the digital sound sources from a sociocultural perspective as objects of variable qualities that have been encoded by following a protocol that needs to be documented. In his research project on the Berkeley Folk Music Festival, he observes a digital photograph of blues singer and guitarist Mance Lipscomb taken during a concert in July 1963 and questions the nature of the gaze directed by the historian on this “digital silhouette”. By drawing on the contributions of the “auditory turn” and the possibilities offered by the processes of “sonification” of data, the historian proposes that computational processes of remediation can illuminate new dimensions of digital archives through defamiliarization of visual representation and an expanded range of sensory comprehension. In this interview, the historian explains his experiments, his methodology and the stakes of data sonification for cultural history.

Dans le sillage des travaux de Jonathan Sterne, lhistorien américain Michael J. Kramer analyse les sources sonores numérisées dans leur dimension sociale et culturelle ainsi que dans leur dimension matérielle, les envisageant comme des objets encodés selon un protocole qu’il s’agit de documenter. Dans le cadre d'un projet de recherche sur le Berkeley Folk Music Festival, il observe une photographie numérique du chanteur et guitariste de blues Mance Lipscomb prise lors d'un concert en juillet 1963 et interroge la nature du regard porté par l'historien sur cette « silhouette numérique ». En s'appuyant sur les apports du « tournant acoustique » (Acoustic Turn) et sur les possibilités offertes par les processus de « sonification » des données, lhistorien nenvisage pas seulement de changer son regard sur larchive numérique, mais d’en faire évoluer sa compréhension sensible. Dans cet entretien, il explique ses expériences, sa méthodologie et les enjeux de la sonification des données en histoire culturelle.

Index

Mots-clés

Sonification, Sound studies, histoire culturelle du sonore, tournant acoustique, numérisation, Berkeley Folk Music Festival.

Keywords

Sonification, Sound studies, cultural history of sound, auditory turn, digitization, Berkeley Folk Music Festival

Outline

Text

Involved in both research in Digital History and in the field of Sound Studies, historian Michael J. Kramer has laid the groundwork for a sonic analysis of images1. As part of a research project on the Berkeley Folk Music Festival, the historian observes a digital photograph of the blues singer and guitarist Mance Lipscomb taken during a concert in July 1963 and questions the nature of the gaze directed by the historian on this “digital silhouette”. By drawing on the contributions of the “auditory turn” and the possibilities offered by the processes of “sonification” of data, the historian does not only plan to change his outlook on the digital archive, but also to change its sensory comprehension in service of generating new interpretations of the past.

Kramer conceives his methodology through three types of action: the use of digital sound design has made it possible to “amplify the meaning” of a historical event. The practice of data fusion (a concept borrowed from Lev Manovich) has led to the production of a new media object useful for historical inquiry by systematically connecting visual archives and sound archives pertaining to the same event. Finally, data sonification proceeds from another approach to visual and sound archives by correlating visual data to sound outputs. Like Jonathan Sterne before him, Michael J. Kramer considers the digital sound sources for what they are, namely objects of variable qualities that have been encoded by following a protocol that needs to be documented2. Kramer proposes to reverse the current practice of visualizing sound data (as in the case of the sonogram). Thus, by inviting us “to hear an image while listening to its digitized data”, Kramer establishes a new kind of historical hermeneutics of visual sources.

“I did not see this until I heard it”. Hearing as a mode of historical investigation

JSN: In an essay published in Digital Sound Studies, edited by Mary Caton Lingold, Darren Mueller and Whitney Trettien, you wrote something which struck me as being particularly telling of your methodological approach: “I did not see this until I heard it”. The expression is self-explanatory but it highlights a very important concern for cultural history. At a time when an increasing number of historians are interested in acoustic archive materials, you outline a different intellectual and sensory relation to sound. For you, sonification constitutes a creative and challenging way to renew historical interpretation. In our classical hypothetico-deductive method, vision often seems to take precedent over the other senses, and especially over hearing. More than twenty-five years after Alain Corbin published the French version of his milestone book Village Bells3, how could you explain that discrepancy and could you please define your own approach?

MK: My approach to sonification arose in a certain moment in the development of digital humanities and out of my archival research. This distinguishes it to some extent from Corbin's groundbreaking scholarship.

In the early 2010s, I began conducting research on the Berkeley Folk Music Festival's history using a large archive of over 30,000 artifacts. It had been the festival director's working archive.4 At the time, digital humanities – and cultural analysis in general, from scholarship to journalism – was entering into its craze for visualization. Large datasets, advocates claimed, could be perceived best through graphical representation, particularly the capabilities of animated visualization that a computer could generate.

I was interested in this turn to data visualization, however I had a problem. The archive I was exploring documented a musical event – a sonic event – however it contained little in the way of sound. There were few audio recordings. What it did contain was a remarkable trove of photographs, over 10,000 easily, of an understudied folk music festival held on the campus of the University of California, Berkeley, between 1958 and 1970. The director of the festival, Barry Olivier, was an amateur photographer and so he had hired students and folk music fans to take photographs of the annual event rather than recording it for audio preservation. The photographs of the Berkeley Folk Music Festival formed the bulk of the archive. The digitization of these images through a grant to Northwestern University Libraries by the US National Endowment for the Humanities has allowed me to explore them not only as digital artifacts, but also as digital data.

The main thrust of digital humanities has been to visualize humanities data in "graphs, maps, and trees," to quote Franco Moretti's book title5, or if thinking about music, to transfo rm sound data into visual spectrograms in order to analyze it by seeing sonic frequencies with the eye; I wondered about the reverse: What would it mean to turn data of photograph which captured musical and sonic experience into sound? Could one start to ask strange questions, such as what does an image sound like? And could this transformation of these historical data that constitute an image into sound yield new insights into the past? Or would it be accurate to ask if they could yield new insounds?

Image 100002010000056A00000362FE3C2077.png

Photographer unknown (possibly Philip Olivier). Courtesy of Berkeley. Processing and analysis of the picture with Photosounder 1.10.

Mance Lipscomb performs at the Berkeley Folk Music Festival, July 1963.

In short, it seemed to me many had visualization mania. So I thought, why not flip the direction? Why not see about (or, more accurately, hear about) the sonification of visual data? When artifacts become surrogate data files, one can transform an input of one sort into a correlated output in another form. It need not only be graphical. Pixel brightness can become tones, features extracted from an image (the shape of a guitar) can become certain melodies or motifs. Vector analysis of where eyes are looking in an audience can become greater or lesser intensities of certain timbres. Since sound must occur temporally, while images are spatial, one also has many options for optically scanning the image: left to right, center to margins, randomly. Yet just as with visualizations, the goal is not so much to discover the one true image sonification so much as to play with the many representations of the artifact that become possible. These can take us more deeply into the suggested meanings, implications, and buried aspects of the image that do not, at first, meet the eye.

My efforts are not to be confused with the noble goal of many archeologists and a good number of "Sound History" historians to reconstruct older soundscapes and acoustical settings.6 This work is worthwhile, and not entirely unconnected from my interest in image sonification. However, my curiosity is far more about semiotic readings of artifacts such as photographs that might be inspired by listening to sonifications of their data. How might we notice significant details and qualities of image artifacts by hearing as well as seeing their form? In this way, my work has turned out as much to be about visual studies as sound studies since it concentrates especially on photography, maps, and poster art from the Berkeley collection. Here I have been inspired by Roland Barthes and Susan Sontag's groundbreaking criticism on the strange qualities of photographs as visual representations that at once capture a moment in time and present a fictional account of the past, as well as more recent scholarship by Margaret Olin on "touching photographs," Tina Campt's concept of "listening to images," and W.T.J Mitchell's arguments about the power of images to call forth responses rather than passively receive the spectator's gaze.7 I have wanted to know if digital approaches can complement these theoretical inquiries into the nature of photographs and images. What do digital humanities tactics have to offer what Mitchell calls "picture theory"?8

My work also takes a philosophical turn by asking: what is a digitized artifact, anyway? If one is always representing underlying data of a digital artifact, is it still the same artifact? How can data be wielded as evidentiary material as it shifts from a photograph to a "sonograph" based on the same exact data? I find the work of Fred Gibbs and Trevor Owens, who wrote one of the best digital history inquiries into "the hermeneutics of data," inspiring on this count.9 So too, Jonathan Sterne and Veit Erlmann's revelatory histories and theorizations of sound alerted me to the how the human sensorium was an ambiguous historical entity, not a timeless set of separate and discrete modes of perception.10 And the theorization of how social relations are embedded in the aesthetic dimensions of listening has been taken up in much recent sound studies scholarship.11 Further back in time, I think of Steven Feld's work as a particularly important entry point into thinking about listening as a contextualized social experience rather than some kind of timeless sensorial act.12

In response to this range of inquiry into sound, listening, aesthetic experience, social relations, the mediations and materiality of our newly digitized sources, and how we access the past conceptually, my focus has increasingly become methodological, driven by the possibilities for new perceptions raised by oscillating between sight and sound, the visual and the auditory, seeing and hearing by way of computational tactics that transpose data from one form to another and between one human sense and another. What if we began to think about digital image sonification as a kind of synesthetic approach to historical sources? What if it was not about a fight between sight and sound as separate modes of perception, but rather the use of image sonification to notice new things with our eyes by hearing them with our ears? How might we look at evidentiary source materials by listening to them via computational shifts of their phenomenological states?

This has led to considering a whole set of tactics for representing image data as sound. There is no one objective way to make this transliteration. There are only sets of intriguing correlations to explore, both in terms of the features of the image data one wants to use and the qualities of the sound one wants to produce. One can create many iterations, many representations, if one thinks as an artist might about how to wield digital computation in service of humanistic and historical discovery.

Image sonification also points to some interesting, if rather strange and philosophical, methodological questions about history itself. We might say that history itself is acousmatic, in the sense that we can never precisely return to the origin point of a historical activity once it has passed. Instead, we are always listening (and looking) back to it through its artifactual representations, which are resonances of the original. As Jean-Luc Nancy argues, "To listen is tendre l'oreille. literally, to stretch the ear."13 Image sonification asks us to extend our senses, to heighten our awareness about an artifact, an archive, and history itself in terms of how we access it.

In digital image sonification, formal data alteration is put in service of more vivid illumination. A shift in the phenomenology of historical analysis can trigger a deeper epistemology. Synesthetically, hearing what we should be seeing through image sonification raises our awareness that access to the past is always mediated by sources that contain but the traces, the residues, the resonances of a moment gone by. Fusing the senses, and confusing them a bit, allows us to ascertain aspects of information, data, feeling, knowledge captured in our sources, but lurking below their surfaces – or so obvious on the surfaces that we missed their significance. Just as visualization lets us see text or data from another angle, so sonification helps us hear vision from another vantage point, in its transliterated echoes.

J.S.N.: If we consider the paper by Irwin Pollack and Lawrence Ficks in The Journal of the Acoustical Society of America14 (1954) to be a pertinent landmark, sonification, as a specific mode of auditory display, has been a proper field of research for at least half a century. For twenty five years, The International Community for Auditory Display has organized annual international conferences and, according to the online archive of the proceedings15, the proportion of papers written by historians is almost negligible, contrary to those written by art historians or musicologists, for instance. The research of cultural historian Axel Volmar is among these few historians who has developed in a very original and stimulating way. Indeed, with Andi Schoon, he edited a collective book laying the groundwork of a Cultural History of sonification in 2012.16 He also published a paper, « Listening to the Cold War: The Nuclear Test Ban Negotiations, Seismology, and Psychoacoustics, 1958– 196317 », analyzing the research of psychoacoustician Sheridan Speeth in the Cold War context. How do you consider their approaches in relation to your own work?

M.J.K.: These approaches are important considerations of how to expand our understanding of auditory perception, particularly in the sciences. Volmar contends that Sheridan Speeth's work was dismissed by other Cold War scientists more for political reasons than their undervaluing of sound compared to vison. I wonder if the political and the methodological can be separated so easily. Perhaps my research will go the way of Speeth's! In the meantime, my interest in image sonification for historical inquiry turns more fundamentally toward a hermeneutical approach rather than a pursuit of some objective sonification or arguing that sonification is better than visualization when it comes to the analysis that constitutes cultural history.

I propose instead that bringing related image and sound artifacts into dialogue with each other is the most productive method to adopt. I also have come to believe that embracing rather than fearing subjective decisions about both the formal qualities of images and their transformations into sound is beneficial. Rather than seek out the one true sonification, which does not exist, I prefer the spark of historical perception, even revelation, that can arise from the artistic manipulation of artifacts into sound in service of noticing new aspects about them.

J.S.N.: You have written that « Digital sound design does not bring us magically back to the past itself, for we can never make that journey18 ». From historian of sound Emily Thompson and Scott Mahoy’s Roaring Twenties (An Interactive Exploration of the Historical Soundscape of New York City19) to archeomusicologist Mylène Pardoen’s Bretez project (restitution of Paris in the 18th century), many stimulating projects have tried to recreate – more than model – soundscapes of the past. Among other competences, these interdisciplinary approaches combined historical interpretation of archival material, musicology and sound design. Thompson’s project offers a mapping of sound or film archive that enhances our understanding of social and cultural life in the different districts of Greater New York during the 1920s. Pardoen’s project explores the “sensory and the sensible20” and is based on an 18th century plan of Paris by French engineer Louis Bretez. What do you think about these uses of sonification for historical interpretation? You have also conceived ways to sonify maps21. Could you describe your own methodology?

M.J.K.: As historians, we dream of reconstituting the past, of time travel to it through our artifacts. But this really is just a dream. This is in fact a point Emily Thompson makes in the introduction to her digital project on noise in 1920s New York City.22 What we can do instead is more consciously mediate our backwards gaze at the past by imaginatively exploring evocations of the past and interpreting their significance. Isn't this what history always is?

What is worrisome about digital representations of soundscapes of the past is their potential "screen essentialism," the ways in which they collapse a representation of the past into a factual actuality of a past world's sounds.23 For this reason, while I admire the effort to replicate past soundscapes digitally, I am a bit wary of these endeavors becoming substitutes rather than approximations, the thing itself rather than a model. As in the map described in the famous Borges tale, "On Exactitude in Science," so too when the digital map becomes the territory, our history is rendered not more, but less true. Imagining that digital recreations are "the past as it was" dissolves the tension between our evidence and the past to which the evidence can gesture but for which it can never permanently substitute.24

Listening to as well as look at images, by contrast, seems to take us one step away from the past, but in fact brings us into the gap between the map and the territory. That is where, I believe, the truths of history reside. We are listening to data from pictures of sound being made in the past, not the sounds themselves in my work on the Berkeley Folk Music Festival. This alerts us to the artifact's palimpsest while also allowing us to activate the residues of earlier impressions.

My urge has been to intensify the awareness of artefactual representation rather than obfuscate it. How do we evoke the past through "messing with" the artifact data even more, rather than feigning objective models? How do we harness the perceptual power of data "deformances," as Lisa Samuels and Jerome McGann famously described in their classic digital humanities essay? How do we defamiliarize historical evidence in order to analyze it more accurately, more rigorously, more truthfully?25

To be clear, I do not mean we should become historical Stalins airbrushing figures out of photographs. It is not a case of anything goes. There are data, and we can treat these data as historical evidence. But computers are perhaps most of all re-representation machines, so how can we make use of that quality more rather than pretend to be presenting the past as it was? Again, there are certain aspects of the past that we can accurately portray using evidence—and we should do that—but so much of the past consists of complexities and ambiguous forces. Rather than use computation to flatten those complexities into statistical models or mediated simplifications, how can we better access the complexities themselves, deepen our understanding of them as ambiguous and complex, intricate and in motion, real and true as objective dimensions of the past?

To hear the quietness of a darkened theater from which thousands of audience members gaze upon Joan Baez intently, or to tune in aurally to the silhouette of a performer such as Mance Lipscomb as he and his middle-class, white audience at Berkeley navigate the cultural and social distance between them—these become ways not of reducing our historical artifacts to fodder for statistical analysis, but rather of expanding their richness for interpretive inquiry.

Heuristic and theorical frameworks

J.S.N.: From a methodological point of view, data sonification needs both clear protocol and grammar. In this perspective, specialist of Cognitive Interaction Technology Thomas Hermann has proposed a taxonomy and a set of definitions that imply four fundamental criteria : “(C1) The sound reflects objective properties or relations in the input data. (C2) The transformation is systematic. This means that there is a precise definition provided of how the data (and optional interactions) cause the sound to change. (C3) The sonification is reproducible: given the same data and identical interactions (or triggers) the resulting sound has to be structurally identical. (C4) The system can intentionally be used with different data, and also be used in repetition with the same data26.”

Do you consider this methodological framework to be relevant for historians and does it fit your own epistemological position?

M.J.K.: No, I propose a more adventurous mode of data sonification in which artistic tactics of collage, fusion, Cagean "chance operations," and formal experimentation might allow historians to hear things in their artifacts, evidence, and data that they might not otherwise perceive. Why not embrace the full potential of computers as re-representation machines? Why constrain our capacities to perceive previously unnoticed aspects of the archival record through creative digital manipulations of our materials? To be sure, it is worthwhile to present the details of how one chose to sonify data, but these more adventurous, subjective, artistic approaches seem as valid to me as the push toward constraints and restrictions in computer science-inspired methods. We can position uses of computation not toward the aggrandizement of statistical modes of truth alone, but rather harness computational power to "read against the grain" of an archive, of an artifact itself. This allows us to realize Gibbs and Owens' call for a fully developed "hermeneutics of data" in which critical play expands the methodological repertoire of historical inquiry. In short, a wider range of data sonification approaches might well "unlock" the past from source materials more effectively.

Additionally, the complicity between seeking to transform the world into data and the dehumanization of lives, particular black and women's lives, must be attended to in how we wish to create rules for how to handle historical evidence as data. Jessica Marie Johnson, Safiya Umoja Noble, Zeynep Tufekci, Jacqueline Wernimont, Catherine D’Ignazio, Lauren Klein, and many other scholars ask us to think more critically about what happens when people are treated as numbers, when lives are turned into data.27 Their critical scholarship calls for a rethinking of the very terms of "objectivity" and "subjectivity" in historical study. While statistics can greatly aid in revealing significant and powerful patterns and truths (one thinks of epidemiological models for handling virus pandemics these days, or of statistical studies that revealed just how dramatically prisons were disproportionately incarcerating people of color), so too quantification has been a force of exploitation. It's much easier to enslave a datapoint than a person. Data are, as Lisa Gitelman evocatively put it, "always already new."28 They are not reality but a closely connected abstraction of reality, often one with vexing implications for questions of power and knowledge. In my research, I seek to establish more aesthetically aware modes of data sonification. This will, I hope, provide a more critical stance for registering the tensions between data and reality.

This approach does, I admit, become a bit strange for many historians. It suggests that sometimes the aesthetically subjective "glitch," the randomized altering of computational data, might in fact provide more promising paths to describing and interpreting "objective" truths about the past than traditional modes of analysis. When it comes to methods for processing computational data, however, I would argue that a robust objectivity must include a thoughtful and imaginative subjectivity. Sonification allows us to enter into a photograph rather than treat it as a flatly empirical overview of a past moment. Sound, as scholars from Edmund Carpenter, Marshall McLuhan, and Walter Ong long ago pointed out, is immersive and relational. It brings us through the constitutive parts of a photograph, reorienting us to their assembly of shapes, portrayals, and qualities.29

J.S.N.: The work of Lev Manovich seems to have inspired you, especially when you describe the method of “data fusion”. In his famous essay entitled The Language of New Media30, Manovich has defined a “new culture of computer”, distinguishing the computer layer from the cultural layer in order to analyze the interaction between them. He also considers the visual culture at the computer age to be profoundly indebted to the culture of film-making and cinematic modes of conceiving, representing and structuring time. Could you explain what aspects of Manovich’s work have had an influence on you?

M.J.K.: Manovich's data fusion concept urges us to consider how the bringing together of large datasets requires new representational forms, how in a sense it produces a new, distinctive artifact from data. In this sense, Manovich is interested in questions of scale: what kind of artifact is it exactly that is based on millions of datapoints rendered into a new representation? It's no longer merely the sum total of its aggregated parts, but a transformed object.

So too, Manovich has been interested in contrasting the modularity of the database that creates and undergirds this transformed object to earlier cinematic modes of presentation, which rely on the linearity of film. He wants us to notice that a new material and representational regime is upon us in the computational age. This is partly what I think he means by emphasizing a "new culture" of computing. Matthew Kirschenbaum's work on how software actually functions as an execution of steps rather than the creation of a fixed work or text raises similar questions.31 What exactly is this thing we call the digital object? What, materially, is the "virtual" artifact? And as that material level changes, what happens to cultural levels of representation?

Although neither Manovich nor Kirschenbaum explore image sonification much, their work leads logically toward the kind of revelations that image sonification might produce. For instance, we cannot really see patterns in one million tweets, or even by looking at 10,000 images of the Berkeley Folk Music Festival, but sonic representation is quite good at presenting patterns in these fused materials. One might hear temporal qualities especially: historical clues about repetition, acceleration, and more through listening to sonic representations of artifacts at scale.

J.S.N.: In his essay on Audio-vision, French composer and academic Michel Chion has also greatly contributed to exploring the dialectic relationship between these two senses. He regards hearing and vision as captors and the human brain as the seat of sensory encounters. He highlights the specificity of an acoustic analysis of data when he write “the ear analyzes, works and synthesizes faster than the eye32”. However, hearing cannot obviously be conceived as an autonomous and anhistoric phenomenon. One of the major questions raised in the field of sound studies relates to the place given to psychoacoustics. Thus, studies about Auditory Culture combine both technical and contextual approaches to the perception of sounds and are therefore indebted to science and technology academic research as well as to Alain Corbin’s sensory history. This conception which ultimately articulates hearing, perception and listening allows historians to understand their sources in all their internal, external, contextual and relational complexity. Where does your own work fall on the spectrum of sound studies ?

M.J.K.: My sense is that Chion's focus has especially been on expanding Pierre Schaeffer's ideas about the acousmatic, which is to say sound that has been perceived without sight of its origin in view.33 This idea of sound without vision is fascinating, but my work has tilted more toward a synesthetic approach: sight and sound, a photograph and its sonification, brought into interplay for the historian to perceive new historical information, meaning, details, and interpretation from the mix of hearing sights and, if one wishes, seeing sounds.

For instance, I have an essay coming out in a special issue of Digital Humanities Quarterly focused on AudioVisual DH that explores issues of performance, embodiment, spectacle, hierarchy, and democracy within the 1960s American folk revival through the digital image sonification of one image of Joan Baez performing at the William Randolph Hearst Greek Amphitheater on the campus of the University of California in Berkeley. This builds on my work on data sonification of another striking image of the Texas African-American blues singer and guitarist Mance Lipscomb performing on the same stage at the Berkeley Folk Music Festival. In that image, sonification led me to sharper, more probing ideas about negotiations of race within the folk revival milieu. In my new article, issues of gender move to the fore.

J.S.N.: Musicologist Jonathan Sterne, a key figure of Sound Studies, analyses the MP3 format as a “cultural artifact” and as a technology needing to be assembled by other technologies. For him, each step of the encoding process has to be documented in addition to the “psychoacoustic model encoded within it34”. In other words, the use of sonification techniques by historians requires precise knowledge of what hearing means, both from psychoacoustic and a cultural or symbolic perspective. In your conception and experience of the “historiographical operation” as defined by Michel de Certeau, how can the transformation of clues into data be documented?

M.J.K.: This is where the analyses of media archeology and media studies in general have much to offer historians. I am thinking particularly of the work of Wolfgang Ernst, but also Kittler, Sterne, Rey Chow, Kevin L. Ferguson, and myriad other media studies scholars who ask us to pay close attention to the qualities of artifacts as forms of media.35 In my own research, for instance, data sonification led me back into the very strange qualities of the photographs themselves.

As we know from the famous scholarship of Barthes, Sontag, and others, photography is weird. It is really weird! The camera captures a moment in time, the very light emulsified into chemical imprints on a film or printed on to paper, so in this sense it is an objective document. Except it isn't. The photograph is also profoundly representational. How do we, how might we harness for historical interpretation this ambiguous quality of the photograph, which is at once an "objective" view on the past and a subjective "capture" of that moment now gone?

Phonography, the transcription of sound to disc, tape, or other medium, perhaps parallels the photograph in this respect, but for my purposes I'm most curious about how we might borrow from the rigorous attention to the material qualities of the media we use as historical artifacts to draw larger, evidence-based conclusions about the cultural and political past. For instance, although we cannot hear the music that was being played in a photograph of the Berkeley Folk Music Festival, image sonification forces us to hear performance spaces temporally. Photographs flatten performance into silent spaces of optic perception. Sonification adds "depth." It enlivens the ear to these spatial representations from the past when performance was happening. Hearing a two-dimensional photographic space as a three-dimensional sonified temporal story opens up the image for more probing analysis. It activates the data of the photograph in a new, defamiliarized manner so that we might notice new qualities, relationships, dynamics in the space of the pictured, but now inert, performance.

Tools, software and the historian’s experience of sonification

J.S.N.: In your own work, you refer to software such as the “Photosounder36” developed by Michel Rouzic. What are the possibilities offered by this technology and to what end have you used it?

M.J.K.: Michel Rouzic's Photosounder offers the beginnings of image sonification experimentation, as do a number of other tools. It allows one to manipulate some aspects of visual data and it enables a few sonic output variables. One can read images left to right, and play with the volume and a few other parameters of the pink noise that Photosounder generates from the image. But that is about all Photosounder can do. So it offers a starting point, but I think far more exploration of digital tools is still necessary. Can we extract features from images and then sonify them? Can we read images in multiple ways? Can we create a wider palette of sounds to produce from image data?

J.S.N.: On the scientific blog “Programming Historian”, archeologist and digital humanist Shawn Graham published a “Gentle Introduction to Sonification for Historians37”. He offers tutorials on applications that require some knowledge of the Python programming language or of more easily accessible software like MUSICalgorithms. For instance, he details the sonification process of a sample from a database hosted on the Portable Antiquity Scheme website of the British Museum. The example is centered on Roman coins found all along one of the main routes of Roman Britain called Watling street. The dataset is organized geographically and Graham effects various transformations on each data point (duration, tuning, timbre). This protocol allows him to represent the spatial evolution of economic phenomena through acoustic progression. Does this experiment seem relevant to you? For the result to be audible, do you think it is necessary to work on the aesthetic dimension of the sonification criteria?

M.J.K.: I greatly admire Shawn Graham's sonification explorations. Along with a number of other projects by programmers and musicians such as Jacque Jones, Brian Foo, Maximillian Laumeister, and Margaret Schedel, Graham's sonifications help us hear large amounts of data developing patterns over time.38 In my own work on image sonification, I am curious about what it means to adapt these sonifications of numerical data to visual data. What similar tactics are effective for producing new representations that can spark fresh and compelling perceptions of the evidentiary record, thus undergirding new interpretations? Are there also different computational approaches required for the shift from numerical data to visual data?

As you suggest, perhaps there is a greater aesthetic sensitivity required for visual data than numbers. For we are not only dealing with something such as numerical frequency and the tempos and intensities it can produce, but also more subtle aesthetic aspects of source materials: shades of light, aspects of lines and form, the representation of humans as subjects before the camera's gaze or the artist's rendering. These can be computationally quantified, indeed they must be turned into binary code for a computer to process them. But to make for the fullest historical analysis possible, they might also—they must also—be interpretively qualified. In my view, and to my ears, it is in the artful interplay of computer operations and the human sensorium, in using aesthetic approaches to the computational transit between image and sound, seeing and hearing, that we might discover the past most robustly.

1 Michael J. Kramer, « “A Foreign Sound To Your Ear”: Digital Image Sonification for Historical Interpretation », in Mary Caton Lingold, Darren

2 Jonathan Sterne, “The mp3 as Cultural Artifact”, in New Media & Society, vol. 8(5), 2006, p. 833.

3 Alain Corbin, Village Bells: Sound and Meaning in the Nineteenth- Century French Countryside. Translated by Martin Thom, New York, Columbia

4 Berkeley Folk Music Festival Project. [Online] URL: www.bfmf.net (accessed May 5, 2020).

5 Franco Moretti, Graphs, maps, trees: abstract models for literary history, London, New York, Verso, 2007.

6 For instance, see the essays in Mark M. Smith, Hearing History: A Reader, Athens: University of Georgia Press, 2004.

7 Roland Barthes, Camera Lucida: Reflections on Photography, trans. Richard Howard, New York, Farrar, Straus and Giroux, 1981. Susan Sontag, On

8 William John Thomas Mitchell, Picture Theory: Essays on Verbal and Visual Representation, Chicago, University of Chicago Press, 1994.

9 Fred Gibbs and Trevor Owens, "The Hermeneutics of Data & Historical Writing," in Kristen Nawrotzki and Jack Dougherty, eds., Writing History in the

10 Jonathan Sterne, The Audible Past: Cultural Origins of Sound Reproduction, Durham, NC, Duke University Press, 2003.

11 Jennifer Lynn Stoever, The Sonic Color Line: Race and the Cultural Politics of Listening, New York, NYU Press, 2016. Josh Kun, Audiotopia: Music

12 Steven Feld. Sound and Sentiment: Birds, Weeping, Poetics, and Song in Kaluli Expression, 1982, reprint, Durham, NC, Duke University Press, 2012.

13 Jean-Luc Nancy, Listening (2002; English translation, New York, Fordham University Press, 2007, p. 5.

14 Irwin Pollack and Lawrence Ficks, “The Information of Elementary Multidimensional Auditory Displays” in The Journal of the Acoustical Society of

15 Proceedings of annual International Conference on Auditory Display, Georgia Tech Library [Online] URL: https://smartech.gatech.edu/handle/1853/

16 Andi Schoon and Axel Volmar, eds., Das geschulte Ohr: eine Kulturgeschichte der Sonifikation, Bielefeld, Transcript, coll.« Sound studies », 2012.

17 Axel Volmar, « Listening to the Cold War: The Nuclear Test Ban Negotiations, Seismology, and Psychoacoustics, 1958–1963 », Osiris, vol. 28, no. 1

18 Michael J. Kramer, “A Foreign Sound To Your Ear”, op.cit., p.190.

19 See the project’s website, [Online] URL: http://vectorsdev.usc.edu/NYCsound/777b.html (accessed September 12, 2020).

20 See the project’s website, [Online] URL: https://sites.google.com/site/louisbretez/home/ (accessed May 5, 2020).

21 Michael J. Kramer, “Humbead’s Map Folk Music Data Viz”, Berkeley Folk Festival Project Blog, [Online] URL: https://www.michaeljkramer.net/humbeads

22 Emily Thompson, "Author's Statement," Emily Thompson and Scott Mahoy, "The Roaring 'Twenties: an interactive exploration of the historical

23 On "screen essentialism," see Nick Montfort, "Continuous Paper: The Early Materiality and Workings of Electronic Literature," Presented at Modern

24 Jorge Luis Borges. "On Exactitude in Science," 1946, in Collected Fictions, trans. Andrew Hurley, New York: Viking Press, 1998, p. 325.

25 Jerome McGann and Lisa Samuels, "Deformance and Interpretation," New Literary History 30 (Winter 1999), p. 25–56.

26 Thomas Hermann, « Taxonomy and definitions for signification and Auditory Display », Proceedings of the 14th International Conference on Auditory

27 Jessica Marie Johnson, “Markup Bodies: Black [Life] Studies and Slavery [Death] Studies at the Digital Crossroads,” Social Text 36: 4 (December

28 Lisa Gitelman, Always Already New: Media, History, and the Data of Culture, Cambridge, MA, MIT Press, 2006.

29 Edmund Carpenter and Marshall McLuhan, "Acoustic Space," in Explorations in Communications, eds. Marshall McLuhan and Edmund Carpenter, Boston, MA

30 Lev Manovich, The Language of New Media. Cambridge, Massachusetts, London, England, The MIT Press, 2001.

31 Matthew G. Kirschenbaum, "An Executable Past: The Case for a National Software Registry,"in Preserving.exe: Toward a National Strategy for

32 Michel Chion, Audio-Vision. Sound on Screen. Translated and edited by Claudia Gorman, New York, Columbia University Press, 1994.

33 Pierre Schaeffer, Traité des objets musicaux, Paris, Le Seuil, 1966.

34 Jonathan Sterne, “The mp3 as Cultural Artifact”, in New Media & Society, vol. 8(5), 2006, p. 833.

35 Wolfgang Ernst, Sonic Time Machines: Explicit Sound, Sirenic Voices, and Implicit Sonicity, 2016. Wolfgang Ernst, Digital Memory and the Archive

36 [Online] URL: https://photosounder.com/ (accessed September 12, 2020).

37 Shawn Graham, “The Sound of Data (a gentle introduction to sonification for historians)”, The Programming Historian [Online] URL : https://

38 Justin Joque, "Listening to the Dow," http://vimeo.com/23965023. Maximillian Laumeister, "Realtime Bitcoin Transaction Visualizer (formerly Listen

Notes

1 Michael J. Kramer, « “A Foreign Sound To Your Ear”: Digital Image Sonification for Historical Interpretation », in Mary Caton Lingold, Darren Mueller, Whitney Trettien, ed., Digital Sound Studies, Durham and London, Duke University Press, 2018, p.178-214.

2 Jonathan Sterne, “The mp3 as Cultural Artifact”, in New Media & Society, vol. 8(5), 2006, p. 833.

3 Alain Corbin, Village Bells: Sound and Meaning in the Nineteenth- Century French Countryside. Translated by Martin Thom, New York, Columbia University Press, 1998.

4 Berkeley Folk Music Festival Project. [Online] URL: www.bfmf.net (accessed May 5, 2020).

5 Franco Moretti, Graphs, maps, trees: abstract models for literary history, London, New York, Verso, 2007.

6 For instance, see the essays in Mark M. Smith, Hearing History: A Reader, Athens: University of Georgia Press, 2004.

7 Roland Barthes, Camera Lucida: Reflections on Photography, trans. Richard Howard, New York, Farrar, Straus and Giroux, 1981. Susan Sontag, On Photography, New York, Farrar, Straus and Giroux, 1977. Margaret Olin, Touching Photographs, Chicago, University of Chicago Press, 2012. Tina M. Campt, Listening to Images, Durham, NC, Duke University Press Books, 2017. William John Thomas Mitchell, What Do Pictures Want? The Lives and Loves of Images, Chicago, University of Chicago Press, 2005.

8 William John Thomas Mitchell, Picture Theory: Essays on Verbal and Visual Representation, Chicago, University of Chicago Press, 1994.

9 Fred Gibbs and Trevor Owens, "The Hermeneutics of Data & Historical Writing," in Kristen Nawrotzki and Jack Dougherty, eds., Writing History in the Digital Age, University of Michigan Press, 2013, pp.159-170.

10 Jonathan Sterne, The Audible Past: Cultural Origins of Sound Reproduction, Durham, NC, Duke University Press, 2003.

11 Jennifer Lynn Stoever, The Sonic Color Line: Race and the Cultural Politics of Listening, New York, NYU Press, 2016. Josh Kun, Audiotopia: Music, Race, and America, Berkeley, University of California Press, 2005. Kara Keeling and Josh Kun, eds., Sound Clash: Listening to American Studies, Baltimore: Johns Hopkins University Press, 2012. Nina Sun Eidsheim, Sensing Sound: Singing & Listening as Vibrational Practice, Durham, NC, Duke University Press, 2015. Nina Sun Eidsheim, The Race of Sound: Listening, Timbre, and Vocality in African American Music, Durham, NC, Duke University Press Books, 2019. Ana María Ochoa Gautier, Aurality: Listening and Knowledge in Nineteenth-Century Colombia, Durham, NC, Duke University Press, 2014. Louise Meintjes, Sound of Africa!: Making Music Zulu in a South African Studio, Durham, NC, Duke University Press, 2003. Louise Meintjes with photographs by T.J. Lemon, Dust of the Zulu: Ngoma Aesthetics after Apartheid. Durham, NC, Duke University Press, 2017. Ashon Crawley, Blackpentecostal Breath: The Aesthetics of Possibility, New York, Fordham University Press, 2016. J. Martin Daughtry, Listening to War: Sound, Music, Trauma, and Survival in Wartime Iraq, New York, Oxford University Press, 2015. Brandon Labelle, Sonic Agency: Sound and Emergent Forms of Resistance, Cambridge, MA, Goldsmiths Press, 2018. Robin James, The Sonic Episteme: Acoustic Resonance, Neoliberalism, and Biopolitics, Durham, NC, Duke University Press, 2019. Douglas Kahn, Noise, Water, Meat: A History of Sound in the Arts, Cambridge, MA, MIT Press, 2001. See also, Michael Bull and Les Back, eds., The Auditory Culture Reader, Oxford: Bloomsbury Academic, 2004 ; Jonathan Sterne, ed., The Sound Studies Reader, New York: Routledge, 2012 ; Deborah Kapchan, Theorizing Sound Writing, Middletown, CT: Wesleyan University Press, 2017; Gavin Steingo and Jim Sykes, eds., Remapping Sound Studies, Durham, Duke University Press, 2019.

12 Steven Feld. Sound and Sentiment: Birds, Weeping, Poetics, and Song in Kaluli Expression, 1982, reprint, Durham, NC, Duke University Press, 2012. Charles Keil and Steven Feld, Music Grooves: Essays and Dialogues, Chicago: University of Chicago Press, 1995.

13 Jean-Luc Nancy, Listening (2002; English translation, New York, Fordham University Press, 2007, p. 5.

14 Irwin Pollack and Lawrence Ficks, “The Information of Elementary Multidimensional Auditory Displays” in The Journal of the Acoustical Society of America, 26(1), January 1954, p.136-136. 

15 Proceedings of annual International Conference on Auditory Display, Georgia Tech Library [Online] URL: https://smartech.gatech.edu/handle/1853/49750 (accessed May 5, 2020).

16 Andi Schoon and Axel Volmar, eds., Das geschulte Ohr: eine Kulturgeschichte der Sonifikation, Bielefeld, Transcript, coll.« Sound studies », 2012.

17 Axel Volmar, « Listening to the Cold War: The Nuclear Test Ban Negotiations, Seismology, and Psychoacoustics, 1958–1963 », Osiris, vol. 28, no. 1, « Music, Sound, and the Laboratory from 1750–1980 » (January 2013), p. 80-102.

18 Michael J. Kramer, “A Foreign Sound To Your Ear”, op.cit., p.190.

19 See the project’s website, [Online] URL: http://vectorsdev.usc.edu/NYCsound/777b.html (accessed September 12, 2020).

20 See the project’s website, [Online] URL: https://sites.google.com/site/louisbretez/home/ (accessed May 5, 2020).

21 Michael J. Kramer, “Humbead’s Map Folk Music Data Viz”, Berkeley Folk Festival Project Blog, [Online] URL: https://www.michaeljkramer.net/humbeads-map-folk-music-data-viz/ (accessed September 12, 2020).

22 Emily Thompson, "Author's Statement," Emily Thompson and Scott Mahoy, "The Roaring 'Twenties: an interactive exploration of the historical soundscape of New York City," Vectors, 22 September 2013, http://vectors.usc.edu/projects/index.php?project=98&thread=AuthorsStatement (accessed May 5, 2020).

23 On "screen essentialism," see Nick Montfort, "Continuous Paper: The Early Materiality and Workings of Electronic Literature," Presented at Modern Language Association Convention, sponsored by Modern Language Association (December 28 2004), [Online] URL: http://nickm.com/writing/essays/continuous_paper_mla.html (accessed May 5, 2020); and Matthew Kirschenbaum, Mechanisms: New Media and the Forensic Imagination, Cambridge, MA: MIT Press, 2008.

24 Jorge Luis Borges. "On Exactitude in Science," 1946, in Collected Fictions, trans. Andrew Hurley, New York: Viking Press, 1998, p. 325.

25 Jerome McGann and Lisa Samuels, "Deformance and Interpretation," New Literary History 30 (Winter 1999), p. 25–56.

26 Thomas Hermann, « Taxonomy and definitions for signification and Auditory Display », Proceedings of the 14th International Conference on Auditory Display, Paris, France June 24-27, 2008. [Online] URL: https://www.icad.org/Proceedings/2008/Hermann2008.pdf (accessed September 12, 2020).

27 Jessica Marie Johnson, “Markup Bodies: Black [Life] Studies and Slavery [Death] Studies at the Digital Crossroads,” Social Text 36: 4 (December 2018), pp. 57–79; Safiya Umoja Noble, Algorithms of Oppression: How Search Engines Reinforce Racism, New York, NYU Press, 2018. Zeynep Tufekci, Twitter and Tear Gas: The Power and Fragility of Networked Protest, New Haven, CT, Yale University Press, 2017. Jacqueline Wernimont, Numbered Lives: Life and Death in Quantum Media, Cambridge, MA: MIT Press, 2019. Elizabeth Losh and Jacqueline Wernimont, eds., Bodies of Information: Intersectional Feminism and Digital Humanities, Minneapolis, University of Minnesota Press, 2018. Catherine D’Ignazio and Lauren F. Klein, Data Feminism, Cambridge, MA, MIT Press, 2020.

28 Lisa Gitelman, Always Already New: Media, History, and the Data of Culture, Cambridge, MA, MIT Press, 2006.

29 Edmund Carpenter and Marshall McLuhan, "Acoustic Space," in Explorations in Communications, eds. Marshall McLuhan and Edmund Carpenter, Boston, MA: Beacon Press, 1960, 64-69; Walter Ong, Orality and Literacy, New York: Routledge, 1982, p. 70.

30 Lev Manovich, The Language of New Media. Cambridge, Massachusetts, London, England, The MIT Press, 2001.

31 Matthew G. Kirschenbaum, "An Executable Past: The Case for a National Software Registry," in Preserving.exe: Toward a National Strategy for Preserving Software, Library of Congress (2013).

32 Michel Chion, Audio-Vision. Sound on Screen. Translated and edited by Claudia Gorman, New York, Columbia University Press, 1994.

33 Pierre Schaeffer, Traité des objets musicaux, Paris, Le Seuil, 1966.

34 Jonathan Sterne, “The mp3 as Cultural Artifact”, in New Media & Society, vol. 8(5), 2006, p. 833.

35 Wolfgang Ernst, Sonic Time Machines: Explicit Sound, Sirenic Voices, and Implicit Sonicity, 2016. Wolfgang Ernst, Digital Memory and the Archive, edited by Jussi Parikka, Minneapolis: University of Minnesota Press, 2012. Friedrich A. Kittler, Gramophone, Film, Typewriter, translated by Geoffrey Winthrop-Young and Michael Wutz, Stanford, CA: Stanford University Press, 1999. Rey Chow, Entanglements: Or Transmedial Thinking about Capture, Durham, NC, Duke University Press, 2012. Kevin L. Ferguson, "Volumetric Cinema," in Matthew Gold and Lauren Klein, eds., Debates in the Digital Humanities 2019, Minneapolis, University of Minnesota, 2019, p. 335-249.

36 [Online] URL: https://photosounder.com/ (accessed September 12, 2020).

37 Shawn Graham, “The Sound of Data (a gentle introduction to sonification for historians)”, The Programming Historian [Online] URL : https://programminghistorian.org/en/lessons/sonification (accessed September 12, 2020).

38 Justin Joque, "Listening to the Dow," http://vimeo.com/23965023. Maximillian Laumeister, "Realtime Bitcoin Transaction Visualizer (formerly Listen to Bitcoin)," [Online] URL: (accessed May 5, 2020). Brian Foo, "Two Trains: Sonification of Income Inequality on the NYC Subway," [Online] URL: https://datadrivendj.com/tracks/subway/ (accessed May 5, 2020). Margaret Schedel, "Sounds of Science: The Mystique of Sonification," Sounding Out!, 9 October 2014. [Online] URL: https://soundstudiesblog.com/2014/10/09/sounds-of-science-the-mystique-of-sonification/ (accessed May 5, 2020).

Illustrations

Photographer unknown (possibly Philip Olivier). Courtesy of Berkeley. Processing and analysis of the picture with Photosounder 1.10.

References

Electronic reference

Michael J. Kramer and Jean-Sébastien Noël, « “Remediatizing Visual Media as Sonic Data. Sonification and Cultural History.” », Revue d’histoire culturelle [Online],  | 2020, Online since 20 septembre 2020, connection on 25 novembre 2020. URL : http://revues.mshparisnord.fr/rhc/index.php?id=348

Authors

Michael J. Kramer

Assistant professor in the Department of History at SUNY Brockport, Michael J. Kramer has specialized in modern United States cultural and intellectual history, transnational history, and public and digital history, he published in 2013 The Republic of Rock: Music and Citizenship in the Sixties Counterculture and is currently writing a book about technology and tradition in the US folk music movement, This Machine Kills Fascists: What the Folk Music Revival Can Teach Us About the Digital Age.

Jean-Sébastien Noël

Associate professor in contemporary history in La Rochelle University (France), Jean-Sébastien Noël works on a cultural history of music in the Atlantic space (late 19th century/20th century). His original field of research focused on the impact of musical repertoires, sound archives and practices on memorial processes: Le silence s'essouffle. Mort, deuil et mémoire chez les compositeurs ashkénazes. Europe centrale et orientale, États-Unis (2016). He develops his current research on the networks of avant-gardes composers.

By this author