interaction – Federico Visi

Videos from the Embedding Algorithms Workshop

_FV — Sat, 12 Apr 2025 09:51:17 +0000

I have made a YouTube playlist with videos from the Embedding Algorithms Workshop.

The event took place in July 2024 at Universität der Künste Berlin, we brought together a community of artists, researchers, and practitioners and had a great time discussing the practice of embedding algorithms in the arts with presentations, exhibitions, and performances. Enjoy!

Full playlist:

Computer Music Journal article: “An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound”

_FV — Fri, 13 Dec 2024 19:52:22 +0000

I finally put my hands on the physical copy of the Computer Music Journal issue on Human-AI Cocreativity that includes our open-access article An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound.

There we report on our work on an instrument that translates the electrical activity produced by muscles into synthesised sound. The article also describes our collaboration with the Chicks on Speed on the performance piece Noise Bodies (2019), which was part of the Up to and Including Limits: After Carolee Schneemann exhibition at Muzeum Susch.

I co-authored the article with Atau Tanaka, Balandino Di Donato, Martin Klang, and Michael Zbyszyński, here is the abstract:

This article presents a custom system combining hardware and software that senses physiological signals of the performer’s body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician’s body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.

Hyperorgan interactions, suspended choirs, and the The Tale of the Great Computing Machine

_FV — Sun, 09 Oct 2022 13:43:07 +0000

R1 Reactor Hall

Skandia Cinema Organ

YUMI robot playing the Vocal Chorder

Testing the Suspended Choir

Back in Berlin after a few days of work down in the R1 Reaktorhallen, KTH KTH Royal Institute of Technology for a unique opera piece: The Tale of the Great Computing Machine. A project led by Åsa Unander-Scharin and Carl Unander-Scharin. I took care of designing the gestural interactions with the Skandia pipe organ inside R1, the interactions with a set of speakers mounted on motorised winches (which we call “the Suspended Choir), as well the interactions between the organ and robots that will perform alongside humans. It’s all going to be live, and there are several other talented collaborators that are taking care of live visuals, lights, sound, and more. We had to network quite a few computers in order to make everything work in such a big and unique space.

The opera is based on the novel “The Tale of the Big Computer” written by Olof Johannesson in the 1960s. The book describes the rise of an intelligent network of computers and its relationship with humans. Olof Johannesson is actually a pseudonym of Hannes Alfvén, a physicist who would win the Nobel Prize for his work on magnetohydrodynamics just a few years after the book was published.

Premiere on the first of December, and apparently many shows are already sold out!

Talk at the “Mapping Social Interaction through Sound” symposium, Humboldt University, Berlin

_FV — Fri, 27 Nov 2020 12:10:12 +0000

I was invited to participate in the Mapping Social Interaction through Sound symposium on 27-28 November 2020. The symposium is organised by Humboldt University, Berlin and – as it is customary these days – will take place on Zoom.

This is the abstract of my talk.

Building and exploring multimodal musical corpora:
from data collection to interaction design using machine learning

Musical performance is a multimodal experience, for performers and listeners alike. A multimodal representation of a piece of music can contain several synchronized layers, such as audio, symbolic representations (e.g. a score), videos of the performance, physiological and motion data describing the performers movements, as well as semantic labelling and annotations describing expressivity and other high-level qualities of the music. This delineates a scenario where computational music analysis can harness cross-modal processing and multimodal fusion methods to shift the focus toward the relationships that tie together different modalities, thereby revealing the links between low-level features and high-level expressive qualities.

I will present two concurrent projects focussed on harnessing musical corpora for analysing expressive instrumental music performance and design musical interactions. The first project is centered on a data collection method – currently being developed by the GEMM research cluster at the School of Music in Piteå – aimed at bridging the gap between qualitative and quantitative approaches. The purpose of this method is to build a data corpus containing multimodal measurements linked to high-level subjective observations. By applying stimulated recall (a common qualitative research method in education, medicine, and psychotherapy) the embodied knowledge of music professionals is systematically included in the analytic framework. Qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Initial results suggest that this process is pivotal in building our multimodal corpus, providing insights that would be unattainable using quantitative data alone.

The second project – a joint effort with the Computing Department at Goldsmiths, University of London – consists in a sonic interaction design approach that makes use of deep reinforcement learning to explore many mapping possibilities between large sound corpora and motion sensor data. The design approach adopted is inspired by the ideas established by the interactive machine learning paradigm, as well as by the use of artificial agents in computer music for exploring complex parameter spaces. We refer to this interaction design approach as Assisted Interactive Machine Learning (AIML). While playing with a large corpus of sounds through gestural interaction by means of a motion sensor, the user can give feedback to an artificial agent about the gesture-sound mappings proposed by the latter. This iterative process results in an interactive exploration of the corpus, as well as in a way of creating and refining gesture-sound mappings.

These projects are representative of how the development of methods for combining qualitative and quantitative data, in conjunction with the use of computational techniques such as machine learning, can be instrumental in the design of complex mappings between body movement and musical sound, and contribute to the study of the multiple facets of embodied music performance.