Introduction

Sonification means transforming astronomical data (telescope images) to sound. It intends to allow people to inspect and communicate astronomical data with the auditory sense as well as the visual sense. Early research studies show its effectiveness in understanding astronomical data. NASA and its partners have carried out various sonification projects with their telescopes such as the Webb, Chandra, Hubble and Spitzer telescopes.

Sonification is interesting and important in many ways for many scientists and space enthusiasts. It's like renaissance in interpreting astronomical data. However, most of the general public don't know about it, probably because sonified sounds are distributed through "pull" media such as YouTube. In fact, most of our our parents, grandparents, friends and teachers didn’t.

Therefore, in this project, we decided to replay sofinifed sounds and display images in everyday objects like mirrors and picture frames, so we can “push” the sounds to nearby people. We also decided to design our own algorithm to sonify constellations. We hope to help everyone, even toddlers, experience and conceptualize the stars in constellations.We approached this project by using our diverse skills and experience in coding, electronics, musical instruments and music theory.

Smart Mirrors and Picture Frames

We built "smart" mirrors and picture frames in order to embed sonification to daily life and better disseminate cosmic soundscape to the general public. Each of our 4 prototypes is equipped with a Raspberry Pi computer, a Blueetooth speaker and a digital display (64x64 LED matrix board or e-paper display). On the Raspberry Pi Computer, Python programs download NASA's sonification data, replay its audio portion with a speaker and display its video/image portion with a display.

Our devices also download and read Wikipedia descriptions on an astronomical object in the sonification data. They are integrated with a cloud data storage called Kintone. Thus, you can turn on and off each device with the cloud, whenever you want. You can also set what time to turn in on. This way, you can experience cosmic soundscape as you brush your teeth and hair, as you cook/bake in the kitchen, when you wake up, or when you go to bed.

Constellation Sonifier

Our sonifier is intended to sonify the stars in constellations, rather than complex telescope images. Starting with simpler datasets, we would like to carefully analyze and validate sonification algorithms. Note that our sonifier still uses three-dimensional data - two-dimensional space with an extra dimension for the brightness of stars (apparent magnitude).

The design of our sonification algorithm is driven by music theory. Given a set of stars in a constellation, we make a grid on it. We interpret the middle line as the third line of a five-lined staff and place four extra ledger lines above and below.

The following picture shows the big dipper as an example constellation. It's Alaska's state flag, which contains the Polaris as well as seven stars.

The following picture shows how to place a grid on it and how to determine a five-lined staff and eight extra ledger lines.

Given a five-lined staff, we place different clefs to produce different voices (tracks). The following picture shows a score of the big dipper, which contains three voices (tracks) with treble, alto and bass clefs.

For each track, we generate a MIDI-like data structure that contains the onset time, note number, velocity and gate time for each note in a score. The brightness of a star impacts the velocity of its corresponding note. Our algorithm can also produce a chord from a single note, as preferred. Then, we synthesize a particular instrument sound for each note, according to a given clef. For example, piano and violin sounds may be synthesized with the treble clef, viola sound may be synthesized with the alto clef, and cello sound may be synthesized with the bass clef.

When mixing multiple tracks, our algorithm can perform fade-in effect and produce stereo sound to vividly express "chamber music of a constellation".

Our sonifier generates the image of a generated score with the ABC music notation and uploads it to a cloud data storage called Kintone, along with WAV file, MP3 file and constellation image. We have tested our algorithm with two constellatios: the big dipper and the orion. See the "project demo" (7-slide project summary) for more details, synthesized sounds and demonstration videos.

We all learn the big dipper and other constellations only through visualization (e.g., picture books and encyclopedias). Why not with sonification? We believe sonification is effective for anyone who learn constellations. In this context, our sonifier can/should be useful for any astronomy learners (both grown-ups and kids, even toddlers) to conceptualize constellations in a new way. Our ultimate goal is to help promote "sound logos" of constellations.

To help raise the public awareness of cosmic soundscape, we plan to:

place our devices in schools, libraries, town halls and other places where many people are passing by.
ask science teachers to cover sonification with our devices in class.
run concerts of constellation sound at local schools (e.g. preschools).

Our GitHub project web site provides hardware purchase and setup guide, circuit diagrams, Raspberry Pi setup guide, installation instructions for required software tools/libraries and ready-to-run programs. Anyone interested can immediately learn how to reproduce our devices and use our sonifier. We hope you find this project interesting and useful and then learn about something about sonification!

2023 NASA Space Apps Challenge

Renaissance Quintet