LiquidScore - 2012

Project Overview

LiquidScore blurs the line between an interactive computer game and a musical performance. A musician interacts with an animated musical score that is displayed for both audience and performer. Real-time audio tracking enables the computer to follow the player's progress and respond to choices made by the musician. Musical accompaniment and sound-design elements are created on the fly based on recordings of the performer's audio stream. As a real-time audio-visual improvisation environment it provides multi-modal feedback to the improvising musician.

This served as my final project for Stanford's CS 248 - Interactive Computer Graphics during winter quarter, 2012. This project was one of six to be selected for the final round of the class video game competition, in which we each presented our creations before a panel of judges.

Graphics Features

  • Written using OpenGL in C++ with GLSL shaders
  • Custom 3D geometry
  • Custom pseudo-physics implementation
  • Smooth camera motions
  • Custom Phong shaders with outlines
  • Fullscreen anti-aliasing

Audio Features

  • Written using MaxMSP
  • Audio pitch and amplitude tracking of a monophonic instrument
  • Automatic audio recording and parsing based on pitch value
  • Custom granular synthesis engine for buffer playback
  • Messages passed to and from MaxMSP using the Open Sound Control protocol

External Libraries and Tools

Score Features

One important goal of the project is to a create score representation that is engaging to both a musical performer and an untrained audience, by balancing two different levels of meaning. The first level is that I want to display readable and accurate staff notation to the performer. I want them to feel that they have real agency within the score to move and affect the graph on a one-to-one basis. The second level is that I want the visual material to be artistically compelling and to have interesting behavior, so that it is okay that the audience doesn't necessarily read staff notation. Hopefully the key to further understanding is acknowledging that the performer is directly shaping the visual and sonic material by their actions.

By default, notes are tracked and represented to the performer in G clef. However, with a slight modification of the startup options, another clef may be used to accommodate lower or higher ranges, or transposing instruments. This is simpler than it sounds: right after the pitch detection occurs, the transposition amount is simply added to the MIDI note representation, so the rest of the system can go on "speaking G clef". Part of the reason this works is because the audio accompaniment is comprised solely of live recordings of the instrument, so no changes need to occur in the playback engine.

Development Screenshots

Early 2D draft in which all rendering used the fixed pipeline, and every line was drawn using GL_LINES with differing thicknesses for staff lines, outer circles, and connection lines.
Slight modification in which the movement of the notes is no longer constrained to the X-Y plane. Notice that since I have not yet updated my connection line code, so there are errors in the lines over- or under-shooting the note perimeters. At this point, the lines are connecting from note center to note center but ending before they enter the note's radius. Additionally, line thicknesses stay the same regardless of distance.
Still using the fixed pipeline, I have switched over to using 3D geometry for toruses and cylinders. I wrote all the code for specifying vertex positions, and used GL_TRIANGLES to draw everything using the color black. I have also updated the connection line code so that two notes will connect from the nearest edges of their toruses, rather than the note center.
I have stopped using the fixed pipeline and have transitioned to using my own shader programs. Using a standard Phong shader, I have assigned a single material to all geometry and specified a light position and color. I have also added in staff lines as narrower cylinders, and noteheads as a flat planar circle.
To make things a little more stylish I have quantized both the diffuse and specular components of my Phong shading to be one of only two values each. This gives a cartoon look with hard transitions between the shaded areas. Additionally, I have updated the normals of my noteheads so that they appear round when shaded, even though they are still entirely flat! Also, I experimented with adding a shaded plane below my notes so that I wasn't stuck with a white background.
All borders and transitions between colors currently appear blocky and jagged, so anti-aliasing is necessary. I chose to render my scene at three times the actual size in each dimension, then perform averaging in a custom downsampling shader. For this image, instead of choosing my nine sampling locations to lie within the target pixel (sub-pixel sampling), I tested the appearance of allowing nine pseudo-random locations to stray outside of the target pixel. The result is a noisy blurring, which, although nice, I didn't end up using.
Once I got a clean version of my downsampling shader working, I still wasn't happy with the style of the cartoon shading. I decided to take the cartoon idea one step further by adding solid black outlines to my shader. This is actually rather simple to implement. If the dot product of the eye position and the surface normal is above some threshold value, color the fragment using the previously created toon shader, and if not, color the fragment black. (Remember, the more perpendicular two vectors are to each other, the closer the dot product comes to zero.)
Putting together my toon shader, my outlines, and my downsampling shader, gave me the final look of the geometry. To give a sense of the absolute movements of the notes and camera I added in a simple but somewhat abstract background texture onto my ground plane. This produced the final look of the environment.