ImageToSoundHaskell /

Filename Size Date modified Message
app
images
src
test
89 B
1.5 KB
4.6 KB
46 B
19 B
764 B
109 B

ImageToSoundHaskell

Converts an image to MIDI.

stack build
stack exec -- ImageToSoundHaskell-exe -i image.jpg -o output.mid -a processing-algorithm

Or to show a stack trace when there's an error:

stack build --profile --executable-profiling --library-profiling
stack exec -- ImageToSoundHaskell-exe -i images/4-Paul-Signac.jpg -o 4.4.mid -a expanding-circles +RTS -xc

To run the tests:

stack test --file-watch --fast

Processing Algorithms

The available processing algorithms are explained below.

shuffle-to-chord

  1. Read the image into a vector of pixels (red x green x blue).
  2. Shuffle the vector of pixels.
  3. For each pixel in the shuffled list, derive notes based on the value of the red component of the pixel where noteMidiValue = pixel.red / 2
  4. For each pixel in the shuffled list, derive notes based on the value of the green component of the pixel.
  5. For each pixel in the shuffled list, derive notes based on the value of the blue component of the pixel.
  6. For each note in the red, green, and blue note lists, create a chord of three notes [red, green, blue]

Because the algorithm randomly shuffles the pixels in the image, the MIDI generated will be different for each execution.

The Pieces

1 and 2

These processing of these pieces was done in Kotlin and isn't available here, yet?

3 - Jackson Pollock - shuffle-to-chord

To generate midi for the Jackson Pollock image run the following:

stack exec -- ImageToSoundHaskell-exe -i /Users/gregorydavidlong/development/ImageToSound/4JacksonPollock.jpg -o 3.mid -a shuffle-to-chord

4 - Paul Signac - expanding-circles

stack exec -- ImageToSoundHaskell-exe -i images/4-Paul-Signac.jpg -o 4.4.mid -a expanding-circles

5 - Pape Pintura - traced-to-melody

This algorithm requires some human "pre-processing" before being applied. Looking at the Pape we can see that the image has some implied "lines" or "paths". I traced each of these paths in a different color (see images/5-Pintura-traced.png).

The algorithm works by looking at each of these paths - it uses the different colors to distinguish each path - and generates a melody based on the height of the path in the image; the higher the path in the image, the higher the note in the melody. This generates a number of overlapping melodies, one per path, that start and end at different points in time.

stack exec -- ImageToSoundHaskell-exe -i images/5-Pintura-traced.png -o 5.mid -a traced-to-melody

6 - Astral Snake - traced-to-melody

For the Astral Snake we used the same processing algorithm as the Pape Pintura, tracing the "body" of the snake and generating a melody based on it. Each colour forms a different melody; the higher the pixel in the image, the higher the note generated.

Although we used the same processing algorithm as another image we get a completely different result.

7 - Kelly Spectrum - color-to-chord

This algorithm turns colors into chords. Trace a line, left to right, through the middle of the image. For each pixel on the line create three notes - fitted to a scale - one for the red, green, and blue and components each. These three notes essentially form a three-note chord.

This allows us to get a chord for each color in the Spectrum.

stack exec -- ImageToSoundHaskell-exe -i images/7-kelly-spectrum-v.png -o 7.mid -a color-to-chord

9 - path-to-melody

The path-to-melody algorithm is a variation on the traced-to-melody algorithm. Where the traced-to-melody algorithm traverses an image from left to right (and only left-to-right), creating a melody for each different-colored path, the path-to-melody algorithm is able follow a path in any direction.

For a path - where a path is a continuous line of the same color - the left-most point is found, and is used as the starting point. The path is then followed, up/down/left/right (with diagonals) through the image. For each pixel in the path a note is added to the melody, the pitch of which is determined by the pixel's height within the image.

It's important that the path all be of the same color. Some graphics applications (e.g. Inkscape) will anti-alias any path you draw, smoothing the path by changing the color of some pixels. Therefore you will need to use an application like "gs" to remove anti-aliasing:

gs -dSAFER -dBATCH -dNOPAUSE -sDEVICE=png16m \\n -r72 -dGraphicsAlphaBits=1 -sOutputFile=9-trace-2.noaa.png 9-trace-2.pdf

And then run the processing algorithm:

stack exec -- ImageToSoundHaskell-exe -i ~/development/imagetosoundhaskell/9-trace-2.noaa.png -o 9.2.mid -a path-to-melody