Original Concept.

Fundamentally originating as an installation-based device, the original concept was simple. A box, relatively well sealed from light (possibly made of wood), rectangular and following a few set of essential design measurements:

1. Long enough to allow a webcam to focus form one end to the other.
2. Around an a5 page high, to allow for a relatively meaningful amount of space to draw on "sketch-cards"
3. A slot to incorporate a sketch-card.
4. Cable outputs to Speaker systems and USB out to computer running applicable software.

The overall concept was to create a system that would interpret a user's drawing and output it as musical tones, based upon the x and y axis of the said drawings.

This caused me to create a checklist of requirements:

1. Materials to create box.
2. A webcam capable of handling relatively macro focus
3. An internal low-power, high level light source (for webcam to see drawings)
4. Software to interpret this input from the webcam.

I felt that the overall design for user experience should be as simple as possible, with all technology hidden from them, and it to be so simple that it could be left for the public to play with. I took inspiration from a Nickelodeon, simple to use by a crank and all of the magic technology required hidden from view, so that a user could enjoy the output with a sense of mystery as to how it functioned.

Stage one, was to research webcam focus length and to build the appropriate housing.

Research

There were various factors involved in choosing the right camera. Low light and short focus were essential, however cost effectiveness and the ability to be seen by MaxMSP on a MacOS driven computer were also crucial.

In the end, it was luck. I had researched as many webcams as possible online, and decided to go for the relatively inexpensive but highly reviewed "Creative Labs Live! Cam HD" because it was allegedly supported by MacOS X natively, and it was able to shoot at hd 720p, higher than its nearest competitor.

On it's arrival, a focus test was performed, and in a relatively dimly lit room, I found that the minimum distance possible was around 19cm from the lens. During my wait time for the camera to arrive, I had estimated a 30cm distance based on the 720p built-in camera on my Macbook, and so had began plans to build the box at 30x10x10cms.

- Illustrating the webcam viewing the sketch card.

The second part of my research (admittedly, had to be done after webcam arrival) was designing a MaxMSP patch to fit the concept. During my presentations to my peers, I discussed the concept of an X and Y axis system, whereby the webcams input would be read by .jitter in MaxMSP and the Y axis would be output as "pitch" with the X axis outputting bangs via a clock. These bangs would as a result have a fixed time, resulting in each card being the same length (which would be determined in the final test stages)

My problem with this idea was the sheer volume of data being output and the matrixing involved to maintain a smooth stream if the user were to draw a long smooth line, for example.

Onward I moved, simultaneously researching, building (and deleting) Max patches and creating the housing.

Housing

In one of the only strokes of luck on this project, I set out to procure the materials in oder to build the housing (which, I expected to create from wood) - and discovered while perusing the isles of B&Q that, on offer, were plastic toolboxes. These were cheap, malleable plastic boxes, with a grid on the lid!

This grid helped greatly in cutting the slot for the sketch cards, and, measured 44cm long. Almost double the minimum focus length, and was designed to seal, so no light entered!

With the housing taken care of, I began to place the webcam, and fit lighting. On my hunting, I had found useful push button LED lights, once procured by bandmates in our first dingy studio, and never used. Somehow, however, between moving from a dingy, damp studio to a not-at-all-damp studio the switches on these corroded, and so although I was left with fully functional LEDs, I couldn't power them on.

With a little bit of circuit bending, I found a way to jump the switch on the very simple circuit board, and, through harvesting christmas lights, was able to add a small switch to power the LED light.

I then "ball parked" the webcam, affixing with double-sided tape, and set up a rear blank card to act as a backing plate for the cards to be inserted.

At this point, I felt I could begin testing and experimenting with software more thoroughly, as I had the camera input guaranteed to be what would exist in the final device.

Software Development

Least lucky than my previous venture, my delving into .jitter and its associated modules were met with failing after failing.

I started thinking deeply about what I wanted max to detect. Did I want it to detect the x and y axis as previously mentioned? Did I wish to have the limitations of a set length of the card's x axis?

While researching and consulting peers and previous lecturers who had worked in this field before, I was pleasantly surprised that many patches existed, ready to be modified, in the world of motion detection, which worked for an infinite length of time not relying on my x axis = time idea.

One that I was recommended and found easy to modify was "VideoTrigger" by Jach Proff. An open-source max match designed to detect zones and output based on movement.

Installing Macam webcam drivers allowed the Alleged MacOS native webcam to actually be detected by the system, and in turn MaxMSP.

First experiments with this were successful, however, the software was relatively limited in terms of allowing files to continuously play. After delving into its patch, I realised it used a simple "sfplay" to trigger samples.

On my version of the software (1.01) I couldn't find the ability to "loop" - so added a prepend loop to the sfplay module:

And, after careful adjusting of the brightness, contrast and colour saturation of the webcam in the input settings to be able to fully see a contrasted and clear image of a sketch;

I set about placing "zones" to match the layout of the cards when slotted in, as well as randomly placed ones.

Through calibrating the software in terms of sensitivity and causing an "end on exit" setting to occur on all "zones" - I was able to ensure that once the card was removed from the box, the sounds would stop.

My intentions were to put together a simple and effective system that any user could use, and as a result I opted for 6 motion detection zones, all calibrated to be looking at a single white background. once the system detected the "movement" of the black lines on the sketches, the triggers would playback samples based on when it saw those lines.

Internal sounds, and final arrangement of parts.

With a focusing camera, a slot and rear card functioning as a background for sketch-cards, a sealed box, and functioning software to detect input, I was ready to trail the box on non-musician users, and found an immediate thing they did was look at the computer connected to it, and watching the card slide in to the view of the webcam.

At this point I was certain I needed to hide the computer, and decided to add an internal speaker to the housing, allowing for the output of the hidden computer to appear to come from the device itself.

Adding a golf-ball sized speaker, although sounding easy, proved to be another trial in its own right. I found it was supplied with a hard-wired cable around 3cm long.

I soldered a female 3.5mm jack to 1/4" jack cable, and ran it alongside the usb power cable to the computer.

My box had finally come to life, and I documented my first use of it, here:

Improvements

Like any project set to a deadline, I am filled with ideas that could have been.

One notable improvement would have been to change the "key" that the songs were in through different mood cards. Varying colour of the cards could have been a signal to play back samples in a different key, creating a "mood" for the users.

The concept of enveloping the video before it reached the webcam was actually and idea thought about during the first sketching process, through the idea of a "pinwheel" between the camera and the card.

The idea was that the use would have control of this pinwheel from the top, allowing for slow or fast blanking of the source image before the webcam could see it.

A number of reasons existed to stop this from happening:

1. The webcam had a fixed frame rate, and any movement on the pinwheel was met with nearly always being too fast to be noticeable.
2. The camera, to be recognised by MaxMSP, was set up to run on an open source driver (Macam) and so ran at a lower resolution (160x120) with a relatively noticeable latency. So much latency, that the concept of moving it to create an envelope to the signal in any kind of meaningful way was impossible.

QR Coding a "mood" was another thought, to allow for an alternative to the previously mentioned concept of coloured cards affecting a pitch from the sine tones. This overall idea was interesting to me but almost impossible in my time frame. Where QR code analysis exists and could be implemented in Max, as well as a zone set aside specifically for this, It wasn't something I was able to create in time.

My concept has been accepted to appear at a proposed "sound design zone" at a local festival, and between now and then, QR code moods are the most likely "next stage" to this project.

Summary

My project suffered from similar time and budget constraints that my peers are also well aware of in their own ventures, however I do feel that I managed to create a device that mostly fills its original concept.

To recap:

I wanted to create a device that would be:

• Simple in design
• Installation-based
• Able to convert basic sketches to audio, varying across sketches

My project was fraught with amazingly good and equally terrible luck. The spectrum across the musical output I feel could be much much wider, each sketch card more discernibly different through more zones, more tones, and possibly the addition of a QR code reader zone to change key depending on what kind of card is sketched on. 

The design although functioning in a simple concept, failed to evolve into a larger instrument that could be captivating a user for a log period of time.

My Sonic Sketch Nickelodeon was a journey through modifying Max Patches, through attempts at simplifying design and although was a failure in a potentially understated grandiose, was far above and beyond what I had expected to create alone, and had caused me to feel a sense of achievement at creating a device that fits a relatively open design concept (and works as a proof thereof.)