Physical Computing Midterm Project
SVA Interaction Design MFA
by Sunnie Sang & Michie Cao
INITIAL CONCEPT:
Brainstorming: performance, gesture, music, DJ, dancing, projecting
Body motion sensing performance: developing a tool or instrument for performers to control certain aspects of performance, creating easier real-time experimentation.
Musical Glove: using gesture to control music/MIDI output (interact with computer through physical object)
(Thanks for the great help from the Visible Future Lab staffs, Leif, Boris and Tak!)
sketching of initial ideas
FINALIZED CONCEPT:
Sensors and switch as input
3 Flex sensors: manipulate 3 different analog qualities in a musical piece, such as volume, frequency, pitch, through simple finger bends.
1 contact switch: by touching conductive pins on thumb and index finger to play notes or sound effects, such as drums.
1 accelerometer: attached at the wrists on the glove. Through rotating the wrist to left or right, to play two different sound effects.
And 1 slide switch for Lilypad: to control the on/off of the input
Processing technology:
– Lilypad Arduino board
– MIDI software: Ableton Live 9 (through a MIDI cable)
– Max (interactive media application)
Output equipment:
– computer build-in speaker
PROCESS
Stage 1: Created a basic circuit using an Arduino on a breadboard with 2 potentiometers and 1 contact switch.
On the first step to realize our concept, we started the experiment from building simple circuit with Arduino Leonardo board, 2 potentiometers, instead of flex sensors, and a contact switch to get accurate output datas. We focused on writing an Arduino code that could read all the digital and analog input values simultaneously and accurately. Getting the data read from potentiometers(from 0-1023), we mapped them to a range of 0-126 (corresponding to the MIDI data range) and execute a specific MIDI command, i.e. noteOn (144,3,64). We also added the accelerometer into circuit testing, and divided its Y-value into 3 ranges corresponding to the wrist rotation movements. When reaching the two thresholds, the accelerometer control two different sound output.
Stage 2: Replace the potentiometers with flex sensors and “communicate” with the MIDI software.
After successfully tested out the circuit, we replaced all the potentiometers with flex sensors. The commands from sensors were then transferred through a MIDI cable to a MIDI music workstation, Ableton Live, where we “mapped” the output data to certain MIDI controllers. To help accurate mapping, we used Max to simulate Arduino data reading and built mapping relationship with Ableton.
Max interface
In Ableton, we import a mp3 music clip as the background music, and specifically programmed the 3 flex sensors to control melody volume, chime volume and chime pitch in Ableton. The contact switch, when connected, played a low drumbeat. Also, when turning the accelerometer to left or right, two different tambourines sound effects were played.
Ableton mapping mode dashboard
Arduino code
Stage 3: Move the circuit onto a glove and use Lilypad Arduino as “control center”
Once we figured out the basic circuit and functionality of the musical controller, we faced the hardest challenge of transferring all the wires and sensors from breadboard onto the small and flimsy surface of a glove. In order to provide portability, we had to replace our original Arduino Leonardo board with the Arduino Lilypad, a microcontroller board designed for wearables that could be attached by thread to the glove.
sketching of the circuit design on glove
testing the circuit with Arduino Lilypad board
We manually sewed the 3 flex sensors onto 3 fingers of the glove. To avoid the conductive threads overlapping (because conductive threads are similar to exposed wires) and to fit more flexible hand movements, we had to trade in conductive threads, thick, rigid wires for thinner, malleable wires (which are fragile after soldered).
Also, utilizing the glove we bought came with two pre-existing two “touch pods” (initially designed for accessing mobile touch screens while wearing gloves), we transformed them into the improvised contact switches. Here we sewed with conductive threads to connect the conductive fabric on the “touch pods” and the power/ground holes on Lilypad.
Stage 4: Adjust sound effect mapping in Ableton
When the physical construction of the glove was ready to go, after a lot of carefully sewing and soldering, we re-connect the glove with computer, and adjust the sound effect mapping in Ableton to guarantee it performed well as ideal.
SUCCESS
In 3 weeks, we successfully realize our initial idea with a good project planning to operate step by step. Through this process, we experienced the excitement and challenge to build circuit on a wearable object, and also manage to learn the infinite possibility of a new protocol – MIDI. Though what we learnt was just the tip of the iceberg, we had fun and looked forward to explore more in the future.
Technically, we achieved highly calibration of the 3 flex sensors. While programming our code, we carefully recorded and refined the flex sensors’ output range many times, and successfully achieving the sensitivity of reading the slightly finger bend and and produce a discernable change in music effects. This create a satisfying experience for users through giving them simultaneous, immediate response to their control.
Another success is the glove’s versatility. By connecting the MIDI cable, the glove could essentially work with any MIDI instrument and also be programmed to control a variety of MIDI synthesizers as desired. We see its possibility for people to create their own music pieces with a controller glove like this.
The controller glove, featuring the cyborg-y all-revealing look, is really stand out. Since the glove controller was complex to build on a tiny space – with multiple resistors, sensors and dozens of wires – the exposure of all the technical construction is what makes the glove more visually appealing. (We are happy that people associate our glove with the IRON MAN glove! )
CHALLENGES
Dealing with conductive threads within a tiny space. When Michie and I started building the circuit on glove, we tried sewing with conductive threads to connect sensors and the Arduino Lilypad. Soon we realized the conductive thread is similar to exposed wires that could be dangerous if overlapping, since the threads would unavoidably go different directions to reach the corresponding pins on Lilypad. Also, the conductive thread was hard to knot that could leave split threadiness touching each other. We once saw tiny (nearly burnt) spark on the glove cause by short circuit, then we replaced the conductive threads with thin wires.
Connecting a pile of fragile thin wires to the Lilypad. By finishing the glove’s physical construction, we faced a main challenge of its fragility and movement restriction. With the wires being so thin and taut, we encountered many instances of accidentally breaking the wires while putting it on or testing on hand, so the glove had to be frequently fixed.
Separately, accurately mapping the sensors with music effects in Ableton was another big challenge. As freshmen to MIDI communication, mapping the glove with Ableton was a painstaking process for us. On the one hand we had to guarantee the Ableton receiving correct data input from the sensors (using Max intermediately), on the other hand, to guarantee the data received correctly mapped to a certain MIDI effect in Ableton by frequently trouble shooting.
Additionally, the hand movement restriction. with the USB cable connecting to the Lilypad at an awkward angle, we discovered the user was highly aware of the limited wrist movement and faces difficulty in utilizing the accelerometer sensor naturally. Obviously, the glove should be improved to fit natural hand gestures, so that create a more worry-free user experience.
FUTURE IMPROVEMENTS
If Michie and I ever do pursue this project further, we will eventually need to replace the wires with thicker ones and to re-orient the Lilypad placement on the glove. A looser and bigger glove will be more ideal to fit in different users’ hand without too tight to drag. If possible, I think building another controller glove to pair up with the existing one will help to create a more enrich music controlling experience with two hands, wearing gloves at the same time, cooperating to perform.
– – – – – –
Arduino Code:
#include <SoftwareSerial.h>
//Variables
//On/Off Switch
int currentPowerState;
const int powerPin = 13;
//Flex Sensors
byte note1 = 0;
byte note2 = 0;
byte note3 = 0;
int AnalogValue1 = 0;
int AnalogValue2 = 0;
int AnalogValue3 = 0;
//Contact Switch
const int switchPin = 10;
boolean ifPressed = false;
int lastNotePlayed = 0;
int lastSwitchState = 0;
int currentSwitchState = 0;
//Accelerometer
int AnalogValueY=0; // x-axis of the accelerometer
const int ypin = A3;
int currentYState = 0;
int lastYState = 0;
//Software serial
SoftwareSerial midiSerial(2,3); // what’s 2?
void setup(){
digitalWrite(powerPin, HIGH);
pinMode(powerPin,INPUT);
pinMode(switchPin, INPUT);
Serial.begin(9600);
//Set MIDI baud rate. Standard is 31250
midiSerial.begin(31250);
}
void loop(){
//Power Switch
currentPowerState = digitalRead(powerPin);
if(currentPowerState == HIGH){
// Serial.println(“ON”);
// delay(30);
//Flex Sensor A0
AnalogValue1 = analogRead(A0);
//Serial.println (AnalogValue1);
constrain(AnalogValue1, 335, 538);
note1 = map(AnalogValue1, 335, 538, 0, 126);
constrain(note1, 0, 126);
//Flex Sensor A1
AnalogValue2 = analogRead(A1);
constrain(AnalogValue2, 430, 518);
note2 = map(AnalogValue2, 430, 518, 0, 126);
constrain(note2, 0, 126);
//FlexSensor A2
AnalogValue3 = analogRead(A2);
//Serial.println (AnalogValue3);
constrain(AnalogValue3,322, 442);
note3 = map(AnalogValue3, 322, 442, 0, 126);
constrain(note3, 0, 126);
//Accelerometer
//AnalogValueX = analogRead();
AnalogValueY = analogRead(ypin);
//AnalogValueZ = analogRead();
if (AnalogValueY >= 355){
currentYState = 1;
}
else{
if(AnalogValueY <=310){
currentYState = 2;}
else
currentYState = 0;
}
if((currentYState == 1) && (lastYState == 0)){
noteOn(146, 75, 64);
}
else{
if((currentYState == 2) && (lastYState == 0)){
noteOn(146, 71, 64);
}
else
noteOn(146,60,0);
}
lastYState = currentYState;
//Contact Switch
currentSwitchState = digitalRead(switchPin);
if((currentSwitchState == 1) && (lastSwitchState == 0)){
noteOn(144, 60, 64);
}
else if(currentSwitchState==0) {
noteOn(144,60,0);
}
lastSwitchState = currentSwitchState;
if(AnalogValue1 > 0){
//continuousController(Control Change Value, Channel #, Note)
noteOn(176, 1, note1);
}
if(AnalogValue2 > 0){
//continuousController(Control Change Value, Channel #, Note)
noteOn(176, 2, note2);
}
if(AnalogValue3 > 0){
//continuousController(Control Change Value, Channel #, Note)
noteOn(176, 3, note3);
}
/* // print the sensor values:
Serial.print(“X:”);
Serial.print(analogRead(xpin));
// print a tab between values:
// Serial.println(” Y:”);
// Serial.print(analogRead(ypin));
// print a tab between values:
Serial.print(“t Z:”);
Serial.print(analogRead(zpin));
Serial.println();
// delay before next reading
delay(30); */
}
else{
Serial.println(“OFF”);
delay(30);
}
}
// plays a MIDI note. Doesn’t check to see that
// cmd is greater than 127, or that data values are less than 127:
void noteOn(byte cmd, byte data1, byte data2)
{
midiSerial.write(cmd);
midiSerial.write(data1);
midiSerial.write(data2);
// prints the values in the serial monitor so we can see what note we’re playing
Serial.print(“cmd: “);
Serial.print(cmd);
Serial.print(“, data1: “);
Serial.print(data1);
Serial.print(“, data2: “);
Serial.println(data2);
}
Student blogs:
http://komichiewa.tumblr.com/
http://sunnie-sva-physicalcomputing.tumblr.com/
helloCircuits! 