STC 309-A Storytelling System

An interactive indoor “fireplace” for reading stories together (STC309 Final Project)

During the second half of the semester, I continued my original idea of an interactive system that augments the group storytelling experience. The system allows listeners to share their reactions to the story by squeezing pillows, which then trigger lights and vibrations in a central tablecloth. This is the same combination of interactions as the midterm, but now with separate pieces that communicate via bluetooth, and a full system that can be moved around and set up for storytelling. The main components of the project were circuits, sewing, and arduino code.

There are three main pieces in this system. First, the book cover (Feather RF52)  is the central hub of the system; it listens to pressure readings from the pillows and sends that information out via Bluetooth Low Energy (BLE). The pillows (Flora + BLE module) contain pressure sensors that I made myself, and periodically read the sensor value and send it out via BLE. Finally, the table piece (Arduino Uno + BLE shield) listens for the sensor values and lights up or vibrates when it hears a high sensor value (indicating that someone squeezed a pillow).

Prototypes and Progress Steps

Makerfaire
Because I had sewn some pieces with nice fabric and had LEDs and a pillow for the Makerfaire, I was able to explain the essence of the project to people who stopped by. I received really helpful feedback from some of the visitors. One teacher liked the idea of using the pillows in her classroom to give the kids something to play with during storytime, so she would be able to see if they were engaged, and having something to do would hopefully keep them from moving around or getting too distracted. A few people brought up the potential of the pillows as a nonverbal communication device for people who have trouble expressing themselves verbally.

I took this system to the Makerfaire and it broke almost immediately. I was trying to implement Bluetooth LE connectivity using a Feather Huzzah32 (with an ESP32 chip) as a central device and a Flora BLE (with RF51) as a peripheral. It worked for about five minutes. I started working with the ESP32 because I wasn’t sure if I was going to do Bluetooth, Bluetooth LE, or Wifi, but once I knew I was using Bluetooth LE for its simplicity, the ESP32 was not worth the hassle. Getting the ESP32 to work well with the Adafruit BLE modules was just too messy and error prone (including bug reports on Github that I could do nothing about), so I decided that I needed to work with a more friendly device, and ended up choosing the Feather RF52 for the rest of the project.

Symposium
The Symposium was shortly after the Makerfaire, so I wasn’t able to fix the bluetooth connectivity in time. Instead, I found a way to hardwire the system so that it could be explained and played with even while I wasn’t at the table. I tried soldering 5mm snaps to the Flora, which allowed me to easily connect and disconnect it, since I didn’t have to cut it out of the cloth every time.

At the symposium, I got the very useful advice to try silicone-coated stranded wire instead of the solid-core wire I had been using. The single-core copper wire was very stiff and occasionally broke if bent in too many directions. When I started using stranded silicone-coated wire, it was much more flexible and easy to work with, particularly with the pillow and table circuits, since they get moved around a lot.


Final Presentation

For the final presentation, I actually switched books. I started the project with Stardust by Neil Gaiman, which is a “fairytale for adults” that I thought would be a fun story but also more interesting to adults than a normal fairytale or something else geared towards children. You can see the book and the blue starry cloth in the symposium demo, that matched this book. But just before the final presentation week, a book called Circe was released. My sister got it for me as a gift, and I decided to go with that book instead. Circe is a retelling of the Odyssey from the perspective of Circe, and the way it’s written works well for reading aloud. I’m very interested in retold stories, and I liked the idea of using something based on Greek mythology, since you could use the system to read many different myths while keeping a similar theme.

I figured out all the visual elements by this point. I changed the color scheme and made a new book cover, and finished making the pillows. I also used a combination of soft white cloth and stiffer tulle to create the cloud-like table piece. This was a move away from the initial idea of a tablecloth, but I wanted to be able to have a visual of the vibration motors, and it made more sense to include the lights within the cloud mesh as well. It was tricky to make the vibrations visible through movement; I think the motors might have to be fixed to some of the upper tulle to get the tulle to rustle.

Before I made the cloud piece, I did experiment with sewing rails of conductive thread for the neopixels. My conclusion is that it takes a lot of testing to make sure the tension is correct, but I was never able to get a really reliable system with the conductive thread sewn by machine.

The bluetooth connection had a lot of issues during the demo, but I was able to use my phone to communicate with the table piece and demonstrate the visual effect.

Post-Final Presentation

After more debugging and experimentation, I finally figured out what the bugs were in the bluetooth communication! It’s a little finicky to get into detail, but there was an issue with the Flora waiting for the serial monitor to open, and also an issue with delays and conversion from integer sensor readings to strings sent via BLE. Here is the setup process for the functioning final piece:

Circuits

Tips and Techniques

Since I spent the first half of the semester exploring different interactions and trying to condense my ideas on storytelling into a project, I didn’t get very far on learning how to make good circuits. This resulted in demo circuits that were literally put together with electrical tape:

Thus, one of the first things I did in the second of the semester was learn how to solder. This allowed me to finally put together reliable circuits that wouldn’t fall apart, and I was able to get a clean workflow: first, prototype circuits with alligator clips and breadboards. Once that was working properly, I would put together a schematic, and then translate that to a soldered circuit. I had never worked with transistors or motors before this, so it took some time to figure everything out, but ultimately the circuits worked quite well! Here is an example of the prototype process for the vibration motors:

Book Cover (Feather RF52 + velcro switch)

The book circuit is very straightforward. There is a simple switch made with velcro that essentially detects whether the book is open or closed. The connections for that switch were sewn with conductive thread directly onto the book cover. I found that the stiff 4-ply thread made a very good connection between the board and the velcro, but the thin 3-ply silver-plated thread, although much easier to work with, was unreliable in terms of connectivity.

Pillows (Flora + BLE + Pressure Sensor)

The pillow circuit uses a Flora connected to a BLE module and a pressure sensor. Like for the midterm, I made the pressure sensor myself using felt, velostat, and copper table. However, I changed the wiring to add a regular resistor in parallel with my variable resistor (the pressure sensor), which greatly increased the sensitivity. With the new wiring, it’s much easier to detect different pressures, rather than simply detecting “squeezed” or “not squeezed” as was the case initially. I tested with different resistors to find a resistivity that gave me the best readings.

Table (Arduino w/ BLE Shield + vibe motors + neopixels)

The circuitry for the table was definitely the most complicated. This was my first time working with transistors and motors, and it took some experimentation and testing to get it right. I tested with different transistors to see what kind of variation I could get in the vibration levels.


 

Code

This is not all of the code involved, but gives a summary of the logic used to read and interpret the sensor values.

Adafruit Flora (pillow, sending values)

const int analogInPin = A9;  // Analog input pin that the potentiometer is attached to
int sensorValue = 0;        // value read from the pot

void setup(void)
{
 setupBLE();
}

void loop(void)
{
 // Check for user input
 char n, inputs[BUFSIZE+1];

 sensorValue = analogRead(analogInPin);

 inputs[0] = (char)(sensorValue / 4);
 inputs[1] = 0;
 ble.print(inputs);

 delay(200);
}

Arduino Uno (table, receiving values)

int vibes = 3; // transistor pin
int pixelPin = 10; // neopixel pin

// initialize neopixel strip
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(6, pixelPin, NEO_GRB + NEO_KHZ800);

void setup(void)
{
 pinMode(vibes, OUTPUT);
 pixels.begin();
 pixels.show();

 // advertise until connected
 setupBLE_noserial();
}

void loop(void)
{
 if ( ble.available() )
 {
   // get value (squeeze) from bluetooth
   int c = ble.read();
   // if very high, vibrate
   if (c > 230) {
     analogWrite(vibes, 255);
     delay(100);
   } else {
     analogWrite(vibes, 0);
   }
   // if high-ish, light up
   if (c > 50) {
     colorWipe(pixels.Color(200, 120, 0), 50); // gold?
     colorWipe(pixels.Color(100, 60, 0), 50);
   } else {
     colorWipe(pixels.Color(5, 3, 0), 50);
   }
 }
}

Reflections and Future Plans

I learned a lot in the making of this project, from soldering and transistors to sewing machines and thread tension. It was really rewarding to learn about things I had never tried before. As next steps, I’d like to clean up the sewing and integrate a few more things into the code, such as the microphone (which I soldered in but didn’t use) and using the book as the switch/trigger for the system.

I feel like now I can go on to make new systems that combine sensors and motors and microprocessors with cloth and other interesting materials.