Pebble: A Social Aide

1. Concept: Supporting Socializing with Thoughtful Technology
2. Product Overview
3. Interaction Diagrams
   1. Step 1: Conversing with the Necklaces in range
   2. Step 2: Contact Exchange
   3. Step 3: Deciding to Remember
   4. Step 4: Machine Learning Time
   5. Step 5: Memory Created
   6. Step 6: Remember through Bluetooth
4. Equipment Diagrams
5. Coding Highlights
6. Final Thoughts

Concept: Supporting Socializing with Thoughtful Technology

Technological interactions and social interactions are seen as polar opposites. Person-to-person interactions require attentive listening, open communication, and an effort to sustain dialogue. More often than not, technological interactions demand a user’s attention away from other people, their environment, and inertia to stay in one place and stare at a screen and/or hit some buttons.

What technology could be designed to enhance social interactions instead of detracting from them?

With this question in mind, our team explored how technology could work “behind the scenes” and help bring people together instead of pushing each other away.

Product Overview

This project showcases a network of devices that communicate with each other. The devices notate in real-time a live conversation that the user is having with another person. As a user is talking to another user, the network updates based on what has been said and temporarily stores what has been said.

A user can either let the interaction pass, or with the tap of a button, a user can choose to receive a summary of that conversation, who they just talked to, and that other person’s interests.

---

Interaction Diagrams

Trying to describe meeting someone for the first time via code is much more complicated than I originally anticipated. There are a lot of unsaid and implied steps in basic face-to-face communication between two humans. It’s even harder to try and track the steps of our memory, which does not have a clear protocol on how things get stored in our brain. At any rate, below is my best attempt at describing the flow of the devices trying to “remember” an interaction.

Step 1: Conversing with the Necklaces in range

Microphone in necklace casing is connected to p5 sketch running locally in browser. P5.SpeechRec translates words into strings for program to temporarily store.

Step 2: Contact Exchange

User has option to press the button on the watch to save conversation. Regardless, upon tapping each watch together (both with RFID tags and readers in an ideal world), each user exchanges pre-defined contact information with the other. For our prototype, it appeared in the p5 sketch.

Step 3: Deciding to Remember

Step 4: Machine Learning Time

Conversation from p5.SpeechRec was stored as an array of strings. Upon button press of watch, String array for conversation goes to python backend where two models from huggingface.co (Name Entity Recognition and Summarization). Thank you to ChatGPT for helping me setup the initial python file, even though some of the code it gave did not work until I tweaked it (a lot).

Step 5: Memory Created

Step 6: Remember through Bluetooth

---

Equipment Diagrams

---

---

Coding Highlights

I won’t share all the code (for free, anyway). However, I will list all the things this prototype ended up using (mainly behind the scenes) in over 1000 lines of code.

• p5.SpeechRec
• Geolocation API
• p5.ble.js with RFID
• Arduino/C
• Python with huggingface.co ML models integrated/JSON input and output
• p5.js

---

Final Thoughts

This project validated that I can do so much (maybe too much) with code and integrating into devices. In respect to ICM, I felt like this project gave me the confidence to label myself a programmer.

I wish I could have done a better job of making something more physically engaging for the end of Physical Computing, such a great and inspirational class. I was thrilled to work on the ‘invisibility’ of a social interaction but I would to make something that helps people bond by sharing a physical experience going forward. However, I feel comfortable doing so much more with an Arduino and pursuing any idea that uses computers with physical interaction and away from screens.

All things considered, I’m proud of making any advancement on a concept that helps people with making social interactions easier. There’s a lot more work to be done. Yet, I learned that starting with a simple system and layering another simple system in tandem is a powerful way of building the complex and far-reaching.