• Light-chat is a Chat-GPT enabled system to control a strip of programmable LEDs using a text or voice prompt. For speech to text we use the OpenAI Whisper API. To program the LEDs we use an Arduino. The user prompt will generate new Arduino code to modify the existing light pattern or to generate a new pattern.

• We sometimes get invalid code from the ChatGPT response. We use the response from the Arduino compiler arduino-cli compile to extract error messages and resend the invalid code and the error message to ChatGPT to get new and (hopefully) valid code.
  

• This is a Golang and Arduino project using the Go openAI interface library https://github.com/sashabaranov/go-openai and the FastLED library https://github.com/FastLED.

The main file is light_chat_main.go

It determines if an Arduino is connected via USB, what COM port it is connected to, and what board type it is (atmega328 or Uno).

Command line execution:

go run .\light_chat_main.go

• -p "make lights kinda blue"= text prompt input. If we do not use this p flag then we default to use voice input with the default microphone- we have 8 seconds to record.
• -r (bool) reset the reference (provided example) light pattern code to xmasTwinkleDuino.c. This file contains information about the programable LED strip chipset (currently WS2812B), the number of LEDs on the strip, and which Arduino pin to use for the data output. If the flag is not used, the reference (provided example) light pattern code is duinoCode.ino. In summary: Use this -r flag when you want to reset the light pattern and make a new one (e.g. "make a new calm light pattern that is inspired by the poet John Donne"). Otherwise do not use the flag to modify whatever the existing pattern is (e.g. "make the light pattern less romantic").
• -nb (bool) = no board. Use this flag if no Arduino is connected but you still want to generate a response that is saved in directory responseHistory.

Previous examples are stored here in directory /responseHistory

go run .\light_chat_main.go -p "Make a light pattern showing Barbie's dream house." -nb -r

![Watch the video]

(The example in the video used an early version of the system with a different codeload than discussed below. The prompt was repeated just before the Arduino updated the lights.)

Response:

Barbie's dream house

Here is a modified code that displays a Barbie's Dream House themed light pattern by alternating between pink, cyan, and white lights, which imitates the traditional color scheme of Barbie. ++ It briefly flashes all LEDs red twice at the initial setup as requested ++ .
Note this last part ++ xxx ++ is an incorrect interpretation of rule #5:
5. Only if you can not provide a valid response, please flash all LEDs red twice (do this just once) and then display the existing pattern.

go run .\light_chat_main.go -p "Now make the lights more romantic." -nb

This uses a text prompt input without the -r flag to modify the code generated by the previous response.
This is the ChatGPT comment on the code:

The above code will create a "romantic" atmosphere by decreasing the brightness to a soft glow (80 out of 255). It also slows down the frame rate to a calm 2 frames per second. Most importantly, it sets all the LEDs to red (CHSV(0, 255, 255) is a hue-saturation-value representation of the color red), which is traditionally associated with romance.

I would like a monochrome black and white interpretation of the song Age of Aquarius.

Response: here
The C code did not compile first time- it used an undefined parameter HALF. But the response did not lack in ambition nor inventive interpretation :)

The function createNewPromptFromBadCode uses ChatGPT to modify such non-compilable "bad" code: the function takes the error message from the Arduino compiler and sends this to ChatGPT with the "bad" code, and asks it to correct the bad code so it compiles. If we again get "bad" code, we abort.

go run .\light_chat_main.go -r 
-p "make a new pattern. 
imagine the LED strip is configured in a ring.
 make a  single led dot go round and round the ring, getting faster, and changing colours over the colours of the rainbow. 
 the dot should go round 8 times and be very fast on the final lap.
  there should only be 1 led active at a time. the led should appear to go round the ring of LEDs."

![Watch the video]

Response

go run .\light_chat_main.go -p "Make the colours suitable for a party at 420 wink-wink nudge-nudge say no more" -nb -r 

This uses a text prompt input, with no Arduino board connected, and using the default reference code in xmasTwinkleDuino.c.

The ChatGPT response is stored in directory responseHistory as a text file with filename the same as the user prompt for the light pattern - example for above.

We add "response rules" to ask ChatGPT to describe how it interpretted the prompt. For the above example, this is the comment it generated on the code:
Your ChristmasLights function was replaced by the PartyLights function. The colours were changed from a repeating pattern of red, green, white to green and purple. For a 4-20 themed party these colours are more suitable as green often represents marijuana and purple represents the feeling of relaxation and royalty.

go run .\light_chat_main.go -p "make a new pattern: trip zone for Burning Man." -r

go run .\light_chat_main.go -p "make the motion of the existing pattern slower and a bit more random." 

go run .\light_chat_main.go -p "The requested display has 4 parts. Please try and do all 4 parts in sequence. Part 1: Display a single blue dot that moves down the lightstrip starting slow and speeding up as it moves. Part 2: When the blue dot reaches the end of the lightstrip turn it to a single yellow dot. Part 3: The yellow dot reflects and returns along the length of the lightstrip slowly flashing. Part 4: When the yellow dot reaches the end of the lightstrip flash all lights bright white 4 times." -r

![Watch the video]

This took 1min42 seconds to receive a response and the code compiled first time.
It got it mostly right, except for the part 3. For some reason, in the response ChatGPT4 makes the yellow dot randomly blink:
Part3(): The yellow dot "returns" along the light strip, randomly blinking on its way back.

1. Obtain desired new or ammended light pattern using a text or microphone recording and convert speech to text using the OpenAI Whisper api.
2. Create a new prompt to replace or update the light pattern accordingly.
   We concat the following text:
   "Here is an example of C code for an Arduino using the FastLED library to create light patterns for an LED strip. This is the existing pattern:"
   < insert C code from example reference >
   " Please generate new C code to satisfy the following request and output the modified code in its entirety: "
   < user prompt >
   "These are the rules:"
   < 1. Always format the code in code blocks. etc...>
   
3. Send this new prompt to ChatGPT.
4. Receive ChatGPT response and parse for valid C code.
5. Compile new light pattern.
6. If the code does not compile, there is a function createNewPromptFromBadCode to use ChatGPT to amend such bad code : the function takes the error message from the Arduino compiler and sends this to ChatGPT with the "bad" code and we prompt ChatGPT to correct the bad code so it compiles. If we again get "bad" code, we abort.
7. We program the Arduino using the valid code from step 5/6.

Download:
golang (tested with 1.21.5 on Windows)

Arduino CLI

Install the go-openai library (provides unofficial Go clients for OpenAI API):

go get github.com/sashabaranov/go-openai

Used library Version Path
FastLED 3.6.0 C:\Users\jusher\Documents\Arduino\libraries\FastLED

Used platform Version Path
arduino:avr 1.8.6 C:\Users\jusher\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6

• Pay $ for ChatGPT-API credit and store your private key somewhere eg ...//privateChatGPTAPIkey.txt
  

• We use ChatGPT4 because it gives better code responses than ChatGPT3.5 Turbo (less errors in the C code). For a typical prompt+response it costs 1-4 cents.

• Install Arduino libraries Used library Version Path
  FastLED 3.6.0 C:\Users\jusher\Documents\Arduino\libraries\FastLED
  

• Used platform Version Path
  arduino:avr 1.8.6 C:\Users\jusher\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6
  

duinoHelp.txt

• It take typically 30-50 seconds to receive the response using ChatGPT4.
• Some colours look bad with RGB LEDS- e.g. brown and yellow, and white if no white LED.
  
• It costs 1-4 cents for each prompt using ChatGPT4. The WhisperAPI is <0.1C per ~10 second prompt. I plan to have a "low res" version that uses ChatGPT3.5 Turbo and minimizes the response size (number of tokens) e.g. by removing comments.
  
• The winmm.dll is pretty janky and about 1 in 10 times the microphone recording will not save: need to switch to using core-audio (for Linux compatability).
• Integrate other sensors onto the Arduino, e.g. microphone or accelerometer to sync. pattern with sound.

• Nimrod Gileadi for the idea of including example code in the response and encouragement: https://github.com/nimrod-gileadi
  Some of the prompt engineering suffix (the "response rules") were taken from a paper he worked on (Wenhao Yu, Nimrod Gileadi, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647, 2023).

• Siggy for golang help: https://sig.gy/

• Support the Arduino project (try to buy at least one legit product!): https://www.arduino.cc/