Hoping to finish a work project soon then I’ll get time to finish my board.
I’m sure many of you have heard of chat-gpt by now. I use it quite a bit like an assistant and also like Google. Free. Highly recommend.
I thought about using that to answer questions/dialogue instead of the tools I’m using now but I’ve decided not to. The main reasons are chatgpt can’t connect to the internet and doesn’t know things past 2021. Whereas my current solution does and with low latency.
Though I did find a new text-to-speech (TTS) AI tool among all the AI hype right now that makes the voice hyperrealistic and exactly like Jarvis! Like it will sound like a real person is inside my board and I think that’ll make a noticeable difference in the experience to where I’m going to try to implement it VS rn I have a british voice that’s robotic enough like Siri/Alexa/google assistant where it is clearly computer generated, refer to my 30s demo above.
Ok, a bit of an issue, this new TTS API is sometimes slow. Like 1/10 or 1/5 requests take 5-10s to hear something back. I tried using a timeout and retrying to circumvent it but it didn’t really help. The old robotic voice was flawless in network performance. Just like voice assistants in your phone, it replies right away. I do very much prefer this voice. I’m thinking I could play a “hmm” track and retry once in the uncommon situation when it happens or make a fallback if the retry times out like have it say “I’m busy right now” or “sorry I can’t help you right now” so I’m not left waiting or wondering what’s going on.
The hardest thing is always going back to this project after a long break. It’s like I have to re-learn what I did and it’s high stakes if I brake something. Right now I’m filling in the last gaps in my Arduino and raspberry pi code, which is challenging for situations that involve interactions with many parts of the different systems. There is a complicated web of software systems in place so I’m writing it out for my own mental clarity and I thought it might be insightful to share:
Arduino:
Accessory/Second Remote
Detect remote’s buttons’ presses (A/B/C/D) and length of press.
Evaluate which function to perform based on short/long press and if menu is 1 or 2.
Write HIGH/LOW to switch via digital pin to toggle on/off.
Send update to touch screen to sync system change via remote.
Send play sound/audio request to raspberry pi.
Security System
Write to BLE module to scan for my presence.
Read results from BLE scan to evaluate if I’m present.
Evaluate security based on presence.
Keep track of lack of presence and trigger series of alarms by sending play sound/audio request to raspberry pi.
Touch Screen
Read commands from touch screen.
Play touch screen press sound/switch sound by sending play sound/audio request to raspberry pi.
Write HIGH/LOW to switch via digital pin to toggle on/off.
Toggle security system.
VESC Reader
Detect sequence of presence of data from VESC to trigger play power on/off sound request to raspberry pi.
Process VESC data and keep track of skateboard metrics/analytics (dist travelled, speed, battery, etc).
Evaluate events like new speed record, dist travelled, battery low.
Send play sound/audio request to raspberry pi to notify about the events.
Continuously update volume depending on speed (can’t hear at 20km/h at same volume level as rest) - send request to pi to update volume.
Raspberry pi:
Input
Continuously stream in audio from mic and detect for wakeword.
If wakeword was said, tell Arduino to pause music. Play listening sound.
Listen to audio and save audio input request. Playing thinking sound if something was said or play sleeping sound if nothing said after being woken up.
Response
Respond to audio input request by sending request to Cloud AI bot. Monitor request status and play sound to notify if request failed. Evaluate cases for whether something new should be said back.
Send Cloud AI response/answer to AI voice generator. Monitor request status and play sound to notify if request failed. Play audio response from voice AI (nice voice) or Cloud AI (computer voice) depending on situation. Based on Cloud AI answer, send message to Arduino to ex. toggle a switch.
Relisten for Cloud AI answer provides that conversation isn’t over. Or tell Arduino to resume music.
More
Listen for commands from Arduino and evaluate which sound to play (if any).
When playing voice, tell Arduino to turn on/off arc reactor at start/stop of talking.
Touch screen:
Send updates to Arduino of local state based on touch screen button presses.
Cloud AI Software:
Use AI to process input, detect intent, send back text+audio response to raspberry pi request.
For more involved fulfillment that requires Webhook:
Read variables from intent and process them.
Interact with Google Calendar API when asked to add event if space is open in my Calendar, or deny.
Send recieve and process various APIs to look up knowledge on the internet using different methods and store situation/number of answers by modifying conversation context in Cloud AI.
Send out new response to Pi. Reclassify/add context to conversation. Store additional responses and keep track of them.
…and then theres the complicated web of hardware systems
“Hmm” triggered when my AI assistant request is taking longer than 3s
“Hmm let me think” is when the AI voice generator takes longer than 3s
The speech recognition sometimes has trouble with one word answers and thinks nothing was said. Idk if I can or will tweak it. I’ll just say yes please instead of yes.
It doesn’t hear me too well when music is loud but does work okay when music low. Thus I added the ability to trigger it to pause music and listen with the remote and the touch screen.
Now I need to add the hydrophobic membrane material under those caps to cover those holes for the mic to seal it off.
Currently using an external power supply on my table because there is noise over the step down converter in my board being supplied to my pi and transmitted through Jarvis’ speakers and the voltage isolation module still hasn’t come yet, usually only takes 3 weeks but it’s been 5… I won’t ride it till I have that noise fixed because it’s annoying. Been 2 weeks since it supposedly cleared customs, usually gets to me in a few days after that. Don’t have tracking past origin country… At least there’s still snow to melt outside and it’s a little chilly
Fixed this and added more sliders for wakeword and other assistant sensitivity settings to my touch screen’s settings screen.
It came and it’s exactly what I needed! All the noise on my Jarvis speakers is gone now and now it’s perfect!
I added two separate layers of the membrane and it’s perfect! It has no impact on the mic’s ability to hear my voice! @MysticalDork thank you for suggesting this material!
One small annoyance is with bass notes on my music speakers, theres a rattle. But not when I lift the lid. I have not observed it till this week. Something is vibrating but it’s not directly against the lid, I tried moving things around and a taller spacer between lid and box but that didn’t change anything. The vibration seems to be travelling through to the rest of the box. It’s not noticeable on all songs but still it takes away from that crystal clear experience. Perhaps there’s a lost screw or something sitting at the bottom? I didn’t see anything last time I checked.
Yeah no idea about the tubing which isn’t included but in the instructions they say you can find everything in home Depot. Never done something like this (just like everything else in this build )
VS rn I have a british voice that’s robotic enough like Siri/Alexa/google assistant where it is clearly computer generated, refer to my 30s demo above.
not planning on anything new from here for a while. Just need to put everything together and make sure it all works 
(just like everything else in this build 
