I present the best thing I’ve ever made.
My magnum opus;
The product of weeks of nonstop work, and a culmination of years of esk8 riding, board building, and TITS development.
Yes, this race happened 6 weeks ago. shush.
the long journey that was producing this video
From the moment I started TITS, I aspired to overlay data onto video. I had very high expectations/requirements, and wouldn’t accept any compromises:
- Had to work with my unique data (obviously)
- Tire temperatures are represented by the changing colors
- Bar graphs for throttle/speed/etc.
- Data shown in numeric text format
- Hardware accelerated video encoding
- Every frame gets new data plotted (no sharing data graphics between frames!). TITS records a lot of data, I want to use all of it!
The Dead Ends
Multiple times, I had to scrap days of effort and go back to the drawing board with a different approach.
Matlab
I started by extending my existing Matlab data parsing script. I was able to import the video as frame bitmaps, then draw graphics elements on top of each bitmap frame.
However, it quickly became apparent that
- Manually drawing pixels would require coding custom drawing functions for text, graphics, shapes, etc.
- Adding transparency to shapes would be possible, but difficult
- Because I was doing everything in strict 8-bit RGB, the quality wasn’t great.
- Everything was running on my CPU (no hardware encoding), so each frame took very long to draw and export. A full render could take weeks.
- Adding GPU acceleration would require code refactoring, and increase the complexity substantially. Not to mention requiring a $50 toolbox package.
It was quickly clear this wasn’t the path forward, so started looking at other options.
Python
Python has always been my Achilles heel.
I’ve failed to become proficient in it 4~5 separate times. I wanted to change that, and I thought this video could become a reason to become passably proficient.
However, Python is undeniably powerful. I totally expected there to be an open source library that would do what I wanted to do… but I couldn’t find one. I tried 3~4 libraries over several days, but I hit weird roadblocks with each one.
Part of me wanted to keep trying to get it to work, as python seems like the “most right” way to do it? But I didn’t think I was capable of getting it to work.
Joe Barnard captures this sentiment fantastically here:
https://youtu.be/4jgTCayWlwc?si=KgTda5t6Jjw8_Dn0&t=735
Times I’ve been bested by python… +1
Dashware
I was really hopeful for this one!
On the box, it should be able to do exactly what I want it to do - overlay data onto action camera footage!
It was really promising, but I got as far as importing my data into the program before realizing it wasn’t the solution. Death by thousand cuts:
- Dashware is abandonware, last updated in 2017. It’s likely to be even more broken in the near-ish future.
- Random Crashes
- Clunky UI required a lot of clicks to do anything. Configuring a custom data importer and custom UI would take days, maybe weeks, of tedious data entry, and would be hard to adjust in the future as I expand TITS.
- Color changing tire graphics might be possible? Would likely be hacky, and/or take a lot of effort to get working.
- Multiple frames shared the same data. It was only refreshed at ~10Hz iirc
- Every time I exported, The video was flipped upside down (with the graphics still upright!). This happened regardless of how the video exporter was configured.
Not the clean solution I was looking for, unfortunately
Telemetry Overlay
Telemetry Overlay is an absurdly expensive($200) program for overlaying Camera data onto action cam footage.
However, it doesn’t support custom data files, data types, or graphics elements. Easy reject.
All of these failed attempts pointed me to needing a powerful video suite to fully realize my vision.
Davinci Resolve
is an absurdly powerful, fully free video editing suite. It comes with Fusion, their node-based video editor
Vonk Ultra is a 3rd party plugin for Fusion, which adds numerous nodes for doing math and drawing animations. Most notably, it supports animating graphical elements with data.
It also can pull that data from a JSON file!
Always wanted to get good at video editing, this was a great excuse to dive into the deep end.
A solution, at last
Some additional code in my Matlab data parsing script exports data as a JSON. This JSON file is ingested by my fusion composition, which then draws the graphical elements.
Technical Weeds
- To make Fusion’s job easier, I resample/downsample the raw data in Matlab to match the framerate of video (in this case, 50Hz) with linear interpolation. That way, each line of data in the JSON file is one frame’s worth of information.
- This greatly simplifies the logic that happens in Fusion, as the data from
- How did I sync the data to the video? I did a few quick throttle punches after starting the camera. This produces very quick ppm spikes in the data, which I can use as a “clapperboard” to sync it with the video.
This composition came together relatively quickly, only taking a few evenings of work to get the basics laid out (including learning!). Early prototypes were haphazardly laid out, prioritizing demonstrating a proof of concept before I invested time in making everything pretty.
note: resolve's benefit over python?
Using a full-suite video editor provided a UI to effortlessly place elements, make shapes translucent, rotate shapes/text, add transitions, etc… with real-time feedback for how it looks. I couldn’t imagine defining the positions and properties of the UI with just code. Would’ve taken ages to look half decent.
The actual video timeline is pretty simple (Fusion Composition in pink)
Music
I wanted something suspenseful, building, and flowy. I almost went with Mombasa, but the vibe was too serious.
Ended up going with
Worakls - Detached Motion
Worakls - Caprice
Tension building like Mombasa, but some funk sillyness thrown in
Rendering
Proved to be an entire project in itself. A week of troubleshooting!
I’m so far outside Vonk Ultra’s intended use case of abstract 3D animations, Resolve crashed when I tried to render. My hard drive was filling up with 100s of GB of temp files… causing the crash before even being a few minutes in.
Projecting how much space would be required, it would be more temp files then the size of my C drive!
and I REFUSE to solve this by reinstall my windows onto a bigger drive, that’s just too dumb (even for me)
Oh, the Joys of a unique use case!
I was entirely unable to find resources online for this problem.
If I could change the location of that cache to another drive, that would be okay? But that specific cache… just wasn’t an option to configure in the Resolve Settings. I burned a lot of time learning about how Resolve Caching works (even the 4,234 page resolve manual provided little help)
Turns out there’s an entirely separate settings menu for fusion caches. Fortunately, I got a better solution before I figured that out…
A German Savior
I put up a flag for help on the Vonk Ultra Forums. Local moderator Dunn graciously offered his time to help debug.
Massive, massive thanks to him. He ended fixing the issue with a custom Lua import script, made custom for this project. I needed to made moderate modifications from his initial version to add/change what/how data was processed, but the overall framework remained untouched.
I couldn’t have done this myself (lack of familiarity with Lua/Fusion/Vonk), Dunn made this video possible. The custom script reduced the amount of temp rendering files from 1000GB+ to <1GB, and increased my rendering speed 220% 
Sharing
When I first uploaded the 1080p export to YouTube, the quality was dogshit. The video was blurry and the text unreadable. Unwatchable. Original (left) vs YouTube (right)
Predictably, YouTube compression *really* doesn't like how much stuff was changing each frame.
Basically, I inadvertently created a torture test for video compression. Probably why every other piece of software out there doesn’t produce new data graphics for every single frame 
YouTube reencodes uploaded video to a lower bitrate when you upload it, somewhere around ~12Mbps for 1080p.
Resolve exported with an average bitrate 288Mbps to display everything…that’s 24x higher!
So it makes sense the YouTube version would look terrible.
I needed more bitrate. YouTube gives higher bitrates to higher resolution videos (~45Mbps for 4k), so I re-rendered the video at 4k resolution.
Much better! Original (left) vs YouTube 4k (right)
Even 1080p quality was better then before!
… And that’s why it took me a month and a half to edit this video.
Hope you enjoyed it!