Hi, and thanks for taking the time to check out Rivet.
My bias.
AI/ML usage in recent years is a very hot topic and I feel that it would be best to give the perspective of why this repository is what it is.
If your stance is simply that you wish not to consume/involve yourself with any artifacts produced by any amount of machine learning - that is a respectable approach and I commend you on being true to your beliefs. Not many people have a spine and that in itself is highly respectable - you are under no obligation to continue reading or use this repository.
For those that are overly enthusiastic and do not understand the outputs of their prompt engineering, you probably have willfully neglected your capability to understand the production and are likely too marveled by its principles to understand the innate nature of non deterministic systems and how models at their fundamentals work - that is my own personal analysis as a consultant for a few boutique cloud engineering firms that I have gathered working with tens of clients and engineers who are told to leverage this under the guise of productivity.
My justification on the other hand as someone who understands the technology, is that I believe it can be used in a carefully curated manner with reasonable limitations understood because I am moderately familiar with how everything works conceptually, down to the transformer and tokenization implementations.
Simply put: Someone who has been writing code and reading bulletin boards for the better part of two decades using ML assisted code is not the same as a new engineer who can't articulate the differences between high level programming and static types leveraging an LLM to generate slop.
The backstory and my previous discovery.
Seven years ago one of my first employers who took a chance on my career, specifically was working in the adult industry working on educational material. I won't go into fine detail here but something since the days of Tumblr making some pretty notable policy changes on what picture material was against their terms of services there has been a lot of uproar in specifically making alternative platforms.
Since those demands, I have found myself to be intimately involved with discovering how hardware acceleration works with various other vendored solutions such as ffmpeg and gstreamer, in fact, if you search the internet carefully, you can see that I've been searching about codec capabilities for the better part of 6+ years.
- https://www.reddit.com/r/ios/comments/nutw7e/is_vp9_support_on_the_horizon/
- https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3015#note_2100853
Since this time to this very day, I have focused my free time very intensely in this field to the point where I feel that I have a decent educated understanding and have a intermediate resemblance of what's going on, and possess the capability to articulate concepts and problems confidently.
My failure in implementations and gained expertise first hand.
(building off the previous heading)
Specifically with all this knowledge that I have gathered, I found ourselves making a queue system specifically that relied on system calls like exec and shell_exec in PHP, real crude stuff, to perform invocations of string-concatenated FFMPEG commands to convert user content.
Eventually, with us running into bottlenecks and trying to avoid patent problems, you can quite imagine with heterogeneous hardware configurations such as different GPU vendors on how one configuration might work on one machine but not another depending on part/generational availability, even so in extreme circumstances like the pandemic of 2020. This was the crux that spawned several re-writes and needing more finely curated wrapper services.
Before this problem of needing something more native and closer to the metal arise, we even tried with python and had something close albeit limited to system calls now to interrogate /proc to see what devices were registered.
It still wasn't enough because there simply wasn't a standard that we could leverage and VAAPI wasn't where we needed it at the time. Nvidia cards needed cuda specific handles, Drivers were complicated, libraries sucked. To be quite honest the only thing that actually worked after a while was a newer linux kernel and native Intel Arc support that we could leverage. We started buying a ton of Arc GPUs and coped for a while.
So proprietary rust rewrites started... (pictured: video transcoder microservice of said organization)
Deciding the world needs something greater (fuck it we ball, let's make the source public)
After a while, I realized that there isn't really anything this able on the internet publically avaialble today. It was time that we had something that wasn't the monolith that ffmpeg was but as a service.
Additionally, cramming this support together into a single serivce and not using a library was a painful firsthand lesson for me, not having isolated components to segregate service fabric integration to cloud providers, and the actual logistical work unit being well defined.
This is ultimately what sparked "Rivet".
So, you got some chops, what is "slop" in this case?
To be quite honest, it is subjective.
Is it slop in the sense that there's a decent amount of AI/ML involvement and written code? => Sure, you can call it that.
Is it slop in the sense that there is no human oversight? => Not at all. There were functional purposes and intents behind the design spanning years. all FFIs have been hand tested and benchmarked.
Is it slop in the sense that the human operator overseeing this doesn't have a technical understanding of what's going on? => Absolutely not, the point of this issue/article is to tell you this isn't /goal on a local claude instance being left to go AWOL.
You're welcome to call it what you like, it's your <insert deity here> given right and I'm not going to stop you. But it would be unfair to say that this project is a total slop for the sole purpose that I have dumped a quarter of my life of trial, error, research and development to get to write this message today.
Why is documentation partially AI written?
To be honest as an autistic and ADHD person I struggle to articulate my thoughts clearly within the confines of the English language. It is my spoken tongue but sometimes things don't feel right. It is easier for me to articulate through refined output.
Hi, and thanks for taking the time to check out Rivet.
My bias.
AI/ML usage in recent years is a very hot topic and I feel that it would be best to give the perspective of why this repository is what it is.
If your stance is simply that you wish not to consume/involve yourself with any artifacts produced by any amount of machine learning - that is a respectable approach and I commend you on being true to your beliefs. Not many people have a spine and that in itself is highly respectable - you are under no obligation to continue reading or use this repository.
For those that are overly enthusiastic and do not understand the outputs of their prompt engineering, you probably have willfully neglected your capability to understand the production and are likely too marveled by its principles to understand the innate nature of non deterministic systems and how models at their fundamentals work - that is my own personal analysis as a consultant for a few boutique cloud engineering firms that I have gathered working with tens of clients and engineers who are told to leverage this under the guise of productivity.
My justification on the other hand as someone who understands the technology, is that I believe it can be used in a carefully curated manner with reasonable limitations understood because I am moderately familiar with how everything works conceptually, down to the transformer and tokenization implementations.
Simply put: Someone who has been writing code and reading bulletin boards for the better part of two decades using ML assisted code is not the same as a new engineer who can't articulate the differences between high level programming and static types leveraging an LLM to generate slop.
The backstory and my previous discovery.
Seven years ago one of my first employers who took a chance on my career, specifically was working in the adult industry working on educational material. I won't go into fine detail here but something since the days of Tumblr making some pretty notable policy changes on what picture material was against their terms of services there has been a lot of uproar in specifically making alternative platforms.
Since those demands, I have found myself to be intimately involved with discovering how hardware acceleration works with various other vendored solutions such as
ffmpegandgstreamer, in fact, if you search the internet carefully, you can see that I've been searching about codec capabilities for the better part of 6+ years.Since this time to this very day, I have focused my free time very intensely in this field to the point where I feel that I have a decent educated understanding and have a intermediate resemblance of what's going on, and possess the capability to articulate concepts and problems confidently.
My failure in implementations and gained expertise first hand.
(building off the previous heading)
Specifically with all this knowledge that I have gathered, I found ourselves making a queue system specifically that relied on system calls like
execandshell_execin PHP, real crude stuff, to perform invocations of string-concatenated FFMPEG commands to convert user content.Eventually, with us running into bottlenecks and trying to avoid patent problems, you can quite imagine with heterogeneous hardware configurations such as different GPU vendors on how one configuration might work on one machine but not another depending on part/generational availability, even so in extreme circumstances like the pandemic of 2020. This was the crux that spawned several re-writes and needing more finely curated wrapper services.
Before this problem of needing something more native and closer to the metal arise, we even tried with python and had something close albeit limited to system calls now to interrogate
/procto see what devices were registered.It still wasn't enough because there simply wasn't a standard that we could leverage and VAAPI wasn't where we needed it at the time. Nvidia cards needed cuda specific handles, Drivers were complicated, libraries sucked. To be quite honest the only thing that actually worked after a while was a newer linux kernel and native Intel Arc support that we could leverage. We started buying a ton of Arc GPUs and coped for a while.
So proprietary rust rewrites started... (pictured: video transcoder microservice of said organization)
Deciding the world needs something greater (fuck it we ball, let's make the source public)
After a while, I realized that there isn't really anything this able on the internet publically avaialble today. It was time that we had something that wasn't the monolith that ffmpeg was but as a service.
Additionally, cramming this support together into a single serivce and not using a library was a painful firsthand lesson for me, not having isolated components to segregate service fabric integration to cloud providers, and the actual logistical work unit being well defined.
This is ultimately what sparked "Rivet".
So, you got some chops, what is "slop" in this case?
To be quite honest, it is subjective.
Is it slop in the sense that there's a decent amount of AI/ML involvement and written code? => Sure, you can call it that.
Is it slop in the sense that there is no human oversight? => Not at all. There were functional purposes and intents behind the design spanning years. all FFIs have been hand tested and benchmarked.
Is it slop in the sense that the human operator overseeing this doesn't have a technical understanding of what's going on? => Absolutely not, the point of this issue/article is to tell you this isn't /goal on a local claude instance being left to go AWOL.
You're welcome to call it what you like, it's your <insert deity here> given right and I'm not going to stop you. But it would be unfair to say that this project is a total slop for the sole purpose that I have dumped a quarter of my life of trial, error, research and development to get to write this message today.
Why is documentation partially AI written?
To be honest as an autistic and ADHD person I struggle to articulate my thoughts clearly within the confines of the English language. It is my spoken tongue but sometimes things don't feel right. It is easier for me to articulate through refined output.