I have had the opposite experience. For complex tasks, LLMs fail in subtle ways which requires inspection of its output: essentially “declarative” to “imperative” is bug ridden.
My trick has been to create DSLs (I call them primitives) that are loaded as context before I make declarative incantations to the LLM.
These micro-languages reduce the error space dramatically and allow for more user-friendly and high-level interactions with the LLM.
TFA covers this (I think, it got real jargony at times):
>Declarative processing of configurations generated via AI is a way to ground the AI, this requires a lot of work since you don't just offload requests to an AI but rather your processing logic serves as a guardrail to ensure what's being done makes sense. In order for AI to be used in applications that require reliability, this work will need to be done.
When I was playing around with AI for data analysis last year, the best results for me came from something like this but more on the imperative side: RAG with snippets of R code. My first attempt was taking snippets directly from existing scripts and adding comments to explain the purpose and caveats. That didn't work well until I put in a lot of effort replacing the "common knowledge" parts with variables or functions. For example, no more `sex = 1`, but `sex = "male"`. Common tasks across scripts were refactored into one or a few functions with a couple parameters, and then placed in a package. The threshold for applying the DRY principle was lowered.
In the end, I decided a custom solution wasn't worth the effort. The data had identifying details of people, so any generated code would have to be checked and run by analysts who already had access to the data. But the process of refactoring stuff into descriptively-named objects was such a big benefit, the AI code wasn't doing enough to justify the effort. Again, this was using a custom system made by a total ML noob (myself) with GPT 3.5. The execs banned usage of LLMs until they could deal with the policy and privacy concerns, so I don't know what's possible these days.
Bingo!!! I use this approach for data science tasks today. Create very specific DSL that has mathematics, set theory, etc. as a context and setup your data science exploration using the dsl as input. Been fairly decent so far. It works for two specific reasons
1. I have a fairly specific dsl that is expressive enough for the task at hand available easily (math notations have been around for centuries now and algorithms for at least 1/2 a century).
2. I use Apache spark - so everything around parallelizing and synchronizing I don’t handle in the code myself (not most of the time).
This is a fascinating topic and something I'm looking into these days, specifically removing need for Data Scientists to write Spark code. Would be great if you can share more details around the DSL. The DSL also sounds interesting in it's own right!
This is a very interesting use case of LLM and something I'm looking into these days. Would appreciate if you could share more details on the challenges you ran into in using DSL's with LLM's and how you solved them?
I have had the opposite experience. For complex tasks, LLMs fail in subtle ways which requires inspection of its output: essentially “declarative” to “imperative” is bug ridden.
My trick has been to create DSLs (I call them primitives) that are loaded as context before I make declarative incantations to the LLM.
These micro-languages reduce the error space dramatically and allow for more user-friendly and high-level interactions with the LLM.