Interesting. Reminds me of my time at university using something called Progol. You gave it positive and negative cases and it's algorithm would (very slowly) try and come up with Prolog code that fit it.
It was an interesting idea but in reality we spent so much time having to spoon-feed it more and more carefully constructed examples that it just seemed pointless.
Not to say that this will suffer from the same issue, but I do wonder if the benefit will really be significant enough to make it worth it.
Indeed, there is rich prior work on example based program synthesis. Often these use inductive logic programming or aided by SMT solvers if these examples can be lifted to solver level values (think integers or booleans, or data types made from those).
However, not all program values can be easily lifted to solvers. So the goal with RbSyn is to allow the programmer to write the same tests they would have written anyway to check their program correctness, without any extra fiddling. As tests subsume examples, we expect more program behavior can be specified than just using simple examples.
> I do wonder if the benefit will really be significant enough to make it worth it.
I have often pondered on what makes a program synthesis tool useful. I expect writing code as art as much as it is science, and people like to express code in the way they model system behavior in their head. In that regard, I do not think program synthesis tools will enable you to automate away large parts of code, but I do think it will automate mundane parts of a program; like writing utility methods, or filling in a partial program when enough information can be gathered from surrounding context (such as the arguments you write for a substring function).
I'd love to give this a try next time I'm working on a Ruby codebase. In the meantime, I'm curious to learn more about programming by example. I found [this Microsoft paper on the subject](https://www.microsoft.com/en-us/research/wp-content/uploads/...) and have started reading it, but I'm curious if there are any reading materials you would recommend.
Specifically, I've been working on a activity tracking app that can take plain text and create structured data from it. For example
> I played tennis with Kevin and won 6-2
would turn into
```
type: tennis
opponent: Kevin
my_score: 6
their_score: 2
outcome: win
```
As you can imagine, when it comes to natural language there are many variations of I won (outcome: win), he/she/they won (outcome: loss), I was beat, etc which means that my inputs are not nearly as structured as the examples used in the paper I'm reading. Given that I'm not trying to cover every use case in the English language, which I assume would require some form of NLP, but rather just a subset that a single user would input, do you think think it would be possible to solve this with PBE techniques?
Natural language is ambiguous, so I do not think any programming by example methods will work well here. I think if you are working with a limited vocabulary in the set of sentences you want to parse, it may be fine to roll out a handmade solution. For general english, there is work in intent identification in NLP that can give you structured intents from raw english text, which can then be lifted to make programs as a secondary step.
It was an interesting idea but in reality we spent so much time having to spoon-feed it more and more carefully constructed examples that it just seemed pointless.
Not to say that this will suffer from the same issue, but I do wonder if the benefit will really be significant enough to make it worth it.