I'd say the elephant in the room is graduating beyond plaintext (projectional editor, model-based editor).
If you think about it so many of our problems are a direct result of representing software as a bunch of files and folders with plaintext.
Our "fancy" editors and "intellisense" only goes so far.
Language evolution is slowed down because syntax is fragile and parsing is hard.
A "software as data model" approach takes a lot of that away.
You can cut down so much boilerplate and noise because you can have certain behaviours and attributes of the software be hidden from immediate view or condensed down into a colour or an icon.
Plaintext forces you to have a visually distracting element in front of you for every little thing. So as a result you end up with obscure characters and generally noisy code.
If your software is always in a rich data model format your editor can show you different views of it depending on the context.
So how you view your software when you are in "debug mode" could be wildly different from how you view it in "documentation mode" or "development mode".
You can also pull things from arbitrarily places into a single view at will.
Thinking of software as "bunch of files stored in folders" comes with a lot baggage and a lot of assumptions. It inherently biases how you organise things. And it forces you to do things that are not always in your interest. For example you may be "forced" to break things into smaller pieces more than you would like because things get visually too distracting or the file gets too big.
All of that stuff are arbitrary side effects of this ancient view of software that will immediately go away as soon as you treat AND ALWAYS KEEP your software as a rich data model.
Hell all of the problems with parsing text and ambiguity in sytnax and so on will also disappear.
This claim is often repeated, but I haven't seen it substantiated even once. It's possible that no one has yet come up with the right answer that "obviously there". But it is also possible that this claim is not true, and a tend to believe the latter more as time passes.
Every attempt that I've seen, e.g. Lamdu, Subtext, any "visual app builder", all fail miserably at delivering ANY benefit except for extremely simple programs -- while at the same time, taking away most of the useful tools we already have like "grep", "diff", etc. Sure, they can be re-implemented in the "rich data model", perhaps even better than their textual ancestors - but the thing is, that they HAVE to be re-implemented, independently for each such "rich data model", or you can't have their functionality at all -- whereas as 1972 "diff" implementation is still useful for 2017 "pony", a language with textual representation.
regarding your example, the "breaking things into smaller pieces" was solved long ago by folding editors (I used one on an IBM mainframe in 1990, I suspect Emacs already had it at the same time, it did for sure in 1996).
the problems with "parsing and ambiguity" are self inflicted, independent of whether the representation is textual. Lisp has no ambiguity, Q (the K syntax sugar) has no ambiguity. Both languages eschew operator precedence, by the way, because THAT is the real issue that underlies modern syntax ambiguities.
I've been waiting for that amazing "software as a data model" approach to show a benefit for almost 30 years now (There's been an attempt nearly every year I looked). Where it has (e.g. Lisp, Forth), it's completely orthogonal to the textual representation.
One advantage I dream of with Lamdu is snapshotting application state and having values annotated in the boxes next to their definition. You could edit the code in the snapshot and see how it propagates through pure parts of the program. It if works, you'd apply the change to the running system.
How do you do this with a text driven program? Typically, there are so many files and unspecified build systems, so you can't view your program as a proper tree as in Lamdu. You can't even treat compilation as a pure function. You can't take a snapshot of an arbitrary program because effects are not isolated. You might not even be able to set breakpoints without breaking the program. If it's machine code you might not even have debug symbols. It's such a mess, and it's different on every platform.
So you claim there is no benefit to be had. I claim the existing solutions are already failing. If one solution gets enough of these things right, there is no reason why it wouldn't succeed just because text-driven. (But it is also possible to do it correctly with text)
If the program code is always to be consistent, it cannot be text, because text allows you to input invalid programs. So how is text not a leaky abstraction?
Also, I had a chance to use time traveling virtual machine (which works independently of how your code was produced) in 2007, and while I haven't had a chance to use them, I know that source-aware time traveling debuggers have made huge strides recently.
> So you claim there is no benefit to be had. I claim the existing solutions are already failing.
These claims are not contradictory. I agree that existing solutions are failing in various ways, but I still disagree that the "code as rich data" would provide benefit.
> If one solution gets enough of these things right, there is no reason why it wouldn't succeed just because text-driven. (But it is also possible to do it correctly with text)
So, if possible to do correctly when text driven, why is text driven a problem?
> If the program code is always to be consistent, it cannot be text, because text allows you to input invalid programs. So how is text not a leaky abstraction?
All programming abstractions are leaky in one way or another, and text is no different. In my opinion, the leak that it is possible, during design time, to have an invalid program (which will be rejected as such prior to execution) is not a real problem. The requirement that "program code always be consistent" is onerous, and, in my opinion, harmful.
When I write code, I often start with sketches that cannot work and cannot compile while I think things through, and then mold them into working code. Think about it as converting comments to code. If an editor didn't let me do that (and likely wouldn't provide any help, because I'm writing comment prose and not "rich data" code ...), I'd open a text editor to write my plans. text editor.
I will restate my belief in a different way: Any benefit from "code as data" that cannot be applied to a textual representation, only ever manifests in extremely simple cases. I offer as support for this belief, the history of visual and rich data tools over the last 30 years, none of which manages to scale as well as text based processes, most of them scaling significantly worse (which is often explained away by critical mass, but there are enough successful niche tools that I think this explanation is unacceptable).
Oh, Lisp is an interesting point. Given that almost all Lisp files on disk are well-formed S-expressions, are there tools for doing diff or merge on the S-expression level instead of the text level? Do any Lisp (elisp, Clojure, etc.) programmers use them in their day-to-day work?
I agree with you for finished programs, but most of the time when I'm coding, I'm working on unfinished programs. I find it infuriating enough when I'm using some "smart" text editor that adds closing quotation marks or braces when I'm not ready for them yet. I know I need to type them, they're buffered somewhere in my wrist, they're going to get typed well before my eye realizes that the smart editor has typed them for me and gets around to telling my fingers.
What is the state of the art in editing software that's continuously stored in a syntax tree instead of in unparsed text? (I say "continuously" because I assume that letting me save a file that doesn't parse, even temporarily, breaks most of these benefits.) How do you, for instance, represent merge conflicts, and how do I resolve them?
This is completely new to me so maybe there are great answers here that I just haven't heard of.
In my experience, this only works in DSLs where a more graphical representation fits a particular problem space better. One good example is Simulink in control systems work; this to me is the real 'killer' part of Matlab, the ability to lay out PID controllers in a way more natural for control engineers to design/analyze, and then with a push of a button autogenerate C/C++ code to use on hardware. But even then the lack of things like diff, merge, etc. are continual reminders that there are disadvantages as well...
That is totally true and it expands the programming possibilities...
But... it make debugging harder without even more tooling. And debugging is most of the time spent in software... which probably explain the origin of the problem.
Especially when you need to reverse engineer code provided by a vendor.
Broadly speaking, I agree with beagle3's point (probably above this comment). But the thing that keeps me from being willing to quite commit to it is that it is generally a bit unfair to expect someone to produce this alternative programming paradigm and immediately be better than the previous one, which may or may not be broken or better or worse but has certainly had five or six orders of magnitude more time poured into it.
If it's truly the "silver bullet" I'd still say I expect to see some evidence of that in some domain relatively quickly, because it is supposed to be the silver bullet after all. But it would take time to discover that.
If you think about it so many of our problems are a direct result of representing software as a bunch of files and folders with plaintext.
Our "fancy" editors and "intellisense" only goes so far.
Language evolution is slowed down because syntax is fragile and parsing is hard.
A "software as data model" approach takes a lot of that away.
You can cut down so much boilerplate and noise because you can have certain behaviours and attributes of the software be hidden from immediate view or condensed down into a colour or an icon.
Plaintext forces you to have a visually distracting element in front of you for every little thing. So as a result you end up with obscure characters and generally noisy code.
If your software is always in a rich data model format your editor can show you different views of it depending on the context.
So how you view your software when you are in "debug mode" could be wildly different from how you view it in "documentation mode" or "development mode".
You can also pull things from arbitrarily places into a single view at will.
Thinking of software as "bunch of files stored in folders" comes with a lot baggage and a lot of assumptions. It inherently biases how you organise things. And it forces you to do things that are not always in your interest. For example you may be "forced" to break things into smaller pieces more than you would like because things get visually too distracting or the file gets too big.
All of that stuff are arbitrary side effects of this ancient view of software that will immediately go away as soon as you treat AND ALWAYS KEEP your software as a rich data model.
Hell all of the problems with parsing text and ambiguity in sytnax and so on will also disappear.