(1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'.
The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation.
(2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN:
True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around.
(And yes, QSN is a regular language [1])
(3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV.
(4) Although you also don't have to decode it for it to be useful.
1. I can use wc -l on a stream of QSN strings
2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0'
It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text.
(5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode.
-----
I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it.
> (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'.
The tokens in question are filenames, which are (and will continue to be) allowed to contain newline characters. Sorry that wasn't clear. This was never meant to be a critique of QSN, I'm just saying QSN is unnecessary and unnecessarily complex for between-process communication:
- In case of filenames, just use NUL. It already works, is supported by a bunch of tools, and will never have the overhead of encoding/decoding.
- In case of arbitrary binary streams, just send the stream unaltered. Why encode/decode it? What benefit is that? In the very rare case that I'm actually manually inspecting a stream of bytes (debugger/printf) there's always some easy way to encode those bytes for easy readability such as `printf '%q\n'`.
> (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN:
It would take me at least a day to write enough test cases to ensure that that regex in fact does what it says. But the point is moot: encoding and decoding should only happen at the interface with a human, not with another process.
> (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV.
I couldn't comment; I don't use awk unless I absolutely have to. Once I need to use awk a shell script is no longer the best tool for the job, except for quick once-only processing.
> (4) Although you also don't have to decode it for it to be useful.
> 1. I can use wc -l on a stream of QSN strings
I can already use `wc --files0-from=-` on a stream of NUL-separated tokens.
> 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0'
You can already `grep` for NUL bytes[1].
> (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode.
But filenames are neither ASCII nor Unicode - they are bytes, which can include 0x80 through 0xFF (not valid ASCII) and things like 0xFF (not valid UTF-8).
I still can't see a single use case where I'd want inter-process communication to use QSN. Human input, sure, I'd love something more convenient that `$''`. Human output, also sure, humans can't read NUL characters or tell the difference between spaces and tabs after all.
If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings).
There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1]
Another argument is that in networking framing, you have a few choices:
(1) delimiter-based (NUL bytes)
(2) escaping (QSN with \)
(3) length prefixed (netstrings)
So Oil supports #1 and #2 now. Oil doesn't support netstrings but it actually does make sense, and avoids encoding/decoding. Though QSN encoding can be made extremely fast, just like it's been done with JSON [2].
[1] GNU grep actually has the ASCII/binary detection problem that was brought up on HN awhlie ago. Honestly it would be worth making a "slow correct utf-8/QSN grep" that avoids this.
> If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings).
I don't see how they are anything like the same idea. JSON is an object serialization format. These objects may contain strings, and the encoding of those strings unfortunately is not binary-complete. But most importantly objects have structure. QSN is a binary string encoding format, and has no structure outside of the sequence of bytes. In any case I'm not sure why you keep bringing up JSON.
On the other hand, if it wasn't for the fact that JSON is so strongly tied to JavaScript QSN might've been a good string encoding format for it.
> There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1]
I know what the words mean (and I've been using JSON and grep for years), but I don't understand what that sentence means.
> Another argument is that in networking framing, you have a few choices:
I don't know what "networking framing" is. If you mean actual network packet structure, I don't expect you'll be able to convince a single network engineer that QSN is a better choice than netstrings.
In any case, I'm going to have to shut down this thread now. I don't expect anyone else is reading this, I'm not learning anything new, and I can't convince you that QSN is a bad idea for IPC.
(1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'.
The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation.
(2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN:
https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498
True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around.
(And yes, QSN is a regular language [1])
(3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV.
(4) Although you also don't have to decode it for it to be useful.
1. I can use wc -l on a stream of QSN strings
2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0'
It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text.
(5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode.
-----
I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it.
[1] http://www.oilshell.org/blog/2020/07/eggex-theory.html
[2] That is, if I get help :)
[3] http://www.oilshell.org/blog/tags.html?tag=awk#awk