Looking at the source code, this seems to work by generating dedicated parser code for a yiven definition which will copy values in a certain order through a flat copy.
I'm seeing little specifications or conversions regarding endianness so I'm guessing that's out of scope for this project. It seems almost completely backwards incompatible and I'm not too sure about their security validations. I don't think this and Flatbuffers are competing in the same space, really.
I definitely believe this is fast, it's as close to a memcpy to a network packet as you can get. I'd be wary to use this on external data in any native language without any kind of fuzzing first.
Generating code per message might not be the right choice anyway: if you have a lot of messages, a table driven approach can save you a lot of code size. Optimizing for speed in microbenchmarks can lead one to pessimize overall program architecture in ways that are hard to undo later.
It depends on your application, but if it's just generated code then I don't really see the problem with code size. As long as it's easy to (de)serialize data or add a nice big facade between the generated code and business logic, the generated backing code can be a complete mess of spaghetti code for all I care.
Generally, message specifications are written by hand, so even a big project may only have a couple of hundred. Doesn't sound so bad.
Also, presumably, if code size really is a big concern, you can decode this in more code efficient ways too, as long as you are less concerned with performance.
You might be surprised to see how quickly mechanically-generated marshaling code adds up. Look at Android's frameworks.jar one day and see how much is just the same AIDL-generated Parcel-manipulation code over and over again. Cache effects and IO costs are so severe in modern computing that my default (but rebuttable!) position is to prefer table-driven approaches over code generation wherever possible.
Windows COM switched from codegen to table-driven "stubless" marshaling decades ago at no noticeable cost to performance and with a huge code size win.
> Karmem has proven to be ten times faster than Google Flatbuffers
I’d recommend not using the word “proven” here. In computer science this word typically refers to a mathematical proof. In this case it seems that you ran a regular benchmark for some schemas.
I’d also like to see more what the benchmark actually does. A typical trade-off of these formats is how much you do up-front vs on-demand. E.g. accessing fields after multiple variable-length field: Here it’s possible during “decoding” to make sure all fields can be accessed in O(1), or you can do nothing and then every time you access a field you compute the field location. Whether the benchmark accesses the field once or ten times will make a huge difference.
In general: If you’re just telling me that it’s 10 times faster without explaining why I will be skeptical.
I'm on the side of the original wording here as far as correctness is concerned - "has proven to be" is a common expression for "in our experience, it is", whereas "has been proven to be" would give the connotations of formal proof.
This is just one of those quirks of human language. Yes it's occasionally annoying similar things have very disparate meanings, but English in particular is never going to shake them off.
Beyond what you already said, "faster" can be contextual in the sense that a more terse encoding results in faster transfer over networks or busses, though it's often at a cost in encode/decode speed.
Not a native speaker, but I thought that's what the difference between "proven" and "proofed" is. The latter is the type of mathematical proof you are referring to, the former the colloquial attribute of having been used successfully for that purpose.
A reasonable guess but ultimately wrong. "Proofed" is not a word, "proven" is the correct adverb form (as in, "In this paper it shall be proven that..."), "proved"/"proving"/"proves" is the correct verb (depending on tense), all these forms of "proof" can be used in a colloquial sense like you tried, but never in a technical setting where an actual mathematical proof might actually be presented. "Shows"/"demonstrates"/"suggests"/"evidences" all better fit your intended use case. The rules in English are arbitrary and the patterns are inconsistent and misleading (and therefore anyone who obsessively enforces said rules is a petty pedant doing it solely for the enjoyment of acting smarter than others).
> A reasonable guess but ultimately wrong. "Proofed" is not a word
And if it were I’d certainly assume it was a reviewing / publishing concept related to proof copies and proofreading (as in, a proofed copy would have gone through proofreading and no error would’ve been found).
No one has successfully connected proof (v) and prove (v) here, though the examples of the former are exhaustive in modern speech.
The difference is the two kinds of proving, one of which is vanishing from the language: the sense of testing something, practically, rather than by reason or evidence. A proof copy is when a type plate is prooved, the rising of dough prooves the culture's activity.
So these irregular noun-to-verb forms exist because proof has largely lost the second meaning.
I’ve never seen “proofed” used as anything but a hyphenated suffix meaning “protected against”. E.g. sound-proofed.
It’s pretty normal to use “proved” in place of “observed” or “measured” in day-to-day speech though. Usually a few repeatable measurements are enough proof (evidence) for most people to consider something “proven”.
Edit: I forgot the proofread meaning pointed out by the sibling comment.
People don't generally use "proofed" like that—I would always use "proven" to talk about something that had a mathematical proof, and "proofed" only comes up in specialized areas like publishing (where it's used as the past tense of making a proof of something to be printed or as short for "proofread").
I use "proved" for mathematics. For me (middle-aged British) "proven" sounds a bit pompous and rhetorical, not the sort of language an honest down-to-earth mathematician would use.
You can't justify this one with the "no true scotsman" fallacy. If you do a test that shows that X is faster than Y, you're allowed to say "X proved faster than Y". You're equally allowed to say "merge sort is proven to sort any list with an upper bound of n log n comparisons" after you write up a mathematical proof of that. (But welcome to the sadness of the real world, where the running time of a sort might not be slow because of the number of comparisons, but "real world" annoyances like cache hit rate.)
HN is largely software engineering practitioners. Practice is one level of complexity above the mathematics; among the mathematically optimal solutions to a problem, you have to find the one that's actually fastest on some sort of computer you can buy. Otherwise the math is just an interesting fact to throw out at parties. Testing in the real world is really the only way to find the solution that is fastest in the real world, but of course, mathematics dramatically limits the search space for the ideal algorithm.
Basically, farmers aren't interested in how much milk a spherical cow can produce. There are no spherical cows.
This is a quite interesting discussion actually! Some thoughts:
First of all: Theoretical results are actually quite important for real-life performance. It’s possible to write an implementation of bubble sort, have a bunch of benchmarks which shows great performance and 100% test coverage. Yet there are cases where you’ll hit the n^2. The theoretical proof (especially the worst case) are quite important to not get bitten by this. You can search for “accidental quadratic” for plenty of cases where this happens in practice: people have implemented algorithms which works well in practice for small n, but fails (unnecessary) for larger n. A theoretical analysis would have caught this (although probably not worth the engineering effort).
Secondly: “proving” isn’t exclusive to theoretical analysis. I’d argue you can “prove” that a method call in Ruby is slower than a function call in C. You can look at the machine code which gets executed and see that there’s no way for a CPU (regardless of cache, pipelining etc) to execute the Ruby variant faster. There will of course be plenty of assumptions (as there is in theoretical proofs as well!), but in my option it’s a proof nonetheless.
However, to claim that “Kermem is provably 10x faster than FlatBuffers”, I’d say that you need more than a few benchmarks. You’d have to argue that every operational is (at worst) 10x faster.
It's not proven. You've demonstrated it was faster on your machine. That's the end of it. From a CS perspective, you've not proven anything at all.
You've also completely misrepresented my point. It's not a "no true Scotsman" fallacy, at all. Not sure what spherical cows have anything to do with anything.
Do you know that a word can have a different meaning in a different context? You describe a formal proof but the Collin dictionary define proof that way :
Proof is a fact, argument, or *piece of evidence* which shows that something is definitely true or definitely exists.
That demonstration is a piece of evidence that shows that an algorithm is truly faster on that computer with that specific input, sure it's not a formal generalizable CS proof but it a valid usage of the word proven nonetheless.
> If you do a test that shows that X is faster than Y, you're allowed to say "X proved faster than Y".
No, you have the hypothesis X is faster than Y, and some evidence to back it up. It's no different than a scientist treating cancer cells with unobtainium, and claiming it "proves unobtanium cures cancer".
You need a lot more rigor to prove benchmarks. Notably absent is some independent verification and peer review. It's possible to write benchmarks in ways that bias the outcome.
If you have multiple different benchmarks written by independent authors and counter-reviewed, all showing 10x speedup, sure, call it "proven". A few benchmarks on 3 arches, no, use the word "shown" or similar.
I'm a computer scientist with degrees and stuff, and I don't mind. I believe this is an example of the proof being in the pudding?
By coincidence I just wrote a proprietary serialization format a few months ago for work, and it's proven to be 42x better than flat-buffers, depth-buffers, proto-buffers and even the equivalent electro-buffers. Unless you care about average space efficiency, then it's worse probably!
> proven
> adjective
> demonstrated by evidence or argument to be true or existing.
Looked for some random news article and found "Trump has proven himself unworthy to be this country’s chief executive again", "India's Toy Sector Proven Its Mettle By Transforming Itself". None of these are wrong usage.
With many definitions, it means any definition is acceptable. If one entry in dictionary satisfies usage, it is correct usage. Also "tried and tested" is not same as mathematical proof, but any of those three usage are fine according to most.
Flatbuffers trades off encoding speed, programmer ergonomics and binary size (it produces many bytes and it's awkward and still pretty slow to encode) for decoding speed (almost a no-op if you forego buffer verification, which you shouldn't most of the time). Imho it's not a good choice for network wire formats, but for storage it's pretty good.
Agreed. Given the number of serialization formats available at this point, it feels like anything that doesn't start off with a discussion of the trade offs they've made and how this affects the various aspects of performance is a bit of a red flag that they're not actually bringing anything novel to the table.
Go never really clicked with me, but isn't the point of serialization formats interoperability?
Like, ok, its 10x faster unzipping than another obscure language dependent format, but how is that better than perl storables or python pickles or ruby ser's other than being "faster"?
How do i call this from java or dotNet, and why would i do this other than to make everyone I work with miserable to adopt yet another format?
Languages
Currently, we have focus on WebAssembly, and because of that those are the languages supported:
AssemblyScript
Golang/TinyGo
~Swift/SwiftWasm~
Zig
C
To get accepted in most of the game engines, the author would need to provide a way to override malloc/realloc/free - even better if no need to realloc.
Seems the benchmark does read into native structs in both Flatbuffers and this in Go. I am confused by the performance claim too. Seems very similar design v.s. fbs, would be surprised if it is from format differences rather than codegen / Go implementation inefficiencies.
You're referring to the fact that Capn Proto claims to be zero copy. That doesn't mean there is no serialisation overhead.
In my experience with Capn Proto, the vast majority of the time the zero copy feature is pointless. The Capn Proto C++ APIs are extremely unergonomic so 99% of the time you end up copying the data into your internal nice C++ structures anyway, completely giving up zero copy.
I've used Capnp quite a lot and I really wouldn't recommend it. It's quite old and complex and the unpleasantness of the API alone is enough to put me off. I would pick Protobufs every day for small amounts of data.
For large amounts you are better off with SQLite or DuckDB.
Capn'Proto isn't that bad. I've used the C++ quite a bit, and while the authors code style isn't to my taste it's fairly well designed (nice layering that lets you implement different use cases, like integrating the RPC framework with your own event loop).
There's also not much out there comparable to its capability passing RPC either.
Agree on the zero copy though, and the RPC framework does malloc a lot.
It's a format. You can serialise data to it. What about it isn't a serialisation format?
Obviously I wouldn't recommend it for small amounts of data, e.g. for RPC calls. But when you need to store lots of data it's much better than JSON or binary JSON style formats.
Don't know if the owner will ever read this comment, but please add some sections on:
* Its design goals and rationale
* How those decisions are translated into the actual performance
* What is the trade off made to achieve that
* Why should/shouldn't anyone else use it
Rather than just a vague performance claim that it's ten times faster than something else. It's not just for this specific library, but applicable to any libraries seeking for broader audiences.
I don't even know if it's a proper data structure in the CS sense, but I couldn't agree more with the sentiment. It's a simple concept and together with pattern matching it just makes life so easy it's hard to go back to the alternatives.
Case in point: I used rust for 1-2 years and am now on a project in Go. Even though Go fits my style and my use case better, I miss enums soo much. Both the std lib types like results and option but also the custom ones.
I suspect a lot of the speed comes from structure specific serialization (avoiding reflect). This can probably done with less unsafe code, and for most use cases that'd be a better trade-off.
Yeah. I have found that for message passing systems the tooling around multiple message types is more important in practice than optimizing for a single specific type. You can solve this with unions/tagged enums or some bespoke "message type metadata", but you HAVE to solve it some way. On the receiver end, you need to have a good story for routing different kinds of messages.
What’s the backwards compatibility story for coding using Karmem? When is it legal to add, modify, or remove a struct field without having to recompile all of the binaries using this format and replace them atomically? When is it legal to add, modify, or remove a struct field without requiring code to be refactored? What about enum variants?
These questions may not matter for every use case (e.g. you ship a single binary from a single codebase) but I think that clearly defining these rules opens up a lot of very cool use cases that are otherwise prohibited.
> In order words: you can't edit the description of one inline struct without breaking compatibility.
and
> Tables: Tables can be used when backward compatibility matters. For example, tables can have new fields append at the bottom without breaking compatibility.
Otherwise, I agree it's rather unclear exactly what you can do with tables.
Keeping some context in mind is probably helpful here. The target is WASM. And if you look at the organization the repo own is a part of, it is a web wallet for the cryptocurrency Nano.
So perhaps using a generic message serialization library is too slow for its use case since WASM's data types are just ints and floats since the parsing code can't behave like on a native CPU with things like bytes and C-structs?
It would have been great if they had disclosed links to issues regarding out-of-bounds access for things like Protobuf or Flatbuffer.
I'm seeing little specifications or conversions regarding endianness so I'm guessing that's out of scope for this project. It seems almost completely backwards incompatible and I'm not too sure about their security validations. I don't think this and Flatbuffers are competing in the same space, really.
I definitely believe this is fast, it's as close to a memcpy to a network packet as you can get. I'd be wary to use this on external data in any native language without any kind of fuzzing first.
That said, I do like the way the generators work.