I'm a huge fan of Asciinema, and this is awesome work.
Regarding the live streaming feature, I hacked a similar thing on top of s2.dev streams[1] (disclaimer: I am a co-founder), which could alleviate the need for a relay in this architecture[2]. Naturally showing off `btop` was the highlight for me as well :-D.
> what are some of the notable issues that were find using the DST approach
We've discovered a few distributed deadlocks. And in general it's been incredibly helpful in exercising any parts of the system that involve caches or eventual consistency, as these can be really hard to reason about otherwise.
> if a LLM system would be able to help analyze the TRACE logs
Neat idea! For us, the logs are typically being dug into only if there is a failure condition for the test as a whole. Often times we'll inject additional logging or state monitoring to better understand what led to the failure (which is easy enough to do given the reproducibility of the failure in the sim). Trace logs are also being analyzed in the context of the "meta-test", but that's just looking for identical outputs. (More about that here: https://github.com/tokio-rs/turmoil/issues/19#issuecomment-2... )
> I suppose the writers could batch a group of records before writing them out as a larger blob, with background processes performing compaction, but it's still an object-backed streaming service, right?
This is how it works essentially, yes. Architecting the system so that chunks that are written to object storage (before we acknowledge a write) are multi-tenant, and contain records from different streams, lets us write frequently while still targeting ideal (w/r/t price and performance) blob sizes for S3 standard and express puts respectively.
Wait, data from multiple tenants is stored in the same place. Do you have per-tenant encryption key, or how else are you ensuring no bugs allow tenants to read others data?
(Founder) We will be using authenticated encryption with per-basin (our term for bucket) or per-stream keys, but we don't have this yet. This is noted on https://s2.dev/docs/security#encryption
Regarding the live streaming feature, I hacked a similar thing on top of s2.dev streams[1] (disclaimer: I am a co-founder), which could alleviate the need for a relay in this architecture[2]. Naturally showing off `btop` was the highlight for me as well :-D.
[1]: https://s2.dev/blog/s2-term [2]: https://docs.asciinema.org/manual/server/streaming/#architec...