Ahh good to know about those -builder repos. It would be awesome to publish the compilers as container images, could also publish the squashfs as an artifact to a registry. And it shouldn't be too hard to back-publish the existing tarballs as containers without rebuilding. Might poke around a bit later
Thank you! I'm also excited at the possibilities and today you can share "long links" (with all the data embedded in the URL's fragment) but in the future short links with storage on the server or loading from a gist for example would be another cool feature. And having a variety of interesting examples ready to go like you mention would be nice too. There's almost too much there that I was worried I would never ship and would still be playing with examples. For fun, here is the most basic version of compiler explorer https://programexplorer.org/#s=eyJjbWQiOiJzaCAvcnVuL3BlL2luc...
You can run any program that is in the container you select. Right now it ignores the entrypoint from the container config (though you could manually copy that into the Cmd field). Shell is a convenient program to run though so you can pass it a script to then run multiple commands. But you can run `date` directly for instance: https://programexplorer.org/#s=eyJjbWQiOiJkYXRlIiwic3RkaW4iO...
Yeah names are hard and it is just a working name. I'm simultaneously using containers as the distribution and runtime mechanism but also wanted them to be an implementation detail and not the spotlight
To add on since this took me a while to understand: for a single token, self attention is permutation invariant because we take the qK (one query dot all the other keys) weighted sum of all the values; that sum is what gives the invariance because + is commutative. But for all the tokens, the mha output matrix will not be invariant, but rather equivariant, where you apply the same permutation to the output matrix as you did to the input tokens. What might be a more useful example is to take one position, like the last one, and compute its mha for every permutation of the previous tokens; those will/should all be the same.
Great read and visuals. I think they typo'd the pun on basically/basisally. It got me thinking about program synthesis in the following scheme: data is embedded as vectors and program operations are metric tensors (or maybe just fields in general?) which tell the data how to move. Then, if you have an input/output pair we seek some program to move the data from input to output along some low energy path. Model a whole program as a time varying (t 0-1) metric tensor (is that a thing?) and optimize to find such an object. Maybe you choose ahead of time the number of operations you're searching over and these are like spline basis points and then you lerp between the metric tensors of each op; or you do it continuously and then somehow recover the operations. Then you want to find one program which satisfies multiple input/output pairs, ie one time varying metric tensor (or generally field) such that if you integrate from the input points they all end up at (or close to, which makes me think that you want some learned metric tensor for closeness) the output points. Right now I'm only thinking of unary ops with no constants, maybe the constants could be appended to the input data symbolically and you also get to optimize that portion of the input vectors, with the constraint that it is a shared parameter across all inputs.
surprised they didn't go straight into cloud-hypervisor, though I haven't actually tested with gpu yet but it is on my todo list. OCI layers can use zstd compression. I wonder if they are defeating layer sharing by splitting in 500 mb chunks. Lambda splits your image into chunks and shares at the block layer (I believe even same chunk different (user's?) container on a single host). Esp for 15 GB images I'd think using lazy pulling with nydus/stargz or whatever would be beneficial. I'd like to test out snapshotting, though my testing already boots a guest and runs a container in ~170ms; and I'm not actually sure how you write the guest init to signal it is ready for snapshotting and then wait properly (maybe you just sleep 1000?) so it resumes from the snapshot in a good state. I know fly has written about their use of snapshotting but I don't think it went into that detail. Cool stuff overall though, not worrying about locations and the yucky networking to do so seems nice
Bit late and I posted essentially this same comment on their site in hopes they'll see it, but this article jumped out at me because a while ago I was thinking about what it would take to get a whole organism single cell atlas so that you could explore it on the computer. The only thing I can really think of is to take a CNC machine, put it in a big freezer, then scan one layer of the specimen with a microscope and whatever multispectral lighting/imaging you need, then mill away a layer and repeat. Pathology slides are prepared with stains to enhance the contrast of membranes and other features which is probably not an option in the frozen state, so that's a big hurdle, but maybe you can do it with some fancy multispectral imaging. And if you can't, maybe you can at least start with lower resolution structures like anatomy (though I think there's still contrast challenges for that). I know there are some single cell atlas efforts for parts of or whole brains for mice and maybe other organs, but like the author's dream idea of getting order thousands of specimens to study variation, it would be great dataset if you could capture everything in one go.
Wow thanks for the info! This was a good overview of one system https://academic.oup.com/mt/article/25/4/14/6815410 - 5um slice thickness at 0.7um per pixel and ~1 TB/cm^3, max sample size 50x50x20mm. Humans are about 75k cm^3 so a 75 PB scan isn't impossible. Probably start with a coarser scan and work on the compression algorithms. "Just" need a (much) bigger scanner. And figure out if the knife edge is okay with bones!
Very nice writeup and I appreciate the effort put into showing the process. I got nerd sniped yesterday playing around with how to find the isle_opt.rs filepath from the core file and didn't succeed but left some notes on scripting with lldb here https://gist.github.com/aconz2/aef366a7b198b8ac151df147fec32...
