Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

No file system attributes or metadata on records which also means no (xattrs/fattrs) being written or updated, no checks to see if it's a physical file or a pipe/symlink, no permission checks, no block size alignment mismatches, single open command.

Makes sense when you consider you're throwing out functionality and disregarding general purpose design.

If you use a fuse mapping to SQLite, mount that directory and access it, you'd probably be very similar performance (perhaps even slower) and storage use as you'd need to add additional columns in the table to track these attributes.

I have no doubt that you could create a custom tuned file system on a dedicated mount with attributes disabled, minimized file table and correct/optimized block size and get very near to this perf.

Let's not forget the simplicity of being able to use shell commands (like rsync) to browse and manipulate those files without running the application or an SQL client to debug.

Makes sense for developers to use SQLite for this use case though for an appliance-type application or for packaged static assets (this is already commonplace in game development - a cab file is essentially the same concept)



> tuned FS + dedicated mount

For example, Ceph uses RocksDB as their metadata DB (and it's recommend to put it) directly on a block device, with the WAL on yet another separate raw device

https://docs.ceph.com/en/latest/rados/configuration/bluestor...


More just this:

mke2fs -t ext4 -b 1024 -N 100000 -O ^has_journal,^uninit_bg,^ext_attr,^huge_file,^64bit [/dev/sdx]

(smaller block size, 100,000 inode file table entries (tuned to the number of blobs), no journal, no checksumming, no extended file attributes, use smaller integer file offset IDs, 32 bit padded vs 64 bit)

Then mount it and run the same test.

You could go even further and tune fopen BUFSIZE to be no greater than 12,000 bytes. You can even create this mount on a file inside your existing mount... which is essentially akin to having an sqlite file without needing a client library to read/write to it.

Anyway - if the purpose is to speed up reads and save disk space on small blob files, there is little need to ditch the file system and it's many many upsides.


> If you use a fuse mapping to SQLite, mount that directory and access it

Related ongoing discussion, if someone cares to test this:

https://news.ycombinator.com/item?id=41085856


Some nice write up on this topic, although not the resolution of all discussion. But rather hints and guesses to keep it alive :)

https://unixdigest.com/articles/battle-testing-php-fopen-sql...

Has been on HN lately..




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: