Right, and the parent comment is saying that `cat` doesn't figure it out very well.
I recall that GNU cat used a constant value that was tuned for read performance on common hard drives, giving no consideration to write performance. Looking at the GNU cat sources today, it doesn't look at all how I remember; I'd have to study it a bit to tell you what it does.
Edit: Hrmm, it seems I was mis-remembering. Perhaps I was thinking of the minimum values (32KiB, 64KiB, 128KiB below)? Or perhaps I was thinking of a BSD cat? Anyway:
How GNU cat sizes its buffers, by version:
- >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out
No, I was talking about dd. As far as I know it does not have any smart tuning of block sizes. It definitely didn't in the past and I doubt it was added.
Not sure what cat does but like I said in my other comment its not really cat itself that does the disk writing in that command but rather the shell redirect. Edit: Nope, I'm wrong there!!
Ah, I misunderstood you. You're correct, `dd`'s default block size was and is 512, which is specified by POSIX.
I interpreted "the command" in the original article as "the command you end up running, whether it be `dd` or `cat` or something else."
But you're mistaken about the shell redirect. In either scenario it's the command (`dd` or `cat`) making the write() syscall to the device. The shell passes the file descriptor to the command, then gets out of the way.
Indeed I was mistaken about the shell, sorry about that. The redirect method I forgot a really long time ago and my memory made it into something it was not.
In that case I guess dd does call a sync() on every output block? Because it's definitely slower and the LED pattern on a USB stick is also much more 'flashy' when using 512 bytes.
For context, choosing the block size has implications on the speed at which data gets written to a USB stick, so not being able to tune that on cat can be a problem.
I guess that made sense when the standard HDD block size was 512b. Then it went to 512/4kb then pure 4kb, not sure what SSD's do maybe also 4kb? Experimenting with speeds and block sizes in the past has shown very quickly diminishing returns increasing the block size. As long as the CPU can keep the queue depth over 1 the device should be flat out.
Generally they do a logocal 4kb, but it's usually physically larger pages of more than 1M per write. a good ssd will helpfully cache and batch up small writes but if it gets it wrong then it'll amplify the wear and kill a drive quicker than needed. That's another reason to do dd with a larger block size, since it'll make it a lot less likely that you write multiple blocks for a single update
Good point. I guess there's so much going on with various types of caching and wear levelling that 'let the device figure it out' is best. And the queue can be on the device now with NVME not on the host so its not a dumb queue any more.
I don't read it that way. He calls out that 'weird' command specifically. But indeed he doesn't specify.
I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!!
Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images.
It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.
> though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat.
No? The shell redirection is just
int tmp_fd = open("/dev/sdb", O_WRONLY|O_TRUNC, 0666);
dup2(tmp_fd, STDOUT_FILENO);
close(tmp_fd);
(plus error handling and whatnot)
`cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
i think he's talking about cat being the command that figures it out automatically