The main problem I see with "just files in folders" is thumbnails. You need some thumbnail cache, ideally pregenerated, for speedy browsing. And once you've got that, you've lost the simplicity of "just files" anyhow.
And you might not want them, but things like facial recognition, search by metadata, and object detection really only work with a central database/index of all the photos you have.
Just store the thumbnails in a cache or in a dotfile subdirectory. This can be transparent to the user. If the user is going linearly down or up the list you can also use a Kalman filter to predict what files and directories the user might browse next and preroad thumbnails for those directories in advance. Don't wait till the user actually scrolls to something to start working on thumbnail generation. UI Design 101.
Also, many consumer cameras embed thumbnails in metadata that can be extracted almost instantaneously. For those there is no image resizing work to do. But you can still load the thumbnails into memory in advance to make it even snappier.
Just don't make me add the folder to a "roll" or "library" just to browse it for 5 seconds.
You've more or less outlined what programs like Lightroom, DigiKam, and Darktable do. The integration with e.g. LR or DT gets you thumbnails rendered with whatever processing needs to be applied.
As I'm typically using whatever to copy images off of a phone or an external memory dongle I don't mind adding things to a catalog. The import process copies the files memory dongle and into folders organized by capture date. Certainly (for me) it's easier to hit a couple buttons in LR than it is to copy everything over manually. You can also set LR to automatically import pictures.
If you really are just allergic to importing photos, in the proprietary world Adobe Bridge can be used to go through your photos in a manner you've described. You can also quite a bit of management from the import screen in LR without having to actually import photos into a collection.
FastRawViewer (only Win/Mac, paid software) shows true RAW decoding (not embedded JPEG previews!) faster than 90% of other software shows cached previews.
Also, Adobe Bridge works with folders without any formal "collection" or "import" concepts. It cache meta and previews in user's home directory, but it is transparent to the user.
And you might not want them, but things like facial recognition, search by metadata, and object detection really only work with a central database/index of all the photos you have.