If you want to test the water, here is a mega script who does it all in one go in a qemu virtual machine (granted, still some standard manual install of some debian based bootstrap iso):
Output is X Org and a falkon (chromium based) browser. But need some memory, diskspace and time.. Still a lot faster than doing it all by hand the first time around..
Trie to do this after about a year of learning programming which was probably too early in retrospect so I ended up kind of cargo culting it a bit and unable to boot. It would probably be more worthwhile for someone who knows something about the various tools and libraries.
Then you realize what a pain in the butt updating is without a package manager :)