You mean start by building something that can be used and tested in isolation, rather than trying to build an enormous system in one go? Isn't that what you've been arguing against?
No I mean solve the problem "we need to build a program that does what it's required to do" (and no more) before trying to build a library that will cure diseases.
Libraries are much harder than applications because they must work for a large number of applications with diverse requirements. They need to be more abstract, and therein lies the danger.
Regarding the size, clearly wrong. It depends a lot on the library. A windowing or font rastering library will be a lot larger than your typical application.
And for libraries that are much smaller than the application itself, why bother depending on them? (Anecdote, I heard the Excel team in the 90s had their own compiler).
How about making the project good first? Let's try to get something done instead of theoretizing.