The start-up I am contracting for is in a similar business (homeware and interior design) and has recently made the same decision to shift focus from a low margin, high volume affiliate model to a low volume, high margin "boutique" model. Curious to see how it pans out.
The low volume & high margin fashion site probably has a good chance of being successful as long as it carries recognizable and fashionable brands. The reason I say that is that sites like Net-a-Porter and MrPorter (which probably are mid volume) seem to be doing quite well based on the few articles I've read. Below is just one in case anyone is curious.
Recognizable fashionable brands such as John Lewis do fine on their own and are not going to give up a large chunk of their margin to a middleman. The biggest challenge for the new business model is try to discover and push less known brands or brands with little/no online presence with high quality products.