Recumbents have terrible visibility (can't see over cars), suck for really steep climbs and sprinting, and are kind of unsafe for fast group rides with multiple cyclists packed close together.
I think the main problem with recumbents is failing harder in borderline unridable conditions. On a conventional bike you can temporarily move body support to the feet for a section of pave, jump over potholes, fluidly transition to walking and back if necessary.
Just imagine recumbent cyclocross, that mental picture is all you need for understanding why recumbents are a niche. Most real life transportation use of bicycles is full of lesser versions of similar challenges.
