You generally have to move carefully to get proprietary software to an open source model. One typical customer reaction to rumors you are open sourcing is "Great! We'll wait until you do that to try it out." The result is that [already not good] sales may tank completely.
RethinkDB wasn't proprietary. It was open source, licensed under the AGPL v2, the same license used by MongoDB, which is doing just fine. MongoDB is one of the top five in the DB-Engines rankings.
One of the key business differences between RethinkDB and MongoDB is that the latter keeps some enterprise-grade features/tools closed-source and available under a commercial license, i.e. the "open core" business model.
