Anyone can send email as anyone else anyway. That lack of security is inherent in the way email currently works. Not sure I see how giving IMAP access makes things worse since IMAP doesn't have a mechanism for sending messages.
I would also hope they're not storing passwords in plaintext. Obviously they need access to the plaintext password to auth with your mail server, but I would hope this is still stored encrypted.
Which is why the key should be physically given to the system when it is started and then only stored in memory. The key file should not be available on any network-attached machine. Of course there's still potential for exploits in this scenario, but it does help minimize the attack surface.
I would also hope they're not storing passwords in plaintext. Obviously they need access to the plaintext password to auth with your mail server, but I would hope this is still stored encrypted.