Only one appears to allow for E2E without the help of a third party. If person A and person B are technically capable and want to run their own servers and use their own clients, without involving a third party, then Matrix appears to allow for that. With Signal E2E is only available if persons A and B involved Signal in the process, including using Signal's software. Matrix appears to be just a protocol. Theoretically, there is no need for users to involve the Matrix organisation in their E2E communications.
Signal: - cannot run own server - writing clients discouraged; centralised control preferred
Matrix: + can run own server + writing clients encouraged
Ideally, what persons A and B might want is a protocol that allows for peer-to-peer connections without passing traffic through the third party. The server might be merely a point of rendezvous to enable one person to discover the network address of the other. For example, person A contacts person B's rendezvous server for person B's address, port, etc. Then person A makes direct connection to person B. Once that peer-to-peer connection is established there is no more need for the server. No traffic passes through it. It could even be shut down.
Signal: - cannot run own server - writing clients discouraged; centralised control preferred
Matrix: + can run own server + writing clients encouraged
Ideally, what persons A and B might want is a protocol that allows for peer-to-peer connections without passing traffic through the third party. The server might be merely a point of rendezvous to enable one person to discover the network address of the other. For example, person A contacts person B's rendezvous server for person B's address, port, etc. Then person A makes direct connection to person B. Once that peer-to-peer connection is established there is no more need for the server. No traffic passes through it. It could even be shut down.