In general, Erlang/Elixir solved this challenge in an interesting way: with peer state awareness and channels (functionally equivalent to most micro-services use cases). It is commonly the back-end secret in a lot of low-latency game back-ends.
If you are stuck on a polyglot loving project, than RabbitMQ or Kafka can bolt on most functions with the standard AMQP services. Erlang/Elixir is weird, but is a single kind of weird... which even has built-in profiling tools without external dependencies.
Erlang asserts two simplifying assumptions about distributed computation:
1. Actors can define incoming message queues with unbounded capacity
2. Actors can always crash in response to any given error condition
These assumptions produce a coherent system model, which is unfortunately incompatible with any physical system that can implement it -- unbounded queues are a fiction -- and which is effectively non-deterministic and unpredictable -- crash-only software makes it impossible to assert any guarantees on a callstack.
Erlang represents a sub-optimal local maximum. It's no panacea.
"It's no panacea." Agreed, but it does offer a unified model that abstracts away the repetitive error-prone duplicated code. The upfront training costs are offset by locking out vendor proprietary extortion-ware.
Admittedly, it probably isn't worth the effort if you have less than 48k concurrent users. But, the distributed consensus implementation still need added to many projects (RabbitMQ essentially packages these features for you).
From my perspective, it comes down to two choices:
1. Clown show Federation and Shovel for an extra dose of chaos
2. Monoculture Clustering with the OTP, and partitioning by region
Erlang asserts that all errors can be grouped together and managed as if they are of the same category, i.e. that any error is crashworthy. This model of errors isn't wrong,
but it is naïve, and it makes it impossible to design a system whose entities are resilient to what are, factually, normal runtime faults.
Erlang and the OTP are
interesting and worth studying and may be a good design choice under certain constraints. But in no way does the crash-only model of the Erlang OTP represent an optimal architectural model in general. Technology has advanced since 1960.
The approach that seems most common with serious Erlang systems is to identify the common errors and handle them without crashing.
Crashing on less common, or impossible to reproduce, errors allows you to gradually improve your error handling over time if it’s worthwhile, or just allow it to continue to crash into a saner state.
"Technology has advanced since 1960"... I would argue not significantly, and in some metrics it is now less reliable. When you see GPU lock up and burn down a power supply, the concept of hardware failure modes will change. LOL
I have no experience with Erlang, but wouldn't ABENDing one or both of the sending / receiving actors if a queue spills over be entirely consistent with both the model described and the finite nature of real systems?
I'm also curious about this:
> ...crash-only software makes it impossible to assert any guarantees on a callstack.
What sort of guarantees are we talking about here?
I would recommend: "Designing Elixir Systems with OTP: Write Highly Scalable, Self-Healing Software with Layers" ( James Edward Gray, II, Bruce A. Tate )
Erlang, and its simplified Elixir wrapper language can allow a OTP child process to continually crash while remaining operational for other users.
Some tend to get irritated by the idea of graceful fail back code. Again, something a dead-letter queue in RabbitMQ handles rather trivially.
If you are stuck on a polyglot loving project, than RabbitMQ or Kafka can bolt on most functions with the standard AMQP services. Erlang/Elixir is weird, but is a single kind of weird... which even has built-in profiling tools without external dependencies.
Best of luck, =)