> how does performance measurement and optimization work in that environment?
SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation.
After that, if you want to be really sure about regressions (which would include fail safes) you load test the whole thing put together.
> Is the key some sort of meta tooling that understands relationships between microservices?
This is quite hard to do when you have a lot of transactions. I don't think there's commodity software that does this because you'd need to configure that software to map on keys, then map those keys to services. Generally, the easiest way is to get engineering teams to declare upstreams and downstreams.
> Q2: How do you ensure the stubbed deps behave like the real thing?
Generally, generation. Something like protobuf or Open API generation will do.
> Q3: how do you handle logging and metrics in an unified way across the stack?
You issue high level standards like, "We'll use JSON logging with UTC time formatting". At the end of the day logging is very contextual and in a service ownership model the service owners are usually the ones reading and alerting on their logs.
> And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?
Shared dependencies. I'm not actually sure what's a cross cutting concern; generally services that are this small should be designed to operate mostly independently. They're small, but "microservices" tend to have a lot of fail safes built in. If you're referring to how do we not write 4000 config loaders then there's usually a team that builds a very generic config loader and everyone or a majority use it.
SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation.
After that, if you want to be really sure about regressions (which would include fail safes) you load test the whole thing put together.
> Is the key some sort of meta tooling that understands relationships between microservices?
This is quite hard to do when you have a lot of transactions. I don't think there's commodity software that does this because you'd need to configure that software to map on keys, then map those keys to services. Generally, the easiest way is to get engineering teams to declare upstreams and downstreams.
> Q2: How do you ensure the stubbed deps behave like the real thing?
Generally, generation. Something like protobuf or Open API generation will do.
> Q3: how do you handle logging and metrics in an unified way across the stack?
You issue high level standards like, "We'll use JSON logging with UTC time formatting". At the end of the day logging is very contextual and in a service ownership model the service owners are usually the ones reading and alerting on their logs.
> And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?
Shared dependencies. I'm not actually sure what's a cross cutting concern; generally services that are this small should be designed to operate mostly independently. They're small, but "microservices" tend to have a lot of fail safes built in. If you're referring to how do we not write 4000 config loaders then there's usually a team that builds a very generic config loader and everyone or a majority use it.