Yes.. The RCE happens at the connection stage before anything is logged.

That's insane. How exactly does this happen? Are there no EDR/IDS who can detect an RCE at the connection stage?

An IDS may detect something depending on what it is looking for. The grandparent is saying that sshd doesn't log anything. Which is not that surprising since sshd is atracker controlled.

An EDR would have detected an inbound connection to port 22. Then it'd have detected the attacker's activity (opened files, executed commands, etc)

If the EDR is capable of intercepting the forks, clone() execves, open(), etc, then you can follow the traces. If it's able to deny certain activity based on rules like modifying /etc/ld.so.preload or download files with curl/wget, it'd have made the attacker's life a bit more difficult.

If the attacker loaded a rootkit, then probably you'd have lost visibility of what the attacker did after that. Also not all the EDRs hook all the functions, or they have bugs, so many times you are not able to follow a trace (without pain/guessing).

This telemetry usually is sent to a remote server, so the attacker could not have deleted it.

It is very difficult for EDR to detect something like this, because it would look like a SSH connection.

Yeah, but then you would have ssh traffic without a matching login.

Wonder if any anomaly detection would work on that

That would look the same as a random failed ssh login, which happens all the time. The connection isn't maintained past that point (unless the payload chooses to do so).

It would be similar but the payload is going to be abnormally large compared to other failed login attempts.

Not if the attacker uses it just to gain an initial foothold and then extends their access through other means.

Interesting... Though you can edit whatever log file you want

Any log that root on that box has write access to. It’s theoretically possible to have an anomaly detection service running on a vulnerable machine dumping all of its’ data to an append-only service on some other non-compromised box. In that case, (in this ideal world) the attacker would not be able to disable the detection service before it had logged the anomalous traffic, and wouldn’t be able to purge those logs since they were on another machine.

I’m not aware of any services that a) work like this, or b) would be able to detect this class of attack earlier than last week. If someone does though, please share.

You would be sending logs to a log collector (a SIEM) in security terms, and then you could join your firewall logs against your SSH auth logs.

This kind of anomaly detection is possible. Not sure how common it is. I doubt it is common.

In any case the ROI for correlating SSH logs against network traffic is potentially error prone and may be more noisy than useful (can you differentiate in logs between SSH logins from a private IP and a public one?).

An EDR tool would be much better to look for an attacker’s next steps. But if you’re trying to catch a nation state they probably already have a plan for hiding their tracks.

You can do it on a single machine if you use the TPM to create log hashes which can't be rolled back.