For sure! The only problem is there should be a non-blocking system-provided API and there isn't.

POSIX as the system, of course.

the system API is not syscalls but libc. so why does it feel strange?

The system-provided API for getting DNS user/system preferences on Unix systems is to read /etc/resolv.conf. Every application is free to implement their own lookup from that.

This isn't even correct on Linux as it won't work if your user has anything other than or in addition to the dns module in their nsswitch.conf. You must use glibc's resolution on Linux for correct behavior. If it's software on your own systems then do what you want but you'll piss off some sysadmins deploying your software if you don't. Even Go farms out to cgo to resolve names if it detects modules it doesn't recognize.

That is absolutely not the API on macOS, which is a certified UNIX.

In this case it isn't in the kernel, but in glibc. Could someone implement an equivalent alternative? Do any language runtimes re-implement DNS resolution?

I think most languages use the OS api by default, but there are plenty of libraries out there that bypass the system resolution.

Go does. And it supports timeouts and cancelation.