macOS should have just used the Linux kernel. The fairly complex and difficult-to-duplicate Cocoa UI layer would have made the odds of a macOS-compatible open-source clone appearing anytime in the near future slim.
Even today, I hope Apple makes a switch to their own patched version of the Linux kernel. It would be a great breath of new air into macOS. In terms of sheer features and compatibility, the Linux kernel surpasses Darwin by massive measures.
NeXTStep was built around BSD, and so BSD followed through to MacOS, which is really frankly just re-skinned NeXTStep. I'm pretty sure it would have been a real PITA to rework it around Linux at that point. Lots of existing applications and tools would have had to have been reworked.
And at that time (late 90s) the Linux kernel was certainly awesome but not clearly that much more awesome than BSD. The amount of driver and architecture support in the kernel wasn't as expansive as now.
Apple has a visceral dislike/distrust of the GPL, and goes to great lengths to avoid it - like authoring an entirely new compiler (clang) to replace the GPL-licensed gcc. This is also the reason why the version of bash bundled with Apple OSes is ancient.
The problem is the difficulty in mixing in proprietary blobs. The only way to link proprietary drivers into the kernel is the way Nvidia does it, and it's a legal grey area to this day.
Even today, I hope Apple makes a switch to their own patched version of the Linux kernel. It would be a great breath of new air into macOS. In terms of sheer features and compatibility, the Linux kernel surpasses Darwin by massive measures.