You can compile most common Linux kernel targets with clang, but the kernel was historically optimized with GCC in mind, so I believe there are still size and speed impacts from using LLVM.
Not to mention plain ol' inertia. The vast majority of people compiling the kernel are using GCC. Switching to a different toolchain or maintaining two toolchains (assuming there are no subtle ABI issues between a GCC kernel and LLVM driver) isn't necessarily trivial, and requiring that just to build a single driver might rub some people the wrong way. That's not a _problem_ with LLVM, but it is an inconvenience.
Not to mention plain ol' inertia. The vast majority of people compiling the kernel are using GCC. Switching to a different toolchain or maintaining two toolchains (assuming there are no subtle ABI issues between a GCC kernel and LLVM driver) isn't necessarily trivial, and requiring that just to build a single driver might rub some people the wrong way. That's not a _problem_ with LLVM, but it is an inconvenience.