It depends - some algorithms exploit correlations between pixels to reduce the computational load. This is bad if the Z channel is not strongly correlated with the others (RGB are very strongly correlated in natural images). Since depth shouldn't usually be correlated with color, this might cause some issues. Some experiments were done in "Learning Feature Representations with K-means", A. Ng. and A. Coates. (http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p...)

Give it a try, but also look at the underlying assumptions of your algorithm if it performs poorly.