I use Chebyshev polynomials extensively in finance and have tried problems like MNIST with Chebyshev and they get close to CNNs in accuracy.
ApproxFun Julia package pretty cool for Chebyshev work:
https://juliaapproximation.github.io/ApproxFun.jl/latest/
What is your architecture? Is it just a fully connected layer of chebyshev?
I use Chebyshev polynomials extensively in finance and have tried problems like MNIST with Chebyshev and they get close to CNNs in accuracy.
ApproxFun Julia package pretty cool for Chebyshev work:
https://juliaapproximation.github.io/ApproxFun.jl/latest/