When you click a link on your SearX instance and you don't use referrers, how can anybody track you? Nobody knows that you are coming from your "backend" engine.

You just reveal your search queries to the hosting provider if he maliciously intercepts them.