I'd like to know more about reasonable expectations. What sort of algorithms are GPUs best for? I'm interested in using them for computation, not actual pixel shading. If you could give an example of something that would be in the sweet spot for GPU parallelism, and something that wouldn't, I'd appreciate it. Or a pointer to some good sources.
Two things: data parallel code is best, and you need to have a large enough amount of data to amortize the high cost of transferring data to the GPU. "Large enough" depends on your data access patterns.