Is anybody using Lambda to run huge MapReduce jobs? Do people still use Hadoop?
Doesn't this basically just let you have something like HDFS for running large distributed computations with some shared state, without having to reach for S3 or redis?
I know at least one team at Amazon that does runs map-reduce style jobs with Lambda, but now that Athena supports user-defined functions, I'd personally be inclined to use it instead over EFS or S3 + Lambda.
We use Lambda with S3 as intermediate storage between different steps, sort of multiple-stages map-only MapReduce. And we still use Hadoop on premise :)
Is anybody using Lambda to run huge MapReduce jobs? Do people still use Hadoop?
Doesn't this basically just let you have something like HDFS for running large distributed computations with some shared state, without having to reach for S3 or redis?