Anybody who argues against deep learning based on energy consumption immediately fails to impress me. This article is particularly bad- claiming you need k*2 more data points to improve a model and using that to extrapolate unrealistic energy consumption targets for DL training.
The sum of all DL training in teh world is noise compared to the other big consumers of energy in computing. That's because the main players all invested in energy-efficient architectures. DL training energy is not something to optimize if your goal is to have a measurable impact on total power consumption.
> That's because the main players all invested in energy-efficient architectures.
If the cost was gigantic enough to make the investment worth it they must have found some really great improvements for it to end up being just noise. Improvements that somehow didn't have a noteworthy impact on general computing.
And yet people here have no trouble crying about electricity wastage of crypto. Also from my limited knowledge I think DNN models are not very transferable in real world setting requiring constant retraining even for a small drift in signal or change in noise modes.
> And yet people here have no trouble crying about electricity wastage of crypto
Which is many orders of magnitude more energy-intensive, on the scale of a small nation-state, and in most cases fundamentally wasteful by design. A very large pre-trained model can be reused very cheaply once it's finished.
> Also from my limited knowledge I think DNN models are not very transferable in real world setting requiring constant retraining even for a small drift in signal or change in noise modes.
This is FUD, promulgated by people who expected deep learning to solve all their problems overnight. All models will suffer from "drift" whenever the underlying data changes.
Part of what made deep learning so good was that it was able to generalize exceptionally well from exceptionally complicated input data.
It is unreasonable to expect that a model pre-trained on a huge generic corpus will be a perfect match for your very specific business problem. However it is _not_ unreasonable to expect that said model will be a useful baseline and starting point for your very specific business problem.
We are not yet (and might never be) at the point where you can dump a pile of garbage data into an API and get great predictions out the other end on the first try. But nobody ever thought you could do that, except the people selling expensive subscriptions to those kinds of APIs. The fact that they work at all should be taken as evidence of how amazing deep learning is; the fact that they don't work perfectly should not be taken as evidence that deep learning is bad/useless/wasteful/hype/whatever.
Don't let the clueless tech media set your expectations.
Professional data scientists and machine learning practitioners for the most part take their work very seriously and take pride in delivering good outcomes, just like professional software engineers. If deep learning wasn't useful to that end, nobody would be using it.
Open ended crying about electricity doesn’t make sense in the absence of specifics.
A big company like Microsoft probably wasted more money on pentium 4s 15 years ago. Electricity is just another resource - if the numbers work, burn away.
I know we’re nowhere near the following scenario, this is just to illustrate how things can go wrong even if the numbers tell you to “burn away”:
Image we have computronium with negligible manufacture cost, the only important thing is the power cost to use it.
Imagine you’re using it to run an uploaded mind, spending $35,805/year on energy.
The 50% of Americans earning more than this [0] are no longer economically viable, because their productivity can now be done at the same cost by a computer program.
Doing this with the current power mixture would be disastrous, doing it with PV needs about 1400m^2 per simultaneous real time mind upload instance (depending on your assumption about energy costs and cell efficiency, naturally).
In a more near-term sense, there are plenty of examples where the Nash equilibrium tells each of us to benefit ourselves at the expense of all of us. Not saying that is the case for Deep Learning right now, but can (and frequently does) happen.
I hate to be the one to tell you, but, it turns out we are living in the middle of an ecological catastrophe, and it also turns out that means that electricity is a resource we are going to have to conserve.
It’s a resource whose cost is flattening due to the rise of PV and wind power, unless you live in some backward place where they are still mostly coal.
The sum of all DL training in teh world is noise compared to the other big consumers of energy in computing. That's because the main players all invested in energy-efficient architectures. DL training energy is not something to optimize if your goal is to have a measurable impact on total power consumption.