Wow, that is pretty substantial difference between AWS and Azure. I’m pretty confident Microsoft is hard at work getting these numbers better, but it shows just how lonely AWS is at the top. I’m pretty deep into AWS-land but I’m rooting for the boys & girls in Redmond; more competition == better service.
Concur; despite being so deep in AWS I've drowned here, competition makes all of the offerings in this space better. It'd be nice to have a migration strategy in my back pocket if I needed it; today that's a bit of a pipe dream.
It’s also difficult to run true x64 bit workloads on azure functions as it invokes the x86 version of the dotnet executable, regardless if the application is set to x64
My experience so far on Google Cloud has been just as bad. I want a simple app that ingests DMARC reports.
- Google Cloud Functions can't ingest e-mail. Also, JS lockin if you care about which platforms are supported.
- Google App Engine can only ingest e-mail in the Standard environment. OK, fine. But the standard environment regularly takes almost a full minute to load and boot my app, after which the request is killed for taking too long. Additionally, the sample code (!) for opening new Stackdriver contexts breaks with a totally inscrutable error message.
You can say "well, maybe your app sucks", but it's been running on AWS (SES + Lambda) for months and every aspect about it has been painless.
I don’t know how magic is in Cloud Functions, but I personally set up Mailgun postbacks in AWS and it was a huge pain in the ass to configure correctly. I’d love I know if google does something differently that would have been easier.
But the parsing code already works fine, that’s not really the hard bit. Then I get all of the lock-in for questionable benefit and something that Just Works on AWS.
GCF still suffers from JS lock-in, as they currently only support Node, thus you can't make a .NET comparison in the first place. That being said, of the various offers I really like the UI and the like.
If you're going to be handling http requests it's almost a given you'll be using AppEngine rather than Cloud Functions. Still "serverless" but more tools and languages to choose from.
Not the flexible environment iirc. Which is basically the reasonable env to expect, now. Standard env has some aging issues that keep getting worse. most notably jvm 7, if that’s your poison, but just all round crippled env lacking standard libs and with lock in substitutions.
> If you're going to be handling http requests it's almost a given you'll be using AppEngine rather than Cloud Functions
Why? Cloud Functions support HTTP.
I mean, if you can stay within AppEngine’s free tier quotas, there's a clear reason to use it, but otherwise it would seem to be a matter of weighing pros and cons, with HTTP alone not deciding that you should use AE over CF.
This face-off will be a lot more meaningful in a couple months when we can have both serverless platforms running on .Net Core 2.0. I do look forward to reading the results of that updated face off.
I did try the preview version of Core 2.0 on Azure Functions but it's absolutely woeful at the moment and the dev team confirmed they've done no performance work at all. As I recall the single request response time was around 15ms faster for Core than for full fat framework but the response to increased load was dreadful - never falling below a several second response time with the 400 concurrent user test and errors creeping in.
I largely set off down this path as even in isolation the Azure numbers were concerning and not at all what I expected from a serverless architecture and so I wanted a reference point.
As soon as I get an update from MS about Core performance work or it leaves preview I'll be rerunning the tests as I'm very aware that doing this across the two runtimes is less than ideal.
However from what I've observed the issues look to be more about the broader architecture used for Azure Functions than the framework runtimes. I'm hoping to learn more from Microsoft.
In general, from the perspective of a dev in a MS-shop, AWS gives you a bit more 'responsibility' for infrastructure than Azure, but that level of control lets you get further in practice sooner.
Azure has a lot of great things going for it, but I've had a lot of issues getting newer tech out on it without considerable delay. I hope both Azure and GCE give AWS a run for their money in coming years :)
The biggest issue with Azure functions is the pricing model where you have to provision and pay for servers to execute your functions... How is that serverless?
Does anyone know if any of these services provide the ability to cycle between IPs? One of the use cases for Lambda (and now Azure Functions) is in web scraping where companies get mad if you access them too many times from a single address. Lambda made it clear while they do use a lot of IPs you're not guaranteed to get distribution among them.
Seems like a long shot but I figured this is a good place to bring it up.
I have the same question. Not for scraping but for APIs that are rate-limited by IP address. Like some Google APIs. I certainly wouldn't count on IP distribution from getting around limits for long. Anyway, I'm more interested in understanding how, if at all, we will have to tweak our client code to be well-behaved.
Interesting article, but you don't mention what tier of AppService plan you tested against in Azure (or what memory limit your Lambdas had)? I would also be curious to see the same comparisons, but with Azure Functions in consumption mode which, despite the perceived drawbacks you mention, would really be the closer apples-to-apples test, wouldn't it?
Does AWS have first class fully production Python yet? In Azure a single import (pydocumentdb) took a function up to 20-30s on Python, I guess they're serious about it being experimental still sadly. Rewriting in Node it takes 50ms for the entire run, crazy difference.
Have to agree with that part, and am in no way taking away from the commendable engineering on the Azure side. But as a developer, it makes more sense for me to invest time and effort getting good at the platform that innovates faster - and the pace at which AWS has been getting out new functionality in the platform has been breathtaking.
so does this mean that the cold run startup time thing has really been improved enough to build web applications? To me that is the biggest killer in moving forward, not knowing what the end users experience is going to be like because of cold startups that forces a user to wait a long time.
From the article's data, it's really tempting to use AWS Lambda now given this benchmark.
I would say leave the average in but also show standard deviation and 95th percentile average. Of course it is still very unlikely that Azure Functions would come out on top in this case.
You are arguing against yourself here. Having the mean and 95th/99th percentile is going to give you a better picture of the distribution than mean alone. Mean alone actually doesn’t give you ANY idea of the distribution. I don’t know why OP is being downvoted, managing to 95th percentile is very common practice and I also thought was a missing addition to the article.
It’s not the best. There’s a Russian joke that goes “One person is cooling in the morgue, the other is running a high fever. The average temperature in the hospital is 36.6 C though”
Averages are very susceptible to outliers and don’t give a proper insight into your data. What you really want is: how did half of my requests perform, 90% of requests perform, 95% of requests perform, 99% of requests perform
: a number that is calculated by adding quantities together and then dividing the total by the number of quantities
: a level that is typical of a group, class, or series : a middle point between extremes
Full Definition
1 a : a single value (as a mean, mode, or median) that summarizes or represents the general significance of a set of unequal values
b : mean 1b
2 a : an estimation of or approximation to an arithmetic mean
(...)"
It's common to see arithmetic mean and average to be treated as synonyms - but the mean is really just a type of average.
Colloquially, median and mode are used as average more than you'd expect. For example, we might say "the average joe is right handed", which is most likely the mode.
this is absolutely not a function of Azure vs AWS, the perf diff is completely between .Net Framework (which was used on Azure) and .Net Core (which was used on AWS). This compares Apples and Watermelons.
