"a Heroku database" was what was known as core-db internally for the longest time. I'm not sure if still the case or not today.
But at one point was the source of everything for Heroku.
Over time things were moved out, so this isn't an everything that exists has been leaked, but it is not a guarantee that attacker didn't move from one area to another.
As someone with some apps on Heroku, having worked there, but no knowledge of the details of the incident more than others... I would:
1. Rotate all creds
2. Ensure logging all connections to the DB (I can't recall how much you can do this on Heroku)
3. Extra heavily audit Github commits and Heroku releases
I feel for the team working on this at this time. I hope this doesn't end up accelerating the culling off of Heroku by Salesforce. One of the smartest and nicest bunch of folks I've worked with.
Completely agree, heart goes out to the team. No harsh judgement of all the engineering team there, not a fun situation and hope they know a lot of folks in their corner.
> but it is not a guarantee that attacker didn't move from one area to another.
The incident notification seems like the customers who are using GitHub integration are the ones who are compromised, If the attacker has gained access to other accounts then it needs to be clarified so that we could take repository level mitigations as you've mentioned; Else most will just reset account passwords and be done with it.
"Access to the environment was gained by leveraging a compromised token for a Heroku machine account"
This is the equivalent of saying "the car was stolen because the car keys were laying on the kitchen table." They still don't know how they got into the house to get the car keys.
GitHub was just one branch that the attacker took to further access, another being the download of the accounts database. We don't know how many other things they did.
The latest report states about "a database" which is presumably the internal database. I don't want to speculate too much, but it seems attacher had access to internal systems. GitHub were the ones that detected and noticed it and reported to Heroku. Do not disagree that there should be more clarity, but best to follow up with Salesforce on that.
There's going to be a question about the expected probability of this across cloud service providers.
I've done security work for multiple cloud service providers and know a lot of people in the industry. I'm not really privy to give details.
I can say: dev teams face limits on what they can build securely, platform teams face limits on what secure by default and monitoring features they have time to implement, security operations teams have a lot of data points to look at, and in theory even changes in personnel in a couple of teams can have an impact on the threat posture for a given set of a company.
People are trying. But if you do the math for introduction of attack surface over time versus risk mitigation effort over time for that attack surface you can derive some estimates for likelihood of attack.
If your cloud provider isn't providing you that data as a customer, it's not simple to make a determination about likelihood of risk introduction and breach of your cloud provider.
This hasn't really been something people talk about because there was a tacit assumption that the biggest companies are mostly getting this right.
I'm not trying to say the cloud is inherently broken, I run a lot of workloads in the cloud and trust sensitive data to the cloud. But I do wish there were better ways for customers to have data points with which they could evaluate platforms besides extrapolating the history of public breach reports.
(Founder of Aptible, a Heroku-like PaaS focused on security and compliance)
> dev teams face limits on what they can build securely, platform teams face limits on what secure by default and monitoring features they have time to implement, security operations teams have a lot of data points to look at, and in theory even changes in personnel in a couple of teams can have an impact on the threat posture for a given set of a company.
I couldn't agree more. It's too bad, because I believe most companies should be solving this by building on a battle-tested platform that provides a safe path for devs. In theory, platforms like Heroku improve cloud security by reducing margin for error. In practice though (as we're seeing), these platforms can introduce new security vulnerabilities in the layer they introduce on top of IaaS.
I also very much appreciate your comment about having a better way to evaluate the security of platforms without relying on public breach reports, or implicitly trusting what platforms say. I think the best thing is for platforms to be 100% transparent in how they implement security, namely by:
1. Running alongside IaaS services instead of layering a black box on top of them (coordinating, not fully abstracting)
2. Providing clear accountability for security defaults: every security default enforced by the platform should be represented in a validation that end users can view (if not alter)
That's how most of our customers use Aptible, yes. That said, we currently have our first customers running Aptible as an integration with AWS, and we believe this will be the most popular way to use Aptible in the future.
With this new product model, you integrate Aptible with your AWS account, and we provide functionality to provision high-level constructs like apps and databases that simply set up and coordinate AWS services like ECS, EKS, RDS, etc. Aptible only needs permission to write to a set of SQS queues in your account. To make sure things stay compliant, we set up AWS Config checks for every security control relevant to your chosen compliance framework(s), and maintain a set of managed IAM roles that you can assign to your dev team to ensure least-privilege access without having to constantly update IAM.
Unrelated to your main point, "I'm not really privy to give details.", that's not how you use privy. If you have the details but aren't allowed to share them then you are privy to the details, but you can't share them.
Privy means "sharing in the knowledge of (something secret or private)", but it has nothing to do with sharing that knowledge with others.
This is something I've been thinking about as a user (provisioner) of cloud services like Heroku, AWS, Google, and VMs from Linode and Digital Ocean. Especially with the potential of state actors trying to cripple large businesses (I don't actually know how serious a threat this is TBH).
Sure they take security further (I hope) than I would provisioning my own hardware, but I also worry about how large a target they are and how much more complex their systems are (increasingly large attack surface).
It's appealing to just let someone else handle my hosting, including security, but I also wonder if I'd be better off running colocated metal. I mean I still have to worry about the hosts network and physical security when colocating, but it's a much smaller attack surface, and also a far smaller target than massive hosts like AWS.
I think cloud provider's might need to level up their security practices, even if it incurs some friction with their customers and staff. I don't even know if it is feasible to evolve as fast as potential attackers from a cost perspective over the long term for many (or all) of them.
Not to single anyone out, but take Digital Ocean as an example. They're constantly adding features to their platform. I assume this increases their attack service? How much weight are they giving to security as they add features? I assume quite a lot, but I don't really know! Another example is AWS, who are constantly growing their features / services (to the point where it's near impossible to navigate all of their offerings!). Every cloud provider is doing the same.
There's also no real accountability. If my database on Heroku (or another provider) is compromised it could have a massive impact on my business, but at most it might mean losing a customer for my hosting provider, and maybe some bad press (not really enough though IMO). So the incentives aren't perfect.
I have no experience in organizations or IT systems that big, but, I can very much imagine that it's not only a matter of not having enough time to check everything, but over time, your systems become so big that it's hard to maintain an overview or, for that matter, control.
I mean, there's been numerous incidents of a random developer having copies of customer data on their systems, or accidentally opening up a database or an Elasticsearch instance to the world.
And I'm afraid the only way to help mitigate that is to restrict what an individual can access and do on the one hand, and bureaucracy on the other. And full-time staff whose only job is to maintain security and juggle access rights around.
Agreed. cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open”. Its so easy to make a resource accessible to anyone, or an IP accessible from anywhere in the cloud without proper restrictions by internal teams in the company; and often the default is to give teams superadmin to “unblock their time sensitive project” rather than maintaining principle of least access which requires more discipline (and thus effort).
Either cloud providers need to assume more responsibility for security or a Federal Agency like the FBI or NIST need to be more proactively engaged in improving the security posture of cloud hosted US corps.
> cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open”
So no different from every VC funded startup (or startup seeking VC funding) then?
The sentiment of imposing tighter regulations around data security feels counter to the general idea that the lack of regulations around data security (e.g. strong data protection laws) are what allows the US tech industry to dominate compared to its EU equivalents.
I'm not disagreeing with this, I'm just pointing out the contradiction and wondering how those who believe the latter would reconcile that belief with demanding the former.
> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation.
Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku?
I'm not asking this rhetorically or in bad faith. It's a genuine question I have based on a project I did. I researched cybersec tech like SOAR, XDR, security logging, and SIEM in depth. On paper, the marketing for such tech gives the impression that by using them, such breaches can be detected and prevented in real-time. But there seems to be a mismatch between the claims and ground realities. If so, why?
Salesforce has been unable to attract or retain security talent. When they acquire a company, they close down the department that does security for that company - and then move everyone into the Salesforce Trust team. Unlike engineering who they typically leave alone (unless they're integrating or rebranding).
In doing so, they typically lose everyone that setup the SIEM and run the SecOps center. Everything "security" ends looking the same.
They don't pay well, executives have pulled talks and fired speakers who do things they disagree with (the same executives are promoted and remain there - no accountability), they've got a pretty bad wrap within the industry.
Let's assume that prior to acquisition, Heroku sec had set up a very secure posture using such tech. Then they lost most of their experienced people after acquisition.
Some questions:
1) Are these tech not enough to enable others - perhaps less experienced, or experienced but not on a particular product - to take over while maintaining the same posture?
2) What kind of additional (perhaps intangible) security does an experienced team add to the posture that gets lost when they leave?
3) As I understand them, things like risk frameworks, NIST CSF, security assessments are all supposed to anticipate people problems (resignations, malicious insiders, etc) and make the posture as independent of them as possible, probably relying on automated tools like XDR and SOAR to do their thing regardless of who's sitting at the console. Does it not work like that in reality?
Btw, thank you for your reply and insights (and to everyone else who replies)! Pardon my probably naive questions. I'm an outsider looking in and having trouble understanding this phenomenon of data breaches in the face of all the tech marketing.
Fundamentally, a security analyst authors detections, reviews surfaced alerts, or identifies hypotheses to investigate. In the case of reviewing surfaced alerts (the firing of a detection which may or may not be authored by the security team), differentiating true positives from false positives is subtle and often requires context or further digging. Of course, this requires time, which costs money, so you can imagine the tension there.
This process can often be subtle, and difficult to automate. In many cases, the issue is automating the economical delivery of enough context to the deciding function that a clean choice can be made. However, even with enough context, and enough documentation, escalating vs suppressing an alert can often be a judgment call. Humans are meat based pattern matchers, and a decade's worth of "ML" and "AI" advancements still not sufficiently precise (as in vs recall) enough to filter out "things that look bad" from "things that are bad, for our specific environment", that knowledge still lies with the security team.
Most security products are snake oil. The vendor selling them and the people buying them (always business execs never engineers) have no interest in how well it actually works as a company is never buying these tools to actually be secure only ever to give the impression to some higher exec within the same company or tick a box on some audit process (which requires something to be in place but doesn't require verifying it works).
It's beyond a joke how the "cyber" industry operates.
Throwaway for obvious reasons. But in my experience Salesforce security org is plagued with incompetent leaders who chase arbitrary metrics that does not improve security at all. At one point in time, security team at Salesforce was stellar and did some awesome work. Dont get me wrong, there are many many smart security engineers still around but their population has been dwindling. This all started when a bunch of new leaders were hired. Instead of promoting tenured smart people, the security leaders decided to bring in their own gang of coworkers from previous employers. At the same time they slowly started pushing out tenured leaders.
These new leaders are typical VPs who have long lost any technical chops and its a huge task to explain any complex technical topic to them. On top of that they don't bother understanding the fundamental business model and just want to push their agenda on to everyone. So they have added "security processes" which requires checking boxes. The more boxes you check the more metric it generates the better the leader looks. These security leaders are so disconnected from the ground reality that they don't even realize that all they are doing is adding hurdles in the path of engineers without improving any security.
I do remember the whole debacle that happened at DEFCON and people got fired for presenting something on stage.
Agreed, there's a lot of interesting stuff that came out of the Security (or related orgs) at Salesforce. Red team tools, chaos tools and JA3 which we use at my current work as well for SSL/TLS fingerprinting.
> GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation.
The three days after being notified to actually revoke the tokens isn't ideal either. Surely if GitHub comes to you and warns you of suspected unauthorised access you'd spend a very limited amount of time and then revoke the credentials to be on the safe side.
I think (based on what I've read on the linked page) the notice was telling Heroku that "hey, someone used a compromised OAuth token to download your source code" not that "the tokens that you are using to read Github repositories of your users are compromised". Both are Github OAuth tokens, but playing different roles. Presumably the compromise of source code might have been used to help get access to the database that had the Github integration OAuth tokens, and realizing that might indeed have taken a couple days.
Because Heroku/Salesforce doesn't have real security. Requiring special characters in passwords and sending out emails that have http and not https links to a password reset page. Their security is a joke.
Heroku _used_ to have their own security team which was quite good and had some scary talented people on it. However, over the last 3 years or so Salesforce has been forcing Heroku to adopt Salesforce's operations practices, and this has not only wrecked productivity but completely destroyed morale and caused many, many of those talented people to quit. I for one decided to quit after only working there for around 8 months due to a horrific overreach by Salesforce into Heroku's operations.
Among other things, Salesforce forced us to adopt:
- their internal ticket tracking system, which _runs in an instance of salesforce_ (barf)
- their slack instance, which lost us many of our customizations and broke a bunch of integrations for weeks (I wouldn't be altogether surprised if this was one of the causes of the delay in notifying Herokai as to what was going on)
- their incident management process, which requires us to notify "Salesforce ops HQ" anytime there's an outage that meets certain criteria.
This last one was especially bad, and meant that we no longer had full agency to act during incident response situations. I had one incident I responded to where the problem got worse while we waited for Salesforce IM to spin up, so that we ended up having what would have been a 10 minute outage turn into a 2 hour outage because the issue got out of control.
In short, the problem isn't the people trying to administer Heroku; they're great folks under a lot of pressure with very few resources. The problem is, and has always been, Salesforce's "leadership" deciding what's best for a cloud platform they couldn't give less of a damn about.
At this point, I'm not sure why SFDC even bought Heroku.. Is there major overlap between CRM users/buyer (salespeople) and Heroku's who are mainly devs, hobbyists and startups (i'm guessing)? Surely they didn't try to buy hero to compete with the big 3 cloud providers?
Welcome to the existential question of Heroku from the inside :) No one knew what the point of the product was anymore. My vibe is that SFDC's goal was to use Heroku as their cloud provider for everything, and when that didn't work (more due to lack of focus than technical issues), they tried to shoehorn Heroku into the Salesforce platform as a tack on feature. It was all very weird, no one knew what we were doing, and everyone was upset about it. I originally joined the org because I was really excited to help the early startup/small business/hobbyist/student sector, and SFDC writ large just did not care about those customers IMO.
I don't want to start a flamewar, but it wasn't an integration; it was a hostile takeover. Heroku was doing just fine without SFDC's interference, and when Salesforce not only refused to believe that employees might have negative opinions about integrating but actively prevented us from speaking up about it, people got rationally angry and left. I remember talking with an architect on the Heroku side about how he felt about all of it, and he told me that it wasn't presented to him as a choice, and senior leadership was convinced that Herokai would see this as a good thing despite his (and others') warnings.
This isn't the first time Salesforce Cybersecurity has left us in the lurch while they perform damage control.
On 17 May 2019, Salesforce performed maintenance on their databases that clear permission sets for users. My team was able to piece together that the incident happened at about 0200 CDT, and Salesforce didn't take ANY noticeable action for at least 9 hours when they locked all customers out of the platform. Salesforce "fixed" the issue, which meant our Admins had to go in and reapply a bunch of profile settings...no big deal, right? Just a little bit of work for everyone to fix their own accounts. Salesforce acted like it wasn't a big deal.
Wrong.
If you were a Salesforce customer that built a tool using the Portal or Community tools Salesforce provides for external users, there was a 9 hour window when a customer could log in and instead of seeing the data you were sharing with them, they would see all data for all users. The permissions that indicated that a user should only be able to see their own data was gone.
The only reason we knew about this was because we were paying extra for advanced logging. We were able to see a few of our users logged in during this time and looked at customer records they should not have had access to.
Salesforce stood fast that exposing data through their Community and Portal tools this did not constitute a breach or even a violation of their SOC-2 Type II compliance. We were lucky that the only people that had access at the time were licensed partners. Nevertheless, our users lost their jobs and were stripped of their licenses.
Anyone that was using those tools at the time for any sort of direct customer interaction that shared order history, customer engagement, referral programs, etc. was not so lucky; doubly so if they weren't paying for advanced logging and/or didn't know what to look for. Salesforce was more concerned about covering up their mistakes than they were about telling their customers that there was a problem.
Seeing the Heroku notification page gives me PTSD. This looks all-too-familiar to me and I sympathize with those affected by this. I still feel like they were negligent back then, and I wish I knew who to tell to warn others.
> Nevertheless, our users lost their jobs and were stripped of their licenses
Why? Maybe I'm assuming good intentions, but a) did they know they were seeing records they shouldn't be; b) did they report that? Even with yes and no, firing for that seems a little too much.
Salesforce’s monitoring, availability, incident handling, transparency, and accountability, are abysmal. We too we’re affected by the incident. I sincerely hope they lose some of their compliance certifications, because their behavior is unacceptable.
> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts.
What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?
I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.
The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.
For any individual/organization remotely competent at security (i.e. not using Passw0rd! as password and reusing elsewhere), a leak of hashed password is probably the least of concerns. Compared to this anyway:
> According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.
former Heroku employee, though long long long ago, with no specific knowledge about this incident, but...
We did so much work in open source it was just easier to assume everything was always publicly viewable, or that what you were doing now might be open sourced in the future along with the full commit history. Whether something was private or public was more a business decision around competitive risks and not a security-led one. To that end I'm far more concerned about a database and passwords getting popped.
But who knows, a lot can change in 10 years. Maybe private repos being exposed is also very bad.
This is also why folks advocate for a separate auth system (keycloak was just mentioned here on HN) since they are different types of information.
>I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.
I feel like they are heading this route as well. Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.
> Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.
Trust takes long to build and is easy to break. For anyone able to convert the Heroku buildpack to a Docker and able to move the database, moving away from Heroku shouldn’t be too hard. There are multiple similar services nowadays.
We moved basically everything but username into an entirely different db, went so far as to hash the username column so we don’t even know what’s it is until you log in.
Is this giving you any real security benefit? (I'd assume the usernames are indexed elsewhere and that it's a reasonable assumption that whoever gains access to this hashed data has access to the username list as well, making a lookup trivial - or are these not safe assumptions?)
Not GP, but one architecture where there would be a difference is there's a distinct identity provider, responsible solely for exchanging the user's long-lived username/password credentials for a short-lived ticket.
Yeah, the comms around this has been very concerning. Do I need to rotate every config var on all of my apps? Re-install every add-on?
While the nature of what limited things they had disclosed to date pointed to this situation part of me wanted to believe it wasn't as bad as I was assuming. And now the trendline on this suggests I should have already done everything I've outlined above. And I'm low confidence anybody is going to be proactive in telling me until it's absolutely obvious that these things have been compromised and exploited.
IIRC the environment variable settings are encrypted in a physically separate database. However it may be a good idea to rotate your secrets anyways. My hunch would be that there are so many "juicy" targets on Heroku that you probably don't need to worry too much right now unless you are or work for a "juicy" target.
Why would the Github API keys not fall into the same separate database and be encrypted as well? It's especially baffling if they already have an example/process of doing this properly.
> What else was in this database? Typically the password field is stored alongside the rest of the user record.
That is where you are supposed to have encrypted as many data fields of the record as possible - in addition to the conventional database encryption which encrypts the database as a whole.
The bigger question to me is how did they leverage a GitHub OAuth token to gain access to an internal database unless they're storing that config in their codebase.
> I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up
They can't, they surely have EU customers so have to follow GDPR disclosure rules, which they might already be afoul of.
My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all.
Who knows how long Heroku's internal systems were compromised.
it might be that it took this amount of time to establish the facts of the events. If they recounted an incorrect version early, it might do more damage than not telling it.
"On April 13, 2022, Salesforce Security was notified by GitHub that a subset of Heroku’s GitHub private repositories, including some source code, was downloaded by a threat actor on April 9, 2022. Based on Salesforce’s initial investigation, it appears that unauthorized access to Heroku's GitHub account was the result of a compromised OAuth token. Salesforce immediately disabled the compromised user’s OAuth tokens and disabled the compromised user’s GitHub account. Additionally, GitHub reported that the threat actor was enumerating GitHub customer accounts using OAuth tokens issued to Heroku’s OAuth integration dashboard hosted on GitHub. Based on the information GitHub shared with us, we are investigating how the threat actor gained access to customer OAuth tokens. The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts. With the access to customer OAuth tokens, the threat actor may have read and write access to customer GitHub repositories connected to Heroku. Given the incident is still active, please review the recommended actions provided below."
You’re missing the 5/3 update about username and password credentials.
> our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts.
This has been an ongoing security incident communicated through multiple channels since the day that github announced it. I've got a dozen emails or more in my inbox, the heroku dashboard includes mention of it, and the status page includes information about it.
A lot has changed at Heroku in the past 8 years since I left, particularly in the direction of being subsumed into the greater Salesforce org. My working assumption is that everything left there is being done "the Salesforce way" at this point. Take that to mean what you will, but it seems pretty clear we're long past the days of openly communicating with customers as quickly as you have relevant/important information to share with them.
Salesforce informed you that their entire DB was compromised, and not just the Github OAuth tokens, as they've been saying for weeks? The first indication that anything except a Github specific DB was compromised was Tuesday, when they started telling people that seemingly-all non-DB non-addon credentials were going to roll.
It seems as though enterprise customers of Heroku that leverage data products tied to their Salesforce instances, such as Heroku Connect and Salesforce Connect, are at risk of serious data breach and possibly haven’t realized it. Nearly a month since the initial 2413 incident, we’re still learning of its full scope. If customers haven’t rotated credentials — mistakenly thinking the incident is isolated to web apps and GitHub — the risk is still present. Can it go any more sideways? Yet the communications from Heroku and Salesforce are disjointed, even amateur.
>On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.
...
> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.
Well, it only seems to be getting worse on this one.
I’m keen to get off Heroku, but waiting for one of the newer alternatives (Render/Fly+others) to implement WAL point in time restore for Postgres. It’s the only thing keeping me on Heroku now, but is indispensable.
Anyone here from them have any update on when we could see that feature made available?
I recently spun up Google Cloud Run as an alternative. GCP has PITR for postgres.
The only weird hitch with Google Cloud Run (and their other serverless products) is you either need to either use public IPs for e.g. memory store cache or other things in your network (minus your database), or set up a VPC Access Connector [0]. Admittedly, that was easy once I realized I needed it, but was very annoying to figure out (because django's default redis socket timeout is never...).
You also don't get SSH access, as it's fully managed. K8s is still the easiest version where you get that, unfortunately.
I’m also interested in this. I have 4 decent sized applications I want to migrate, but I don’t have the resources to self host on GCP or AWS. K8s is nice in theory, but I know from experience it requires too much hand holding (let alone other managed services like a database and cache).
From what I’ve seen in the market, fly, render, and railway are the only real Heroku competitors. All seem to be missing a few critical pieces of functionality that is preventing me from migrating. Railway doesn’t appear to allow me to remotely run Python commands, they also don’t have automated nightly backups. Fly and Render respectively have a handful of missing features. It would be amazing if one of these services came out with 1:1 feature parity with Heroku. I’d move in a heartbeat.
Azure web apps is pretty similar to heroku and I find it very easy to use. It has had a Linux version (instead of Windows) for quite a while now and it's pretty solid. Haven't had many real issues with it in many years of use.
They also have managed postgres db with very powerful tools.
You should check out tools that help bring a Heroku quality experience into your own cloud, where there are lots of database options that meet your needs.
Coherence (disclosure- I’m a cofounder) - https://www.withcoherence.com - is one option. A defined workflow for production-quality full-stack web apps with dev and production built in alongside automated test environments, including CI/CD and cloud IDEs - all configured with one high-level YAML. We’re in a very early private beta on google cloud right now - if you’re interested, please check out our site above and let us know!
I really feel like people don't value intrusion detection enough. Why was Heroku's intrusion detection system "rely on Github's intrusion detection system"?
That email made it clear that Heroku lacks fundamental knowledge about security. I’m sure they lost some enterprise customers, I know I don’t open accounts on websites with silly password complexity requirements.
Google allowed 6 character passwords for a while, and didn't expire them when they increased minimum to 8 for google workspace accounts. This has been fantastic, as users can remember their password forever even if its higher complexity (google does a password strength eval). No rotations either.
I'm pretty confident google will pick-up someone trying to brute force a 6 character password. That google will notice connections from new / different IPs or browsers. That's because google asks for my 2FA in various situations but doesn't annoy me by asking for 2FA all the time.
I use one govt system that has something like a 14 character password requirement. For even more security if you don't log in for 90 days your account goes inactive and the password EXPIRES! Very secure you say? Well, to regain access you have to provide the answer to a security question - favorite pet! That's a 5 letter word that doesn't change (and is probably pretty guessable).
Here is another example:
"(b) Information systems must be designed to require passwords to be changed not less frequently than
every sixty (60) days." - SBA IT Security Policy - 90 47 4
Right, we've found actually simple passwords but with the mandatory 2FA turned on works really well. The 2FA google uses is a gentle touch in most cases (can persist on a device for 30 days).
Google has nice 2FA controls. In a workspace setup you can actually tweak them to match your needs because the lockout / reset path (was) pretty reasonable (when it was onsite). Ie, we could disable certain methods and for some higher security groups you can provide hardware keys and then turn that group up a bit.
Never had to rotate passwords and users are glad for that I think.
I do wish google offered "Cloud Chrome" for admin staff to open email / click on links etc. Basically a remote VM with chrome but no file access directly.
I reset my password and then closed my Heroku account today. What a sad end for something that was once a model for software deployment and developer experience.
Obviously Heroku have handled this horribly - but are any small startups out there considering replatforming? Still seems like a lot of hassle and the competition I've tried (Cloud66, excid3's thing) haven't been as good.
I'd maybe use this as an opportunity to prioritise moving everything over to AWS.
I'm sure for some it may be an unwieldy amount of work but for others (depending on tech stack etc.) it ought to be fairly doable. In the long run it'll save money too.
I’m interested in peoples experience with this and if it’s relatively true.
We (like many others I assume) pay more for Heroku than AWS as it allows us to “outsource” our dev ops. We are a small team (sub 15) with a decent sized, decade old app. We’ve had it on AWS before (and used platforms like BuildKite) but both required much more overhead (in terms of employee salary). Anecdotally I’ve heard the same from friends, though I understand AWS works well and is cheaper if you know AWS well.
> I’m interested in peoples experience with this and if it’s relatively true.
My personal experience is that AWS is always more expensive. You don't use AWS because of the cost saving you use it because:
* Top of the line h/w
* Always the first to bring out new features
* Very high availability of resource, like seriously I've never had a time even during the pandemic where they struggled with resource availability
* Resilience of AWS systems are very high
* Very good support
There was an article a few months back where a teams tried moving from Heroku to AWS to save money and ended up spending nearly 3x the amount. Heroku give more resource than you actually pay for by default and it turns out they were using the extra resource during normal operation. When they done their calcs for a switch to AWS they used the quoted resource that Heroku say you get and there system died due to being under resourced. They had to up the resource which pushed them well over their budget. I'll try and find the article.
AWS isn’t that bad when you use IaC like CDK. Clicking around in the console and trying to reproduce those steps when you have a new project or API is what will kill you
I looked at Render but they move data out of the chosen region (so out of EU) and that is a huge issue for our clients. They also proxy through Cloudflare which is another big problem when you are dealing with sensitive data.
The main problem is that data is transferred out of EU, it is simply not acceptable for a EU company regardless of any DPA (history have shown here that any such DPA would be invalid in the near future).
However some of our clients (for example in health or financial industry) would be concerned that data is proxied through a third-party, doesn't matter if it is encrypted (also it's unclear to me how keys etc are managed and what data Cloudflare can access).
btw: Render looks awesome but at the current offering is it not an option for us.
Understood on data transfer out of the EU. It's high up on our list. Also very happy to share details on CF encryption over email (support@render.com).
How does a company like Salesforce mess this up for such an extended period of time? I understand that companies can make mistakes early on in a critical incident but this has been going on for weeks?!
Root cause analysis must have failed pretty bad. I worked at Heroku and with how much headcount they've lost my Salesforce starving the beast over the years, it's not surprising that it would take this long for them to react like this. When you scare away the best people with Salesforce policies, their Jira clone built on Salesforce, and overall being difficult to impossible to get more people on the team; yeah people will pack their shit up and leave.
Given how much of a web of interdependent and undocumented pain heroku is, I'm not surprised it's taken this long. A security team without any context to Heroku must have had to trace through everything system by system. Especially if core-db got popped.
I've used the DO App platform, generally it's pretty solid but getting up and running was miserable. The documentation is really thin and there's a bunch I just had to discover along the way, in particular while getting up and running with a DB as it's not clear how the certs are delivered into the environment variables etc.
It's been solid since I got up and running but took me about a day to get a simple CRUD app moved over to it.
If they can improve the documentation it'll be much better. Right now I'm not considering it for future projects though just because platforms like Fly.io and Render seem to have better docs and additional functionality that DO Apps doesn't have yet.
Heroku alternatives have gone in a few different directions:
- next-gen PasS that are more opinionated and offer wider range of services. Also can be cheaper. examples are Digital ocean, Railway or render
- Performance-focused PaaS like fly.io
- "Heroku in your own cloud" like porter.dev, architect.io or quovery
- k8s tooling like garden.io, ReleaseHub, even gitlab - these are often geared more towards internal DevOps teams at larger orgs when compared to the very low-lift PasS providers
- serverless providers - like cloudflare functions, AWS lambda, GCP Cloud Run
- Vercel/Netlify - SPA + serverless with a great developer experience
- Replit - kind of in a category of their own but they have integrated hosting/datastores/user auth
Lots of awesome products here, I'd argue that only replit is a true 10x change from the Heroku innovations in terms of providing a next-gen developer experience.
I'm the cofounder of a new company called Coherence, that we think creates a new direction and offers a better platform for the next leap forward. By integrating from dev to prod and capturing the whole SDLC, as well as by operating in your own cloud, we're focused on delivering the best developer experience possible, without compromising anywhere else. Check us out at https://www.withcoherence.com. We're in an early closed beta so not yet a fit for all teams, but feedback is welcome!
(Render founder) Stuff like ^ worries me even more. What keeps me sane is our engineering team obsessing over reliability, and learning from every single incident, no matter how small. We're improving every day.
rule of thumb: company A acquires company B, then company A does not change to become like company B, company B changes to become like company A. there are exceptions, but few. I think Slack's communications are going to become more crap, personally.
Cheers to admin for changing the title, but old title better reflected the last update (and I used it because Heroku doesn't have a permalink to each new update).
I'm interested if application ENVs were stolen. If they stored Github APIs in plain text I'm sure our ENVs were too.
I went ahead and reset my planetscale passwords and moved my app out of Heroku. But I only had two apps hosted there I feel for anyone with a large number of apps on there.
Always interesting to me with events like this that the actions or intentions of the "threat actor" are never discussed. The conversation is always finding tiny holes in the victim's systems and admonishing them for not being prepared.
Actions other than... (1) obtained access to a Heroku database, (2) downloaded customer GitHub integration OAuth tokens, (3) enumerated metadata on customer repos with the OAuth tokens, (4) downloaded some Heroku private GitHub repos containing source code, and (5) exfiltrated customer hashed and salted passwords?
I have seen nothing but bad news about Heroku recently. Albeit nothing provides that much ease to use on a free or affordable tier when you have minimum computing demands. I use it as my fundamental cloud provider for proof of concept stuff, so does many people.
Salesforce is doing a terrible job managing this lucrative platform. I have no idea why muck up a good service like that. They have some plugins and a postgres connector but the drive to innovate, the drive to even care stops there. All these news act as a reminder that I should move my code to a "real" cloud provider.
> Access to the environment was gained by leveraging a compromised token for a Heroku machine account.
Any idea if this involves AWS EC2 Instance Roles? It’s incredibly convenient, but has got to be the scariest feature to enable on a platform that allows arbitrary user code to execute.
last I've seen the Oauth permissions for the Heroku Dashboard given by Github are excessive and include write access to all public repos - as read-only is not an option if I recall correctly, see https://github.com/dear-github/dear-github/issues/113#issuec...
Newer integrations like Github Apps are more granular and can restrict the scope , also ssh deploy keys are an option for other purposes, but specifically the tokens issued for the Heroku Dashboard can write to the public repos of a user or org.
Why not just say hacker ? Know we are on hacker news and hence it is semantically not exactly right. But threat actor … sound more like threaten actor. Just a movie star or drama queen. If one say Heroku was hacked or just sales force … I know one want to manage but somehow the title is not exact right. Too pr.
But at one point was the source of everything for Heroku.
Over time things were moved out, so this isn't an everything that exists has been leaked, but it is not a guarantee that attacker didn't move from one area to another.
As someone with some apps on Heroku, having worked there, but no knowledge of the details of the incident more than others... I would:
1. Rotate all creds
2. Ensure logging all connections to the DB (I can't recall how much you can do this on Heroku)
3. Extra heavily audit Github commits and Heroku releases
4. Maybe keep rotating all creds?