Author here, surprising to see this blog post of mine trending, but it has held up pretty well! Happy to answer any questions about my time at Stack Overflow.
I've since moved on to Heap (https://heap.io/) and we are hiring. We're virtual first so we don't have private offices, but we have the same values about treating developers well and giving them the space they need to do deep work. We're hiring for basically everything https://heap.io/careers/departments/engineering-product-and-...
This is exactly what we did to diagnose (source: I was on the call). The only tricky part was figuring out which post it was, since it wasn't in the stacktrace. To do that, we grabbed the 3000 most recent posts and ran the regex against them. By that point we already had the code fix (another dev working on it in parallel), but if we hadn't we also could have gotten back up by just deleting the post.
Kudos for thinking clearly under tremendous pressure. A production server down is always a highly stressful situation.
If time pressure is not a factor and the production server has the VStudio or WinDbg installed, attaching the debugger to the process can see the data related to the post. But the symbol file and source files might be needed; it's just more hassle to set up. For stressful situation, simple steps and simple tools are more useful. Whatever works.
If it was malicious they would have made a bunch of them, not just one. I personally have seen many files with unreal amounts of whitespace at the end.
Yeah, that's kinda what we're thinking. If it were malicious, the question itself would be unlikely to actually look like a real honest question, too. It'd just be a bunch of garbage input.
Good news, I actually do know everyone who was part of the original build of Wasabi! None of them left because of the language. I think this accounts for everyone who was there at the time:
1. Original author left because his wife was going to medical school out-of-country and Fog Creek didn't allow remote work at the time.
2. Second author left because his wife was going to medical school out-of-state and Fog Creek didn't allow remote work at the time (see a pattern?). Later came back because Fog Creek offered remote work. Went on to author the blog post we're talking about.
3. Developer left to go work on Stack Exchange (me!)
4. Developer left to go make the world a better place at Khan Academy
5. 2x developer left to go work on Trello
I think that was all of us. People move on in the course of 5+ years. Turns out most of those reasons don't have to do with programming language.
FWIW, I think Wasabi was a bad decision and I'm not going to defend it. But I really don't like these massive assumptions about people's motivations for leaving.
Can I guess at why you think it was a bad decision?
(a) Too incremental to be worth it, given where the .NET ecosystem was heading
(b) FC couldn't commit the resources required to adequately support a whole language, and it's better to commit to a lower common denominator than limp with a poorly supported language
(c) If you're going to create an additional obstacle to on-ramping employees, it had better be something every project in the company takes advantage of --- like, even if you had built FogBugz in OCaml, that would be a problem since the company is not designed to take advantage of OCaml.
(d) Unless you're getting a truly transformative advantage from a custom language, it's not worth it to be out of a "Google your way out of most problems" mainstream sweet spot
(e) No matter how good the language is, using a different language makes you incompatible with toolchain, so edit/test/debug cycles are needlessly painful
I obviously have no idea if Wasabi was a good decision or not, but a workplace where people are allowed to deploy basic computer science to solve problems is (sadly) an attractive stand-out to me.
So, I'm not David, so I'm not going to pretend to know what his thoughts are, but I'll say that I've always had really mixed feelings about Wasabi.
Let me start by saying that Wasabi as a strategic move was brilliant. If David disagrees there, I'm a bit surprised: FogBugz represented an awful lot of battle-tested low-bug code, and finding a way to preserve it, instead of rewriting it, made one hell of a lot of sense. I'm with you that the general thoughts in this forum that we'd have to be insane to write a compiler are misguided. Wasabi let us cleanly move from VScript and ASP 3 to .NET without doing a full rewrite, and I'd be proud to work at a place that would make the same decision in the same context with full hindsight today.
That said, I think Wasabi made two technical decisions that I disagreed with at the time and still disagree in with in retrospect. First, Wasabi was designed to be cross-platform, but targeted .NET prior to Microsoft open-sourcing everything and Mono actually being a sane server target. At the time, I thought Wasabi should've targeted the JVM, and I still think in retrospect that would've been a much better business decision. I really prefer .NET over Java in general, but I know that it caused us an unbelievable amount of pain back in the day on Unix systems, and I think we could've avoided most of that by targeting the JVM instead. Instead, a significant portion of "Wasabi" work was actually spent maintaining our own fork of Mono that was customized to run FogBugz.
Second, Wasabi worked by compiling to C# as an intermediary language. There was a actually an attempt to go straight to IL early on, but it was rejected by most of the team as being a more dangerous option, in the sense that maybe three people on staff spoke IL, whereas pretty much everyone could read C#. I also think this was a mistake: the C# code was not human-readable, made debugging more complicated (VS.NET had something similar to source maps at the time, so it wasn't impossible, but it was very indirect and quirky for reasons I can get into if people are curious), and that decision meant that Wasabi had all of the limitations both of its own compiler, and of Microsoft's C# compiler. IMHO, these limitations are a big part of why the ultimate move away from Wasabi was even necessary in the first place, since they increased both the maintenance and developer burden.
So from my own perspective, I think that Wasabi was a mistake in that, if we were going to go to C#, we should've just got the translation good enough to really go to C# and then ditch Wasabi; and if we weren't, we should've actually owned what we were doing and written a genuine direct-to-IL compiler so we'd have more control over the experience, instead of going through C#. But I still really do genuinely believe that our going to Wasabi was a brilliant strategic decision, and I think Fog Creek would have suffered immeasurably had we not done it.
I'm particularly interested in your thoughts on Wasabi compiling to C# rather than CIL. What characteristics of Wasabi led to the C# output being suboptimal for human reading and editing? If a compiler is going to output human-readable code, are there any general design pitfalls to avoid?
To add to Ted's comment, the main mistake we made in generating readable C# from the start was using `System.CodeDom` as our code generator, which explicitly does NOT care how readable your output is.
A better idea would have been to hand-code the generator, though of course that would have been a lot of string manipulation as well as a little extra effort.
Roslyn solves both of those issues for us, but it didn't exist until very recently.
Beyond what tedu and krallja pointed out, the debugging required inserting tons of #line markers in the C# output. But a single line of Wasabi could map to multiple lines of C#, making the definition of stepping ridiculous. Throw in that Wasabi necessarily grandfathered ASP globals that C# lacked and you also had fun variable display.
The semantics of wasabi (VB) and c# are slightly different. A fair amount of the code was actually the result of various code generators. It dumped everything in one giant file (delimited by #file markers, though). Nothing intractable, but nothing high priority.
Having done something similar, but entirely different, several times, I'm surprised you didn't choose to slowly refactor the code to be more and more native C# over time. iYou start with 100% Wasabi / 0% C# and slowly work up the native C# parts, in code units, until you reach a level sufficiently high to feel confident to do a final push to switch entirely to C#.
(In my experience, you need to build up an inter-op layer first to make working in C# somewhat sane, but it's usually not hard to identify the necessary helper modules needed. Having the .NET runtime actually is a boon here since the IL is designed for inter-language inter-op.)
Why did you find yourselves maintaining a fork of Mono (versus fixing upstream)? Was it something like forking, although being problematic, had lower impedance than doing the necessary rituals for getting your changes accepted upstream?
You can't exactly tell customers to go check out the latest subversion head and compile it themselves. Changes were pushed upstream, but that doesn't push them to customers. Neither could we ship nightly builds, because who knows what changes get introduced? So we had a fixed release with a pile of patches on top of it.
The transparency of the window is a weakness. Some folks have started putting up posters and curtains.
But the real issue is that they transmit basically all the sound. Fortunately there usually aren't conversations going on just outside people's offices, but if two people are talking while walking down the hall or the person in the next office is yelling at folks on hangout, it's not much quieter than a cubicle.
Oh crap! "Private" offices with glass doors, yeah. So that you never immediately see who enters your office because the door is behind you and you have to turn around. Also with monitors visible to everyone who passes by. This is way worse than any open plan office.
Polished concrete is beautiful and striking, of course, and at some point software development became a creative profession rather than an engineering one, and we stopped mocking people who put form over function.
I used to think this was right (trained in the school of Joel), but now I'm not so sure. I look at it this way: the best predictor of future success is past success. School is pretty non-predictive of your success in industry. General "intelligence" is as well. The best thing I can look at is: do you have a track record of tackling problems and delivering solutions? If you do, you'll probably keep doing that. If not, you're at best a gamble.
I've since moved on to Heap (https://heap.io/) and we are hiring. We're virtual first so we don't have private offices, but we have the same values about treating developers well and giving them the space they need to do deep work. We're hiring for basically everything https://heap.io/careers/departments/engineering-product-and-...