I'm going to mention what I think the other big definition of magic is, because I haven't seen it spelled out yet. It's when there is something that is occurring in code but you have no handle on how to follow it back to where the magic occurred. For instance, you write a class that looks something like
class A:
thing1 = 0
thing2 = 0
but somewhere in between your definition and your usage, the class is suddenly an ORM mapping onto a database. As cool as that may be, just looking at class A, you have no idea where to look for that behavior. If you had to subclass something to get that behavior, you could have looked there; if you had to call a function on the class to get this behavior, you could follow it back from there, but when you have this sort of magic you can't. The forensic trail has gone cold.
If you think about it, you can see how the two definitions are related; this sort of magic tends to stay something you don't understand because you don't just naturally pick it up over time, you must make great effort to figure out what the problem is. But it's a different problem with a different solution, certainly.
Generally the two places this magic lives is in some all-encompassing framework, and in the language itself. Perl has numerous magical constructs, some of which turn out to be very difficult to even search for in the docs because it isn't even obvious what the operator is. The flip-flop operator comes to mind: http://perl-tricks.blogspot.com/2007/01/flip-flop-operator.h... If you encounter that for the first time you can be really lost trying to figure out how the list construction is occurring and it's not at all obvious what to search for.
I've seen some backlash against 'magic' lately and it surprised me. 'Magic' doesn't need to be understood to be used.
I don't care how my car works, just that it gets me to work every day. If it breaks day, I have the choice of learning another craft (car repair) or paying someone that already knows it.
Likewise, if I find a bug in Ruby's base code, I can choose to learn enough about Ruby's code to fix it, or write a bug report and let someone else.
I'm not advocating ignorance, I'm saying it's a perfectly valid option given that we aren't immortal.
But i do want to learn more about how my car works, theres very little technology at home that i haven't seen disassembled(washing machine, microwave oven, tv, pc, mp3 player, cell phone). I've at least skimmed over parts of the source of a lot of software i use daily(emacs, slime, sbcl, HN). Even though i barely understand the electronics in my tv, and even though I've barely scratched the surface of emacs, i still prefer a more hands-on approach to the things i use in my daily life. I've been like this since a very young age, apparently i didn't have any toys that weren't destroyed by my curiosity :) I had the impression that this was true for a lot of people on this site.
As mentioned in the posts abstractions are leaky. You can't drive a car without having a rough knowledge about how it works or how forces act upon your car but if you drive a hybrid you don't have to know how the engines interact nor do you need to know how energy is stored when you brake. If you are interested in that you take a look otherwise you accept that it just works.
Actually, all I need to know is the controls and the daily maintenance, like putting gas in it. All the rest can be left to a specialist. I don't have to care anything about why pushing 1 pedal makes it go forward and the other slows it down. Or how turning a wheel changes the direction. Or how various buttons activate various features.
They just do, and that's good enough.
(I'm not actually ignorant of these things, but it certainly wasn't necessary to learn them. I chose to. It hasn't really helped me much, except to satisfy my curiosity.)
Not only aren't we immortal - the stuff that you might want to learn is probably being produced at a rate that exceeds anyone's capacity for education and retention.
Nick, I'm reading two really cool books that you might enjoy when you're in a "bottom-up" phase: The Elements of Computing Systems [0] and Computer Systems: A Programmer's Perspective [1]
CSAPP is unfortunately a hulking $100 hardback, but the authors have put close to 200 sample pages online. Despite looking a bit fearsome, it's extremely clear. And TECS is just awesome--they take you all the way through building (in a hardware simulator) a simple 16bit computer!
I've been toying with the idea of a series of blog posts (or possibly even a separate site) that give a bottom-up explanation of all the stacks we work with - starting with electronics => circuits => computers => programming languages. It would be a way of teaching myself the material and provide a useful resource to others.
Using TECS to go from NAND gates to assembly is 4-6 evenings worth of work and totally worthwhile. I have a rough idea how compilers and operating systems work but until TECS everything below that was run by little laptop pixies.
Haha I love "little laptop pixies" :) That's exactly what I was talking about.
I think I'm going to have to get this TECS book (no pun intended) - it sounds just what I'm after. Need to finish SICP first though :P Not enough hours in the day.
Along those lines, I think that Rubyists would really benefit from a "Fundamentals of Unix for Rubyists" book (well, not just Rubyists, but you know what I mean). I'm thinking a book at the same level as The Well-Grounded Rubyist--essentially a translation of CSAPP into Ruby. What are the fundamental abstractions that the OS provides you?
I know that for me personally, having gotten into programming via Ruby rather than an academic CS background, practically everything having to do with Unix felt like magic. What's a process again? How do pipes work? Wtf is TCP? Everything is a file?
Interesting. Yes, I came a similar route - no CS background. I read ESR's TAOUT [0], which assumes quite a high level of Unix knowledge - I'd love a book at that level!
Just so people know, I'm not talking Rails magic here - more the idea that there's stuff we don't understand that's going on that to all intents and purposes is magic. In other words:
> anything below your current level of abstraction that you don’t understand
"Where is Joel's peer reviewed article containing a mathematical proof describing leaky abstractions?"
The proof is so obvious that it borders on a "by definition"... an abstraction is something that has had details removed from it. When those details are still important for some purpose, as they almost always are, your abstraction is leaky. When you suddenly have those purposes, you suffer.
The idea of a non-leaky abstraction is almost nonsensical; if it doesn't leak at all it pretty much isn't an abstraction, it's an isomorphism.
The leakiness of an abstraction sounds like its dependent upon the value of the details that are abstracted away. Clearly there are cases where these details don't matter. If those cases constitute 100% of the usages of the abstraction, then the abstraction doesn't leak.
The point here is we're talking about a "principle" of un-intended abstraction leaks, not a "law of leaky abstractions", which sounds more authoritative than it should be and leads to shoddy thinking on the part of people who cite it as some immovable law.
The function fib is abstracting away all the details of calculating the nth member of the Fibonnaci Numbers. Sadly we are working with machines, not pure mathematical concepts - so when poor Bob will try to do something like [fib x| x <- [1..9999]] with the first implementation and will see his computer grinding to a halt he will have a leaky abstraction.
Basically the law of leaky abstractions just says that if you don't know everything about the abstractions you use you will eventually get hit in the head by one of them.
I like your example, and it's a good one for the specific case. I imagine a more formal proof would involve a reduction of any "non-leaky" abstraction to being capable of solving the halting problem, which is already proven impossible.
I didn't think physical laws are amenable to proof in the mathematical sense - for a start that would presumably make them unfalsifiable so therefore not really scientific.
If you think about it, you can see how the two definitions are related; this sort of magic tends to stay something you don't understand because you don't just naturally pick it up over time, you must make great effort to figure out what the problem is. But it's a different problem with a different solution, certainly.
Generally the two places this magic lives is in some all-encompassing framework, and in the language itself. Perl has numerous magical constructs, some of which turn out to be very difficult to even search for in the docs because it isn't even obvious what the operator is. The flip-flop operator comes to mind: http://perl-tricks.blogspot.com/2007/01/flip-flop-operator.h... If you encounter that for the first time you can be really lost trying to figure out how the list construction is occurring and it's not at all obvious what to search for.