I'm an everything debuggerer. I use print statements when its convenient (mainly when i want to know what's going on in a longer process), i use interactive debuggers when its convenient (mainly when i want to inspect complex data structures), sometimes i run the interactive debugger and then dump a complext data structure to STDOUT with a print (mainly when i want to reference it later on).

I don't get the people who insist on using only one thing, and claim that their thing is the best. There's lots of tools and all of them have different optimal use cases.

Upvoted and agreed. It really depends on the complexity of the problem at hand what tool is most suitable.

In basic I used the beep() debugger ;)

The fact remains that when we think of debugging tools we tend to build monolithic debugging environments rather than unix-style primitives that do one thing well. Thinking deeply about when prints work best and how we can ameliorate their drawbacks is a fertile place to change this. A couple of concrete ideas:

a) For a while now, I've experimented with editor macros to comment/uncomment code that insert a special marker after the comment leader (like say //? for C++ or Java) that allows me to programmatically distinguish real comments from commented-out code. An immediate benefit is that I can select large ranges of text and enable/disable prints without messing up interspersed prose. A more subtle benefit is that it lets me leave prints in version control for weeks of time and watch which prints I use most often when debugging. I've also experimented with a couple of subsidiary experiments: a1) highlighting //? comments differently (much less saliently) than real comments, so they don't bother me as much in my editor, a2) having my comment macro set/increment a per-line counter which provides even more fine-grained telemetry about how my codebase is used. It's illuminating to save all the prints you made while debugging a problem that ends up being just a one-line fix. Even if you end up deleting all the prints immediately in the next commit. And if you think it makes merging harder, well that's just a task for another simple unix-like tool: to filter out commented-code before merging, and add it back in afterward.

b) A big problem if you do a lot of printing is to find the right prints that help you debug your problem now. Print too much and it becomes easy to get lost in all the verbiage. I've tried battling this problem with a 'zooming editor' which reads a potentially huge log but displays only stuff at the highest level of abstraction, letting the user selectively click on lines to expand more and more detail around them. I can even bring up this zooming editor automatically when my program dies in debug mode: http://akkartik.github.io/mu/html/090trace_browser.cc.html [1] This is an incredibly useful tool, because it turns a two-sided problem into a one-sided one; instead of having to walk the tightrope between too much logging and too little I can just log everything and poke at it in my leisure.

Not all these ideas are good. I gave up on a2) above after a few months, for example. But there's lots of room for questioning conventional wisdom and trying new things.

[1] Details on my literate format: http://akkartik.name/post/wart-layers

> Print too much and it becomes easy to get lost in all the verbiage.

very much related to how much is too much. If you can simply filter out lines you don't want to see, like `grep -v '^first comment'`

Re: I only want to see the stack trace in certain cases

(C++) One of my favorite middle grounds between console debugging and loading up the debugger is to output the actual stack walkback as a string of hex addresses in production code (on a conditional log level). This is relatively very cheap and avoids the entire cost of loading up the machinery to resolve the addresses to symbols in the critical path. It is assumed that you know what binary is running where and that you associate the stack walkback with the unique binary and thus can resolve the symbols out of band at a later date. (.. or have some handy automation that is notified and does it for you out-of-band and puts the results back into the central logging facility :))

At Bloomberg we do this using part of our infrastructure libraries and have wrapped up the logging-the-stack-as-hex part into a simple drop-in call: https://bloomberg.github.io/bde/group__balst__stacktraceutil...

All of the conditional logging machinery is pretty important as well, as it lets you maintain all of your detailed logging statements with the flexibility to configure however you wish at runtime in a production build while debugging. There's many different solutions for this in the C++ world, but ours is here: https://bloomberg.github.io/bde/group__ball__log.html The rest of the components are all in that package: https://bloomberg.github.io/bde/group__ball.html

In the embedded world, sometimes you can't even be a puts/printf debuggerer. There have been some situations, typically early in development, where I've had to be a "turn the LED on or turn it off" debuggerer. Amazingly, that binary output is usually enough to fix whatever problem is happening and bootstrap into printf debugging or full ICE interactive debugging.

One time I couldn't get to an LED so I used an IR thermometer pointed at chip to debug it.

Telling the difference between hot and cold and whether it was getting warmer or colder told me what I needed to know.

Impressive.

Saying as someone who once debugged something by ear making the motherboard emit a cathode ray tube like hum instead of blinking or beeping.

Nice! How'd you end up generating the hum? I assume you oscillated something, remember what it was?

I think it involved a NIC, but I don't remember what it was exactly that I was doing to it. It was over 10 years ago.

Back at an old job, I was working on a new embedded system. It took us about a month before we bothered setting up UART for printf debugging. Fortunately the CPU/devboard supported gdb out of the box. What we ended doing was having a global char pointer point to somewhere in the middle of the memoryspace, and we would log things by appending to that string. Then we could gdb in and print the contents of the string.

I have done things like modulating a GPIO pin or LED to return a byte or two and see it on a oscilloscope. One time I was debugging an embedded device from a customer for a temperature controller where periodically the readings would be out of range. Turned out the CPU IP had a bug with interrupt handling when a particular instruction was used and after confirming no other side effect in simulations, the solution was to avoid using that instruction in the firmware.

That sounds really interesting. How did you get from modulating a GPIO pin to figuring out that particular instruction was bad?

It is from a long time back so things are fuzzy now but here is my recollection. I wanted to determine the source of the corrupted temperature readings but every pin on the MCU were utilized so no easy way to read it out. I noticed one pin was used just for an activity LED so I modified that routine to pump out a serial bit pattern read from a variable. Then I instrumented various parts of the code related to the temperature measurement algorithm to see the earliest point where it was corrupted. I traced it to the interrupt routine that reads the temperature sensor. In investigating that code I modified it slightly which caused the problem to disappear. Experimenting showed that a particular instruction would be the source of the problem. At that point I consulted with the ASIC designer and the confirmed the bug and identified the scope of the issue. Also should add that various debugging hardware like ICE didn't work maybe due to timing differences.

Sounds like you were lucky to have that bad instruction disappear with your modification. Cool story.

Once upon a time in the olden days, if certain makes of PC couldn't successfully complete the Power On Self Test and couldn't initialise video output to report the problem, it would beep the fault code through the system speaker.

Most motherboards still do this.

Just not as often, I imagine.

Except, for some makes, it wouldn't get that far but at least that narrowed down the possibilities.

A Bus Pirate can sometimes save your bacon or at least make your life easier in these situations!

Just did that the other day trying to track down a hard fault at startup. Binary code on LEDs was all it took to zero in on the cause.

Oh I've done this too on my TI Launchpad projects. Supposedly there is a way to attach a debugger, but I haven't figured out how to do that yet. I turned the LED on and used longer and shorter "on" times to differentiate between 1's and 0's. It was fun, but the feedback loop was very slow.

I've noticed that the more stateful my code, the more use I get from a debugger. Being able to poke and prod and see the various values in memory can be extremely helpful when I'm writing in an OO language, that just spitting to console doesn't really do (because I don't know up front what I'm going to care about; what influences what, what values are in the stack, in a containing object, etc).

Conversely, the less state I have (and, typically, the more functional my code), the less useful debugging is. It becomes easier to see my bugs up front (as they have more to do with explicit logic, that is expressed directly in my code), and to trace through in my head what must be going on, and all I need to do is check to confirm my understanding, and that my fix addresses it correctly, which can often be done more easily by just running the code twice (with a puts/print statement), than attaching a debugger twice.

Yes, this exactly! With OO-based PHP 7 codebases I work on, XDebug with the amazing Codebug client is brilliant. Breakpoint, then poke around the actual data structures within a running request, and see what happens!

When I'm working on functional codebases (our JavaScript is done in a very functional style, and I personally write Clojurescript and Wisp for my personal projects), I find printing the result of function calls completely sufficient, though I suppose one can consider a REPL not that conceptually different from an interactive debugger: in fact, Codebug and I think Komodo's XDebug client actually give you a REPL to play with, set to the context of the given breakpoint

I have the exact same experience. Being able to work with a value substitution model is incredibly helpful for mental debugging, just from reading the code. It just tires me to have to track where values are being modified, by who, putting watchers on variables, etc.

printf()-based debugging is okay when these statements are temporary, or when they are only present in development branches.

At $previousjob we had a guy who literally spent a year adding printf() statements everywhere to production code in order to help himself understand the existing code base he had to work on.

"It's not printf()-debugging, I'm doing extensive permanent instrumentation".

He even developed a whole framework for printf()-based debugging, to print messages in an artistic way.

The actual code became very hard to read due to the omnipresence of debugging cruft.

He also made the whole test suite dependent on the output of the debugging printf() debugging statements, making any kind of refactoring impossible.

I showed him how to use gdb and bought him a hard copy of the dtrace book, but it didn't help.

Oddly enough, I've spent some time working on "white-box" tests that check the log. It takes taste to do well: you have to be careful to log what your program figures out about the domain, rather than just random details. Done right, I find it actually helps refactoring. My unit tests now usually call some top-level function rather than just the sub-component they test. As a result I can make radical changes, like going to client-server or from sync to async, without having to change any tests.

More details: http://akkartik.name/post/tracing-tests

I would ask myself why he felt he needed to add such statements. Provided he isn't an idiot it is probably because the flow was hard to understand. Was that his fault?

I've found that if I can't understand the code it is because it is unnecessarily complex or, to mimic an above poster, there's too much state going on. Refactor to remove confusion.

Imagine yourself as a newbie asked to support such thing in production. Could you do it? If no why not?

Maybe he should have added log points instead of printfs. He could have used an existing logging framework instead of writing his own. Still

1) writing your own code is a valuable learning experience

2) wasting time is a drag on the team and on the company so he should compromise between learning and producing (but where was management?)

3) too much logging pollutes the code too

Since we're all sharing debugging lessons I thought I'd include one for .NET. In System.Runtime.CompilerServices, there are attributes that can be very useful for a logging interface: CallerMemberNameAttribute, CallerLineNumberAttribute, and CallerFilePathAttribute. These add compile-time info to method calls like so:

    void Error(string message, 
    [CallerMemberName] string callingMethod = null, 
    [CallerFilePath] string filePath = null, 
    [CallerLineNumber] int lineNumber = 0);

You then simply call `Error("something wrong with the frobnicator")` and the source line you need to come back to is already there. The only caveat is that this won't work in conjunction with variadic formatting for the message (`params object[]` arguments can't coexist with default arguments). `string.Format` feels like a small price to pay.

It's nice to see I'm not alone! I haven't really used a 'legit' debugging tool since my days as a C# developer. And that was because how easy debugging was in Visual Studio. I have yet to find something that just works, as easily. (I imagine XCode works perfect too, never programmed in Obj-C though).

Gdb works well for embedded Linux targets, provided you have Ethernet or a serial port. You can bring gdb up I the target and then connect the host to it and debug from there. If you're using an Eclipse-based tool then you can set that tool up to connect to a remote gdb.

At least, that's how it works when I'm developing under Linux. I'm not sure about Windows or Mac OS.

I've used gdb a few times, but I've always defaulted to print statements. It just... it just works, okay?

I get that there are definitely times when a legitimate debugger would come in handy, but I've never gone wrong with "puts debugging."

I don't even understand the 'have to learn a debugger' statement. Is he referring to things like pry? I rather enjoy the debugger in Rubymine, and part of the reason for this is that it more or less works like every other debugger I have ever used (eclipse, vs, chrome debugger, etc.)

There is not much to learn if you have used any debugger in the past.

Debuggers are very powerful tools and if they're worth anything, they'll have a learning curve.

For example:

- When debugging multi-threaded code, you can test interlace events by manually pausing executing of certain threads at certain points. I've used this many times to manually reproduce a rare race condition.

- Tweaking a conditional breakpoint at runtime can be easier than rewriting code and re-running.

- Visual Studio has "tracepoints" which are like inserting a puts/printf statement in the code, but allows you to output things you might not easily want to write code to do like the current callstack or function name, e.g. "Print the callstack when I enter this function every 100 times I enter it"

- Poking around in memory can reveal things you wouldn't have thought to output, or is invisible in output, e.g. discovering a BOM in one string and not the other and having that be the cause of a comparison failure or a different path through the code than you think it should be taking.

- Trying to understand control flow when exceptions might be thrown or assertions might be triggered just isn't as easy with puts/print. You're likely to make a bad assumption. Understanding exceptions and how to make your application easy to debug is definitely a skill that begins at coding, not opening the debugger.

If you don't think you "have to learn a debugger" you're missing out on a lot of opportunities to be better at finding bugs.

I agree, the concepts of debuggers are all the same but there is a definite friction to using a debugger if it's not already setup with a familiar interface. Sometimes the pain of setting up a debugger isn't worth it when you can solve the problem in a few minutes with echo debugging. And there can be plenty of pain in getting debugging working when it's not as simple as just clicking "Debug" in your already setup IDE.

That which is not learnt is innate.

GDB is not innate.

Therefore GDB has to be learnt.

---

I don't understand your lack of understanding :)

EDIT:

Addendum: Symbolic debuggers like GDB are non-trivial to pick up, is that what you're arguing against?

Writing to the console is good if you have a console, and don't need to stop the program and examine the state of memory, files, processes immediately before the problem occurs. Otherwise it's a big fail.

Are we supposed to believe that some people can understand unix, c++ etc but cannot learn/keep a cheatsheet handy so they can employ the ~10 commands needed to use gdb fruitfully?

It's great find people like you and me :)

I'm a pprint, fmt.Println, panic, console.debug, log.Print debugger !!

does golang even have a debugger with horrible gdb hacks?

There's godebug[0], which has all the hacks you could ever want! But I'm guessing you meant 'without', to which the answer is: no.

[0] https://github.com/mailgun/godebug

ah yea I meant without.

Godebug doesn't work with 1.5 vendor-ed deps, sigh !

I guess it is by far easier to be a puts debugger than a printf debugger (I'm saying that as someone who prefers printf over a debugger but pry on Ruby over simple puts).

You can get a much longer way with what Ruby lets you do with puts than you would with printf in C/C++.

Like the example with `method(:render).source_location` from the article. You can't do that in C/C++ on a general base.

Or pretty printing a struct is cumbersome. In Ruby you just do `puts obj.inspect` and get a readable representation of the object.

__FILE__, __LINE__, __FUNCTION__

Yes, there's no default pretty print for structs, but no sane C programmer would ever expect or want that blubber to come available by default. Mainly because there's very rarely a need to printf-dump anything more than a select variable or a struct field. Different language - different debugging realities.

Those only act on the currently defined context.

Especially __FUNCTION__ is by no means an equivalent to source_location.

__FUNCTION__ tells you, in what function you currently are.

source_location tells you "in what file has this method of this object been defined". Because of duck typing this is much harder to answer in Ruby. But even in C, because of linking it can be non obvious what function actually gets called.

I am a printf/gperftools/asan/gdb/valgrind debuggerer too:

https://github.com/vmorgulys/sandbox/blob/master/stackcity/r...

Sometimes but rarely, I use Nemiver:

https://en.wikipedia.org/wiki/Nemiver

Any time you use the console or a printed output to debug something, that is an embryonic test assertion in the making, and all you have to do is move it to a proper suite. ;)

If you are doing it because you are deep in some nested conditionals... you need to refactor.

"Tender love making"? "Debuggerer"? If this safe for work?

I've always found people skilled with debuggers look like they have super powers. Symbolic debuggers really are tricky, and these are some good tips.

I used to think symbolic debuggers were tricky until I actually bothered to pick up GDB. It seems hard and complicated at first, but it's really not. You can get very far with just "break", "next", and "print". It's so much nicer than printf debugging (and I used to be a hardcore printf debuggerer).

it's the 21st century. i use a debugger.

Sometimes, just dumping stuff out to the console is the best.

Especially in multi-threaded code, it can be the only way - hitting a breakpoint and stepping through causes other threads to time out in unnatural ways, and you're hosed tracking down things that wouldn't actually happen. It's not uncommon that I've run into issues where attaching a debugger or a profiler gets its hooks into something in the depths of whatever COM library I'm using that breaks it anyway (I love RedGate's performance profiler, but I can't use it for some things on Windows 10, because of some security something or other, while it works on 7 & 8...)

Log4net is, without a doubt, my favorite dependency. ColoredConsoleAppender FTW. Even better, if I take the slightest care with my log levels in code, I can tune the log output up and down just by tweaking the config. So I can leave full diagnostics in a release, tune things down to hide the verbosity, with minimal performance hit, and dial it back up to debug levels if I encounter an issue in production.

It looks like you are the novice here, Tenderlove is a well known Rails core contributor and all around tender guy.

https://twitter.com/tenderlove

Just wanted to point out that you may disagree with him here but @tenderlove is far from a novice https://github.com/tenderlove?tab=activity

I think being a core contributor to major pieces of software does imply you are a good programmer rather than a novice, whether you know of them or use that software or not. "Pure verbosity" - Really? You want to disparage his contributions? Take it easy, man.

He doesn't "dismiss the use of a runtime debugger", he says "I don’t say this to disparage people that use a Real Debugger. I think Real Debuggers are great, I’ve just never taken the time to learn one well."

Yes, people who try and inject a little fun into their interactions are just the worst, aren't they? In a professional setting too. It's just shameful behaviour. Also, I'd be really cross if one of my children should stumble across this "tender lovemaker's" blog. I hope this guy is ashamed of himself.