This is a big implicit FAQ for Octave (actually, frequent incorrect bug report), so this is our response mostly written by me:

http://wiki.octave.org/FAQ#Why_is_this_floating_point_comput...

I think people just forget that when they see "0.1", that this can't be represented exactly in binary.

I also link to the following long-winded explanation of how floats work:

http://floating-point-gui.de/