In fairness, if you let the word compact do a lot of heavy lifting, I don't see why a sufficiently compact power source would have to be dangerous to fly. Give it enough redundant cells and enough AI to keep you upright, sure why not? From an abstract physics perspective, flying is only a little more complex than keeping a Segway upright. Look at what modern ejection seats can do.

The problem is that we are orders of magnitude off in terms of power density, and it seems unlikely that we'll bridge that without some sort of major revolution in physics. So sci-fi it remains.