Nice! That's the Matlab way of doing things. I used to miss the Matlab workflow a lot when I was transitioning from Matlab to python. Although somewhat surprised they didn't just go with "## title" to match Matlabs' "%% title" (difference is comment character, and "# %%" reads like a python comment of a Matlab cell, I suppose it makes sense if you start with a m-file, python comment the whole thing and then work your way down translating cells from Matlab to numpy/scipy).
Personally I've since gone full literate programming mode to the point that I care far more about the narrative and documentation (of methods and results) that I will build and modify tools rather than go back to the Matlab way. I have been looking at Quarto but haven't had the time to see if I can transition my existing (and target/ideal) workflows.
I know it gets a lot of hate but ipynb have a lot of advantages as a format for building small custom tools for modification/transformation. Most of the complaints ultimately seem to boil down to not having tools that do what you want. Only want to diff the code cells? That's easy in a python utility that loads the notebook and looks at it intelligently. You can also use pre-commit to modify the notebook and strip out things that don't belong in git.
(Also nbdev... exists... and is a good example of how tools can help. Unfortunately it's too tied to GitHub functionality and the developer is a GitHub zealot who is oddly brittle and takes offense and demands justification if anyone mentions not wanting to rely on GitHub)
I must be missing something because I am not immediately seeing the value-add. Do you prefer the separation of input/output or is it something else. I believe all of the debug, extensions, and hinting work the same as the standard notebook.
I'm glad you asked, because you made me think about why. Initially, I guess I just thought it was kinda neat and stuck with it, but on reflection this is what I personally feel I get out of it:
- Same interface for analysis, scripting, and building more complex multi-file pipelines. I can also use the #%% notation to break up and debug scripts, which is probably teaching me all sorts of bad habits but it's something I find helpful.
- Similarly, as another commenter in this thread notes, .ipynbs just don't play as nicely with the other dev tools (e.g., Git, Black) and generally feel like second-class citizens in VSCode.
- I much prefer having the VSCode interactive window on the right, as opposed to having my output dumped out below my code block. I now find using the classic notebook style makes the document much longer and harder to navigate, particularly as I work with text a lot and I'm often outputting large chunks of text for inspection.
This noted, I think this is all possible because I'm rarely producing my final products in notebook format. Neither my boss nor the stakeholders I typically present to can (or have any inclination to) read code, so I don't really need a format others can execute or inspect. I just take the charts and figures and dump to presentations and other normie-friendly documents.
I am torn on the separation of the output from input. I expect half the time I would be happy for the extra vertical space and the other half I world be annoyed I could not immediately correlate code with output.
Anyway, thanks for spreading the workflow, and I will definitely try it out in the coming weeks.
I wish more data scientists used light percent format notebooks `#%%`. It can be combined now with other powerful tools (linting, formatting and git) that is impossible with the `ipynb` format
I always like this better than true notebooks for a lot of purposes, but it's long overdue that we standardize on a format here. Knitr and RMarkdown never caught on, and Org Mode and VS Code both just do their own thing. It's a shame there isn't something more "portable".
I like to have the relevant code and output side-by-side, and dislike scrolling past outputs to get at code. Again, pure preference.
My screen copes fine with two tabs and the sidebar hidden most of the time, but more real estate would be nice.
What I'd love would be to pull tabs out into separate windows, like in a browser, and have the Jupyter output and variable inspector on a second screen. If anyone knows a way to do this (not new window) I'd love to hear. Last time I looked seriously this wasn't possible.
Most of my analytical work now is done in .py files, broken up into blocks with `#%%`. Real notebooks feel really clunky since adopting the approach.