For a quick, easy-to-digest introduction to the topic, there's a fantastic video with explanations and an interview with Bengio that uses Galton peg boards as a starting point for understanding generative flow networks. Highly recommended if you're a researcher or practitioner interested in learning about this:
Hi, first author of the original NeurIPS paper here! Someone just shared this HN post with me. I'll go through the comments but happy to try to answer questions as well.
I'm still wrapping my head around this (it seems rather magical; can it really be just that?)... but I find the idea of "sampling from a distribution of trajectories that reach a terminal state whose probability is proportional to a positive reward function, by minimizing the difference between the flow coming into the trajectories and the flow coming out of them, which by construction must be equal" to be both beautiful and elegant -- like all great ideas.
How did you and your coauthors come up with this? Trial and error? Or was there a moment of serendipitous creative insight?
--
To the moderators: My comment is now at the top of the page, but manux's comment above is more deserving of the top spot. I just upvoted it. Please consider pinning it at the top.
I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state.
But, yes in the moment it felt like some very serendipitous insight!
Thank you. It's nice to hear that you had one of those shower/bathtub Eureka moments!
> ...we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state
Makes sense. FWIW, to me this looks like a Conservation Law -- as in Physics. I mean, it's not that the flows "must be" conserved, but that they are conserved (or go into sinks). Any physicists interested in AI should be all over this; it's right up their alley.
Could you highlight the difference between this and training a permutation invariant or equivariant policy network using standard supervised or RL methods? Assuming I also have a way of having an invariant/equivariant loss function
What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward:
- RL says, give me a reward and I'll give you its max.
- GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)).
Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.
I started reading through the linked tutorial, but I think I need to first back up and review reinforcement learning (I took 2 Coursera classes in RL, and helped with one project at a previous job, so I am at a beginner stage).
I love deep learning, a practical way to solve many problems (and most of my work over the last 6 years) but DL seems incomplete as far as explainability, causation, etc. Anyway, I just retired last week (mostly to take care of my wife who is having a health crisis) so I have plenty of time to study GFN, probably starting with re-taking the 2 Coursera classes on RL and then finishing the RL specialization.
Having just skimmed the linked tutorial, it looks like GFlowNets could be useful for automated theorem proving. Anyone with more knowledge know if this is possible?
Yes, although they may not be ideal unless you're able to define a "distance" between a proof that the agent is proposing and whether that proof is correct (i.e. unless you're able to define a reward or energy function).
It may be possible to infer/learn a score from existing proofs though. We have a paper that manages to both learn a flow and an energy function (the score) from data: https://arxiv.org/abs/2202.01361
I don't know much about theorem proving though. Can some value be attributed to partial proofs?
What do you see as the application to GNN / Transformer tasks? Both of those networks are best suited for different tasks, and GFlowNetworks seem well suited to quickly approximating the partition function of say an Deep RL reward distribution, something that MCMC is ill suited for because of the incredibly high dimensional space. I see these all as different techniques, perhaps the ideas will combine at some point but GFlowNetworks are not an evolution of GNNs or Transformers.
Yeah the "net" in GFlowNet refers to how the underlying state is interpreted, not to an architecture. It is a way to train generative models, on any kind of discrete data (like graphs).
Source: am first author of original GFlowNet paper.
Thanks for your comment! I originally finished my PhD in computational neuroscience and have been riding the deep learning wave for the past five years. I listened to the audio Machine Learning Street Talk episode on GFlowNets, and that was my general introduction. It wasn't until I looked at the image of the GFlowNet on your blog post a connection to actually biological plausible architectures became apparent. The individual firing of a neuron can be interpreted as a Poisson process, but with populations of neurons we often would approximate the firing rate of the entire population as a Gaussian distribution. That would be one patch of cortical column, but these populations are linked together to form larger scale functional brain networks. Anyway, thinking about that scale you could imagine the neural activity communicated between these patches of activity as a type of flow from one region to another. I think your paper raises interesting questions about how this inference engine in our brain might be doing some sort of MCMC like sampling and constructing different belief hypothesis.
Yeah! This kind of reasoning is why it's exciting to think that we could use the GFlowNet machinery to construct latent representations--ones that map more directly to the notion of "beliefs" that we're used to think about as humans, something "discrete" and relational.
Concretely what this could mean is using these tools to generate causal hypotheses, like what's been done here: https://arxiv.org/abs/2202.13903
https://www.youtube.com/watch?v=M49TMqK5uCE