>the borrow checker, I do conceptually understand lifetimes, but actually using them is tricky.
I've been using Rust for a little over year, almost daily at work, and for several projects. I have a pretty good intuition about how the borrow checker works and what needs to be done to appease it. That said, I don't think I'm any closer to understanding lifetimes. I know conceptually how they are supposed to work (I need the reference to X to last as long as Y), but anytime I think I have a situation that could be made better with lifetimes, I can't seem to get the compiler to understand what I'm trying to do. On top of that very little of my code, and the code I read actually uses lifetimes.
I've been writing Rust code since before the 1.0 days, and I still can't understand lifetimes in practice.
When the compiler starts complaining about lifetimes issues, I tend to make everything clone()able (either using Rc, or Arc, or Arc+Mutex, or full clones).
Because if you start introducing explicit lifetimes somewhere, these changes are going to cascade, and tons of annotations will need to be added to everything using these types, and their dependent types.
I think you're really intended to do the latter rather than the former. I mean, Rust lets you do either–it gives you the choice if performance isn't your concern–but usually it's better to not clone everything.
I find that lifetimes are ok, albeit annoying sometimes, especially the cascade part, as you mention. The one thing I can't get to stick in my brain is variance. Every time, I need to go back to https://doc.rust-lang.org/nomicon/subtyping.html#variance
It's worth noting that every reference in Rust has a lifetime, but the compiler is usually smart enough to infer it. What you are talking about is explicit lifetimes.
As someone in a similar description as you - i find my lifetime understanding... moderate. Complex lifetime usage still can tweak my brain - notably how i can design it. But simple lifetime usage is intuitive.
A simple example i often run into is wanting to do something with a string, without taking owned parts of the string. Very intuitive how the str matches the lifetime of the owned value.
On the otherhand, the other day i was trying to write a piece of software where:
1. I wanted to deserialize a large tree of JSON nodes. I had the potential to deserialize these nodes without owning the data - since Serde supports lifetimes, i could deserialize strings as strs and hypothetically not allocate a lot of strings.
2. In doing that, because a tree could be infinitely large i couldn't keep all of the nodes together. Nodes could be kept as references, but eventually would need to be GC'd to prevent infinite memory.
3. To do this, i _think_ lifetimes would have to be separate between GC'd instances. Within a GC'd instance, you could keep all the read bytes, and deserialize with refs to those bytes. When a GC took place, you'd convert the remainder partial nodes to owned values (some allocation) to consume the lifetime and restart the process with the owned node as the start of the next GC lifetime. ... or so my plan was.
I have, i think, just enough understanding of lifetimes to _almost_ make that work. I _think_ some allocations would be required due to the GC behavior, but it would still reduce ~90% of allocations in the algorithm.
Unfortunately, i got tired of designing this complex API and just wrote a simple allocation version.
Conceptualizing allocations and the lifetimes to make it work are.. interesting. Especially when there is some data within the lifetime that you want to "break out of" the lifetime, as in my example (where i had a partial node, and i made it owned).
I still think i understand enough to do it - it'll just take a fair bit of thinking and working through the problem.
These kind of optimization are incredibly painful in Rust one common suggestion is to sidestep the issue and store an offset + length in the nodes and then you take that to look up the value from the original string when you need the value.
I've tried and tried but I've never found a situation where explicit lifetimes was the answer. It's almost always more complex than that. I mean, everywhere that is complex enough that implicit lifetimes don't work is also too complex for explicit lifetimes and almost always required Rc or Arc to solve it. Maybe I'm missing something, but it seems like there are so many other missing topics the Rust Book could be spending time on that would be more effective than teaching about explicit lifetimes.
There are a lot of situations where explicit lifetimes are the answer, TBH. However, you have to have a very good working model of lifetimes in order to get the annotations right.
I wrote a parser that needed it. But yeah for the most part whenever explicit lifetimes came into the picture it means that I have made some sort of mistake and need to rethink my approach.
I've been using Rust for a little over year, almost daily at work, and for several projects. I have a pretty good intuition about how the borrow checker works and what needs to be done to appease it. That said, I don't think I'm any closer to understanding lifetimes. I know conceptually how they are supposed to work (I need the reference to X to last as long as Y), but anytime I think I have a situation that could be made better with lifetimes, I can't seem to get the compiler to understand what I'm trying to do. On top of that very little of my code, and the code I read actually uses lifetimes.