I could be wrong, but this seems to be exactly what Zep’s “Graphiti” library is made for, build as a semantic temporal knowledge graph.