I work on a project that builds an SVG compliant vector scene graph and allows for detailed interaction via its API. Writing client code that can plug into the rendering pipeline or just altering the existing code is also fairly easy - comparatively at least.

It's at https://github.com/parasol-framework/parasol but fair warning though, it is in alpha right now and going through an overhaul.