I think it's more of a question if box2D is the correct solution to implement for physics-calculations in the first place. As Vuo already has a lot of the base components that you'd want from box2D (layers/bounding boxes/fast memory management etc), I'd wager that it would be a neater and better solution to implement a native system - especially when considering 3D as well.
The physics calculations themselves are pretty straightforward, it's all the stuff around it that is the hassle. I have tried to make a few physics-ish-nodes, but the issue is mostly how to make a good user experience in the line of; do you want it to input layers/objects and manipulate them? Or do you want to generate layers from a master node? Do you want a "world" node to connect layers to? Or could a "Make Layer/Object with Physics" node be simpler? Knitting together a nice workflow is probably the largest piece of the puzzle, but that is an issue if you use box2D as well
In a custom library node, all changes should be done through published ports, not through edits of each node itself. If it is for creating more instances of similar nodes, you can save as to the user modules folder. If it is for edited copies of a node, you can copy/paste instead of making a library node.
Looking good! The Hold-node approach also enables feedback of everything, not just images. In theory you don't need anything but an image layer, and then you can feed that back to itself. I think I've done this before, but trying it now it seems that it breaks the composition.
You can however use an image layer or a 3D square instead of a blend node to achieve more or less the same results with fewer nodes. This also enable multiple feedback paths at the same time if that is desired. Fractal-like patterns may emerge!
(You'll need to feed the composition an image with transparency for it to work)