Another thing you could try is to go to System Preferences > Mission Control and toggle the "Displays have separate Spaces" setting, then log out and log back in, and see how the performance compares.
To get a more objective measure of performance, you could add a visualization of skipped frames as in the attached composition.
One thing we're thinking of changing based on your comments so far — As you observed, currently you need a Resize Image to scale the image back up to the screensaver's expected size. We could change the wrap mode on the output image from "clamp" to "stretch" so you wouldn't need the Resize Image (and extra processing that it incurs). Assuming that doesn't break anything else.