jmcc's picture

Jean Marie (@jmcc)

Groups

  • Vuo Founder
  • Team Vuo
jmcc's picture
Jean Marie commented on keithlang's Feature Request, “CoreML integration

Vuo has built-in support for hand tracking using the Leap Motion, but not with normal RGB (non-depth) cameras.

HandPose OSC uses TensorFlow's handpose model (not CoreML). That model uses the Apache license, so we could potentially integrate it into Vuo (though, as you noted, it currently only tracks a single hand at a time).

Apple provides VNDetectHumanHandPoseRequest (only available in macOS 11.0+). We briefly tested their sample code; it's slower than HandPose OSC, and it also only detects a single hand at a time.

So, for single-hand detection, we have 2 options, at a two-dot complexity (and Pro only).

For multiple-hand detection, we could probably train a model to segment the image into separate per-hand images and run the existing model on each of those sub-images, but that would bump it up into 3-dot complexity.

Let us know if you want single or multiple-hand detection, and we'll modify the FR accordingly and open it for voting.

jmcc's picture

Thanks for providing more information. We followed the steps shown in his screen recording (using the same nodes and the same top-level composition name and subcomposition name), but it doesn't crash for us. We think it's related to a third-party node or nodes that you have installed.

Does it help if you uninstall Parabox (and/or any other 3rd-party nodes you have installed)? Does it help if you temporarily rename your Modules folder on the Desktop? If so, can you narrow down which 3rd-party node or modules causes Vuo to crash, and send it to us so we can try to diagnose the problem?

jmcc's picture

We've converted this to a bug report and accepted it.

jmcc's picture

We're marking this closed since we can't reproduce it, but will reopen it if you or another community member can provide more information.

jmcc's picture

videopiglet,

In Vuo, images and layers really have different roles. Images are grids of pixels that you can manipulate. You can resize them (Resize Image, Resize Image if Larger), crop them (Crop Image, Crop Image Pixels) or apply any one of a number of image filters.

Layers are designed to be stacked on top of each other, a bit like layers in an app like Photoshop. You can create layers in Vuo, either from shapes or from an image, but you can't load layers from a file. Layers are sized based on the Vuo coordinate system.

As you noted, images can be made into 3D objects.

It’s not computationally expensive to make a layer from an image. Creating a Layer or 3D Object from an Image (nodes like Make Image Layer and Make Lit Image Shader -> Make Sphere) is computationally fairly efficient, since Layers and 3D Objects just keep a reference to the image you feed into it.

It is more computationally expensive when you start to create new images. One example would be creating an image from a group of layers or 3D objects, such as Render Layers/Scene to Image.

There is a tabled feature request Execution time, which includes a composition (which can be packaged as a subcomposition) for testing the execution time of a node or group of nodes in a composition. This might be helpful as you examine your own compositions.

Pages