A node for loading structured data from file, URL, etc.

Imports: XML, JSON, CSV, TXT, etc. for parsing Filters: ability to filter structure data streams Multi dimensional: ability to have nested data (not just 1-dimensional lists)

Could go as far as ability to connect to DB's but that's probably pushing the complexity of this feature.

Component: 

Tags: 

Notes from Team Vuo

Vuo Pro: 

No — available with both Vuo and Vuo Pro licenses

Complexity: 

●●○○ — A few weeks of work

Potential: 

●●○ — Could expand community

Implemented in Vuo version: 

Comments

@joeladria: You're right. We

smokris's picture
Submitted by

@joeladria: You're right. We're already planning to implement much of this during the next few releases. Let's reexamine your structured data needs after Vuo has dictionary support.

Could you submit a separate feature request for database connectivity? Which databases would you like to connect to (ODBC, SQLite, ...)? If you could also provide descriptions of a few scenarios in which you'd use it, that'd be great.

Hey Joeladria,

tmoles's picture
Submitted by
Feature status:
Waiting for more information from reporter
»
Tabled by Team Vuo

Hey Joeladria,

We looked into what it would take to pull this off — it's no small task! We're interested in making this happen but won't be able to pursue it until well after 1.0. We're going to table this feature request for now and look at it again after 1.0. Thanks!

I think you don't need all

volkerku's picture
Submitted by

I think you don't need all kinds of formats(xml, cvs, json) supported. a simple txt file i/o would be fine. Existing nodes such as split text could be used extract the data to a list. This is an absolute must in my view, things like storing and loading preferences, logging events and so on. v.

As discussed with @jstrecker

Bodysoulspirit's picture
Submitted by

As discussed with Jaymie (@jstrecker) here on this Websocket feature request : https://vuo.org/node/276

I was wondering if this would include HTML structure parsing for websites ? Something like XML parser from the Kineme Network Tools ?

Or should I make a seperate feature request for online website structures ?

And as @volkerku said, before some specific nodes for every kind of different structures, wouldn't it be great be able to open local or online XML, Texts, DB's HTML and JSON things and retrieve some rough unsorted (or savage parsed) I/O text file where we could extract some data using text sorting descriptors or structure get items ? A faster easier node to wait for some more powerfull one's.

@khenkel released a text file open node as far as I remember, but it will only open real text files, and not turn all files into rough I/O text.

So this would be a big mix of the Kineme File Tools and the Network Tools

This is great, but I guess

Bodysoulspirit's picture
Submitted by

1 - What is understood with "Add filters to Parse" ?

Will it allow us to retrieve some specific data from those structures ? O can I encourage people who voted on this to also vote for Request by Steve (@smokris) here : More List Nodes

2 - Steve (@smokris) : If this will load HTTP too, could you please update the title for this feature request too ? Might be useful for people to vote for this !

@Bodysoulspirit, I updated

smokris's picture
Submitted by

@Bodysoulspirit, I updated the title to mention HTTP(S). For structured data loading to be useful, yes, we'd need to implement some additional list nodes. When we're plan the release that includes structured data loading, depending on the voting results, we'd either implement that entire other feature request, or just the most important parts needed for working with structured data.

This feature would greatly

Project26's picture
Submitted by

This feature would greatly extend Vuo's usefulness, and genuinely add value. I would be using it tonight, instead of hacking together a Serial bridge in Python. :)

(Don't get me wrong - not hating on Serial - Serial support super helpful and very glad to have it.)

Hey, thanks again for all the

cwilms-loyalist's picture
Submitted by

Hey, thanks again for all the amazing features you've been adding to VUO over the past months! I can see you've all been very hard at work adding some amazing features to the Nov 2015 release! I figure "Loading Structured Data" hasn't even been slated yet but I'd like to say that I am still VERY interested in using XML importing and parsing for a large project, then I can completely replace QC. It would be amazing if it can be put on the list for the next update cycle! :)

We weren't able to get this

jstrecker's picture
Submitted by

We weren't able to get this entire feature request into the upcoming Vuo 1.2 release, but we were able to implement one key piece: reading text from a file. With the existing text nodes, you'll be able to do some rudimentary parsing (split by line, split on commas).

I don't want to put my foot in my mouth by promising anything prematurely, but I can say that since this feature request is one of the top-voted ones (#7 overall, #4 among features up to a few weeks of work) it's high on our list of candidates for Vuo 1.3.

alexmitchellmus, keen observation. Yes, the Fetch RSS Items node does some XML parsing. And we already use the json-c library in many places.

Awesome Jaymie!

alexmitchellmus's picture
Submitted by

Awesome Jaymie!

Chicken or egg kind of stuff- but list in list is a very necessary data type for this feature as well. No point loading in 1D values... Always more fun with friends. Keep up the great work.

Is the load text file in alpha 4?

Just playing with the new 1.2

alexmitchellmus's picture
Submitted by

Just playing with the new 1.2 A4 vuo.data.fetch, great job guys!

One question for loading structured data: will we have the ability to use Keys to access data? And if so can the data be output in correct Vuo parlance? (if formatted correctly)?

XML Such As:

<myxml>
<foo>1</foo>
      <bar>
          <x>2</x>
          <y>3</y>
     </bar> 
</myxml>

Would output if using a "Vuo.list.get.with.key" node (yet to be created):

  • KEY: foo = 1
  • KEY: bar = "x":2,"y":3

This way one can use Index & Keys to control certain aspects of complex compositions. (I already do this for controlling compositions in QC- so I really would love to be able to rebuild my work in Vuo!)

In the above example, bar could be the key for the x,y position of a layer, so then I could add multiple values for bar- and simply move through them via indexes! If an index does not contain a 'bar' key then 'Vuo.list.get.with.key' wouldn't spit out anything.

UPDATE: I think this is covered by the request "Dictionary Support", however I believe it may have low votes due to assumption that this feature request would encapsulate the other? I don't know how useful a list generated from XML would be without its dictionary.

Thanks all for your feedback.

jstrecker's picture
Submitted by

Thanks all for your feedback. Update on the plan — it includes:

  • Table-style (CSV, TSV) and tree-style (JSON, XML, HTML) data
  • Higher-level types (than lists or dictionaries) for efficiently working with data
    • Table with ability to get row/column by header or index, ability to add/remove/change row/column
    • Tree with ability to search by XPath/name/attribute/content, extract attributes/content, build tree hierarchically
  • Save Data node to save to file
  • Converting tables and trees to/from the Vuo Data type for use with the Save Data and Fetch Data (read from file) nodes.

New, related feature request: Advanced HTTP(S) query nodes

@jaymie. In xml you can

alexmitchellmus's picture
Submitted by

Jaymie (@jstrecker). In xml you can have an item of item:

<red><blue><green>"1"</green></blue></red>

So green 1 belongs to blue and blue belongs to red.

If you read out red you get a list of blue, which has a list of green which is value 1.

So my question here is if XML is developed in 1.3- then won't team Vuo need to re-code aspects of it to work with list in lists?

This is to say the above list would not work with 1.3 XML reader as its not a 1D structure? If I am wrong let me know.

` 1 100 `

alexmitchellmus's picture
Submitted by

This is the first 3 cues from a show I worked on (Wonder if anyone can guess the show?!). Here you can see the main XML element is data, then each other element can have its own elements. In QC I can use this method to cue animation, or switch off parts of the composition. Its very powerful as all I need to do is edit the xml file- and my whole show is there. I also don't have to worry about cue numbers, as if I want to insert a cue I simply do. Yet if I wanted to I could simply call the scenes cue numbers anyway. Does this explain a bit more Jaymie (@jstrecker)?

<data>

<blackout> </blackout>

<sunrisePrideRock>

<prideRockIn>1</prideRockIn>

<playTo>100</playTo>

</sunrisePrideRock>

<blackout> </blackout>

</data>

@alexmitchellmus, with this

jstrecker's picture
Submitted by

@alexmitchellmus, with this feature request, a composition to parse the "pride rock" example you gave would work something like this:

  • Fetch Data -> Parse Tree
    • output data type: Tree
  • Find Tree Item with Name
    • inputs a Tree and an XML tag name (e.g. "playTo")
    • outputs a list of matching children, as Trees
  • Get Tree Item Content
    • inputs a Tree (e.g. the match found above)
    • outputs the content (e.g. "100")

Instead of multidimensional lists, you'd be dealing with the Tree data type. (That's what I meant by "higher-level types" in my comment above.)

I'm really excited for the

David's picture
Submitted by

I'm really excited for the ability to parse CSV files. I'm working on a number of video tools to generate special effects. and CSV tables are ideal for structuring the actions to be taken to multiple input streams of video, Each line of the cvs table specifies the actions for the input streams.

@Jaymie if XML parser looks

alexmitchellmus's picture
Submitted by

Jaymie (@jstrecker) if XML parser looks at the xml like a tree structure, can we also output a tree from an index? Such as asking the parser for XML item number 3? Etc?

That is how I would need to use the XML as well, moving though the elements as both index and dictionary support.

So that means I could find tree item with index number.

Do we know what release

cwilms-loyalist's picture
Submitted by

Do we know what release version this feature has been scheduled for? The project I need it for isn't until early June 2017 but I wanted to make sure I had enough lead time to test and ensure my composition will be working smoothly first.

I found myself in a very awkward spot if I stay with QC for this year because the composition I am running not only needs a computer with a very specific set of ports but also one able to run OS X 10.7 for QC feature compatibility which leaves me a very short list of Macs this composition can even run on. My 2011 17" MacBook Pro is dying a slow and painful death and so I am hoping to switch over to VUO which makes my list of possible compatible Macs much bigger. Here's hoping it's coming soon.

Somehow I missed your

cwilms-loyalist's picture
Submitted by

Somehow I missed your response but am SO glad to hear it's coming in Vuo 1.2.6... if you need any alpha/beta testers I can do a little testing with my planned project. :)

My initial need will just be to have my project read a datamatrix code with a camera, extract a 4-digit code from the delineated text string and then use that 4-digit code to find the appropriate replacement text in an XML file for displaying on screen.

@Chris and anyone else who'd

jstrecker's picture
Submitted by

Chris (@cwilmsloyalist) and anyone else who'd like to be testers — we just published a Vuo 1.2.6 alpha release that includes XML/JSON nodes. We'd appreciate if you could try them out and let us know if they do what you need them to do.

The final 1.2.6 release will include much more stuff, including CSV nodes, but we wanted to get your feedback on the XML/JSON nodes sooner.

If you find any problems with the alpha release, please post a bug report. If you have questions/suggestions about the scope of this feature request (what the new nodes will/won't be able to do), please comment here. For other questions/suggestions, probably best to a new discussion thread so it's easier to follow the conversation.

I'm having a level beginner

Bodysoulspirit's picture
Submitted by

I'm having a level beginner question about XML trees. I'm trying to get the Artist Name, Song Title and File Location for the iTunes songs from the iTunes Library XML.

I'm able to somehow locate the subtree of a song, but then I can't find how to locate the Name or Title because they all start with subtree "Key" as name, then have "Name" or 'Title" as content, but the title or name itself is yet another subtree with name "String", all the same again on the same line.

Should I be using Split Text from here on ? Can't I do this because the way the XML is written or because of the node not considering the same line as a subcontent ? Thanks

Joining the composition, it assumes your iTunes Library is in -/Music/

I may be misunderstanding how

cwilms-loyalist's picture
Submitted by

I may be misunderstanding how to use the Find Subtrees using XPath node. I've created an expression that I thought would allow me to find value code and it's sibling values fullname and school but it isn't working like I thought it would.

The way I want it to work is to search by the program code and return with the fullname and school values that are it's siblings.

@Bodysoulspirit, the iTunes

jstrecker's picture
Submitted by

@Bodysoulspirit, the iTunes music library is in a special (Apple-specific) XML format called a property list — which is convenient for some purposes, but unfortunately not for locating items in the XML.

Possibly we could enable the Make Tree from XML node to detect if the XML is a property list, and if so, turn the keys into element names and the values into content.

Anyone else interested in using the iTunes file or other property lists in Vuo?

@Bodysoulspirit, the iTunes

Bodysoulspirit's picture
Submitted by

@Bodysoulspirit, the iTunes music library is in a special (Apple-specific) XML format called a property list

Thanks for the info about that !

Possibly we could enable the Make Tree from XML node to detect if the XML is a property list, and if so, turn the keys into element names and the values into content.

Ok so either like that or as a specific Make Tree from Apple Property Lists node perhaps ? Would perhaps keep the regular node faster if less code ?

Well ok if this is so specific to Apple I could imagine a new feature request though if it is more work !

Anyone else interested in using the iTunes file or other property lists in Vuo ?

So you mean all other .plist OSX files would work as well ?

Somehow related interest from @vascoarmartins : Read iTunes playlists Feature Request

So you mean all other .plist

jstrecker's picture
Submitted by

So you mean all other .plist OSX files would work as well ?

Good question — I meant that Make Tree from XML would handle property lists stored as XML text, like the iTunes file. But usually macOS stores property lists in a binary format (if you try to open the file in a text editor, it's just a bunch of weird characters), so that would be a separate node, beyond the scope of this feature request.

Thanks for pointing out the existing feature request for reading iTunes playlists.

Good question — I meant that

Bodysoulspirit's picture
Submitted by

Good question — I meant that Make Tree from XML would handle property lists stored as XML text, like the iTunes file. But usually macOS stores property lists in a binary format.

Ok yeah as you said, I travelled the computer a bit and the iTunes library pretty much seems like the only xml property list, its icon also says XML where most other say PLIST. The funny thing is when you open all those PLISTS with Text Edit you get a binary like thing, but with a spacebar hit you actually see all those plist structured as XML even with the XML name at the top of it so that disturbed me ;)

But yeah if the iTunes library is the only XML Plist I would personally NOT put a constant extra port on the Make XML Tree node for that case only would you ? Don't know.

I was actually just testing the XML tree with the library but I personally have most of my music in Spotify so that would require DB which is another Feature Request anyway.

So perhaps either more have a separate "Make Tree from Plist XML" node which the user could use to build up some iTunes library like database for vuo, or a "Make Tree from Plists" as a separate Feature Request, or wait for that "Read iTunes playlists Feature Request" by @vascoarmartins to be implemented.

Actually why I played with the iTunes database was because I wanted to retrieve the Name And Title from the songs which does not come up with the "Get Audio File Info". That could be added there, as a feature request to get ID tags, but I've read iTunes has a special ID behavior though and more plays with it's iTunes library so there would always be people to want ID tags and others to be able to read the iTunes library.

Not sure how much people in the community use iTunes tough ;)

OK, Jamie... the first XML

cwilms-loyalist's picture
Submitted by

OK, Jaymie... the first XML was working the way I expected however I was using the output of the first XML as an input for searching the second XML and ran into a stange issue where it's returning an empty list. I can't figure out why... it looks right to me. I've gone over it about 30x and made changes to the naming I was using in the XML to simplify things and everything and I can't figure it out. Any ideas what I am missing? It's probably simple. I'm really confused because after your suggestion above about using quotes I had no trouble getting the first XML to work and the second one is even more basic than the first.

Feature status

When we (Team Vuo) plan each release, we try to implement as many of the community's top-voted feature requests as we have time for. Vote your favorite features to the top! (How do Vuo feature requests work?)

  • Submitted to vuo.org
  • Reviewed by Team Vuo
  • Community voted
  • Scheduled for implementation
  • Released in Vuo 1.2.6

Who voted?

mnstri's picture
stylique's picture
gabe's picture
volkerku's picture
Mika-TS's picture
jvolker's picture
tobystereo's picture
scuenya's picture
Project26's picture
oneseconds's picture
1000.Watts's picture
alexmitchellmus's picture
Bodysoulspirit's picture
2bitpunk's picture
MartinusMagneson's picture
kyrrelys's picture
cwilms-loyalist's picture