A node for loading structured data from file, URL, etc.
Imports: XML, JSON, CSV, TXT, etc. for parsing Filters: ability to filter structure data streams Multi dimensional: ability to have nested data (not just 1-dimensional lists)
Could go as far as ability to connect to DB's but that's probably pushing the complexity of this feature.
Comments
I see now that the [FAQ])
I see now that the FAQ and Roadmap pages lists "File I/O" as a possible 0.5.10 feature and the "Dictionary types" for 0.5.9, which hopefully cover most of the ideas of this feature request!
Nice feature request.
Nice.
The DB connection reminds me of my only foray into Kineme/QC development, which I never finished. :(
@joeladria: You're right. We
joeladria: You're right. We're already planning to implement much of this during the next few releases. Let's reexamine your structured data needs after Vuo has dictionary support.
Could you submit a separate feature request for database connectivity? Which databases would you like to connect to (ODBC, SQLite, ...)? If you could also provide descriptions of a few scenarios in which you'd use it, that'd be great.
Hey Joeladria,
Hey Joeladria,
We looked into what it would take to pull this off — it's no small task! We're interested in making this happen but won't be able to pursue it until well after 1.0. We're going to table this feature request for now and look at it again after 1.0. Thanks!
Now that we've released Vuo 1
Now that we've released Vuo 1.0, we've opened this feature request for voting.
I think you don't need all
I think you don't need all kinds of formats(xml, cvs, json) supported. a simple txt file i/o would be fine. Existing nodes such as split text could be used extract the data to a list. This is an absolute must in my view, things like storing and loading preferences, logging events and so on. v.
As discussed with @jstrecker
As discussed with Jaymie here on this Websocket feature request : https://vuo.org/node/276
I was wondering if this would include HTML structure parsing for websites ? Something like XML parser from the Kineme Network Tools ?
Or should I make a seperate feature request for online website structures ?
And as volkerku said, before some specific nodes for every kind of different structures, wouldn't it be great be able to open local or online XML, Texts, DB's HTML and JSON things and retrieve some rough unsorted (or savage parsed) I/O text file where we could extract some data using text sorting descriptors or structure get items ? A faster easier node to wait for some more powerfull one's.
khenkel released a text file open node as far as I remember, but it will only open real text files, and not turn all files into rough I/O text.
So this would be a big mix of the Kineme File Tools and the Network Tools
@Bodysoulspirit: Yes, this
Bodysoulspirit: Yes, this feature request includes the ability to retrieve data via HTTP, and the ability to parse XML.
@smokris Ok great ;) !
Steve Ok great ;) !
This is great, but I guess
1 - What is understood with "Add filters to Parse" ?
Will it allow us to retrieve some specific data from those structures ? O can I encourage people who voted on this to also vote for Request by Steve here : More List Nodes
2 - Steve : If this will load HTTP too, could you please update the title for this feature request too ? Might be useful for people to vote for this !
@Bodysoulspirit, I updated
Bodysoulspirit, I updated the title to mention HTTP(S). For structured data loading to be useful, yes, we'd need to implement some additional list nodes. When we're plan the release that includes structured data loading, depending on the voting results, we'd either implement that entire other feature request, or just the most important parts needed for working with structured data.
This is one of the last major
This is one of the last major features I'm looking for to complete the transition from QC to Vuo for our major annual project that I direct and produce. Really hoping this is implemented by the fall so I can finally finish the transition and drop QC for next year. Here's hoping!!!
This feature would greatly
This feature would greatly extend Vuo's usefulness, and genuinely add value. I would be using it tonight, instead of hacking together a Serial bridge in Python. :)
(Don't get me wrong - not hating on Serial - Serial support super helpful and very glad to have it.)
Hey, thanks again for all the
Hey, thanks again for all the amazing features you've been adding to VUO over the past months! I can see you've all been very hard at work adding some amazing features to the Nov 2015 release! I figure "Loading Structured Data" hasn't even been slated yet but I'd like to say that I am still VERY interested in using XML importing and parsing for a large project, then I can completely replace QC. It would be amazing if it can be put on the list for the next update cycle! :)
+1 Chris!
+1 Chris!
I'm so needing file input that I am considering adding data to a custom node! ;-)
Most of the libraries are
Most of the libraries are already used in Vuo correct? I think I saw LibXML2 show up somewhere?
We weren't able to get this
We weren't able to get this entire feature request into the upcoming Vuo 1.2 release, but we were able to implement one key piece: reading text from a file. With the existing text nodes, you'll be able to do some rudimentary parsing (split by line, split on commas).
I don't want to put my foot in my mouth by promising anything prematurely, but I can say that since this feature request is one of the top-voted ones (#7 overall, #4 among features up to a few weeks of work) it's high on our list of candidates for Vuo 1.3.
alexmitchellmus, keen observation. Yes, the
Fetch RSS Items
node does some XML parsing. And we already use the json-c library in many places.Awesome Jaymie!
Awesome Jaymie!
Chicken or egg kind of stuff- but list in list is a very necessary data type for this feature as well. No point loading in 1D values... Always more fun with friends. Keep up the great work.
Is the load text file in alpha 4?
list in list is a very
Yep, adding structured data support beyond 1D lists is one of the challenges of this feature request.
Yes indeed.
Just playing with the new 1.2
Just playing with the new 1.2 A4 vuo.data.fetch, great job guys!
One question for loading structured data: will we have the ability to use Keys to access data? And if so can the data be output in correct Vuo parlance? (if formatted correctly)?
XML Such As:
Would output if using a "Vuo.list.get.with.key" node (yet to be created):
This way one can use Index & Keys to control certain aspects of complex compositions. (I already do this for controlling compositions in QC- so I really would love to be able to rebuild my work in Vuo!)
In the above example, bar could be the key for the x,y position of a layer, so then I could add multiple values for bar- and simply move through them via indexes! If an index does not contain a 'bar' key then 'Vuo.list.get.with.key' wouldn't spit out anything.
UPDATE: I think this is covered by the request "Dictionary Support", however I believe it may have low votes due to assumption that this feature request would encapsulate the other? I don't know how useful a list generated from XML would be without its dictionary.
Chosen to be implemented.
Chosen to be implemented.
Does this include saving data
Does this include saving data as well?
Thanks all for your feedback.
Thanks all for your feedback. Update on the plan — it includes:
Save Data
node to save to fileSave Data
andFetch Data
(read from file) nodes.New, related feature request: Advanced HTTP(S) query nodes
As list-in-list will no
As list-in-list will no longer be developed in 1.3- can this node be easily updated from a 1D list to "n"D list?
@alexmitchellmus, I'm not sure
alexmitchellmus, I'm not sure what you mean. Maybe it would help if you gave an example / describe the task you hope to accomplish?
@jaymie. In xml you can
@jaymie. In xml you can have an item of item:
<red><blue><green>"1"</green></blue></red>
So green 1 belongs to blue and blue belongs to red.
If you read out red you get a list of blue, which has a list of green which is value 1.
So my question here is if XML is developed in 1.3- then won't team Vuo need to re-code aspects of it to work with list in lists?
This is to say the above list would not work with 1.3 XML reader as its not a 1D structure? If I am wrong let me know.
` 1 100 `
This is the first 3 cues from a show I worked on (Wonder if anyone can guess the show?!). Here you can see the main XML element is data, then each other element can have its own elements. In QC I can use this method to cue animation, or switch off parts of the composition. Its very powerful as all I need to do is edit the xml file- and my whole show is there. I also don't have to worry about cue numbers, as if I want to insert a cue I simply do. Yet if I wanted to I could simply call the scenes cue numbers anyway. Does this explain a bit more Jaymie?
<data>
<blackout>
</blackout>
<sunrisePrideRock>
<prideRockIn>1</prideRockIn>
<playTo>100</playTo>
</sunrisePrideRock>
<blackout>
</blackout>
</data>
This is awesome that XML is
This is awesome that XML is going to be supported this next update! Great news! I am so looking forward to moving a project into VUO from QC and this is going to be a huge step forward in doing that. Thank-you!
@alexmitchellmus, with this
alexmitchellmus, with this feature request, a composition to parse the "pride rock" example you gave would work something like this:
Fetch Data -> Parse Tree
Find Tree Item with Name
Get Tree Item Content
Instead of multidimensional lists, you'd be dealing with the Tree data type. (That's what I meant by "higher-level types" in my comment above.)
I'm really excited for the
I'm really excited for the ability to parse CSV files. I'm working on a number of video tools to generate special effects. and CSV tables are ideal for structuring the actions to be taken to multiple input streams of video, Each line of the cvs table specifies the actions for the input streams.
@Jaymie if XML parser looks
@jaymie if XML parser looks at the xml like a tree structure, can we also output a tree from an index? Such as asking the parser for XML item number 3? Etc?
That is how I would need to use the XML as well, moving though the elements as both index and dictionary support.
So that means I could
find tree item with index number
.@alexmitchellmus, yes, we
alexmitchellmus, yes, we plan to support accessing tree items by index. You'll be able to feed a tree (or sub-tree) into a node that outputs a list of the tree's children, which you can index into with
Get Item from List
.I need this too, so happy it
I need this too, so happy it's on it's way. Where do we download alpha/betas?
Do we know what release
Do we know what release version this feature has been scheduled for? The project I need it for isn't until early June 2017 but I wanted to make sure I had enough lead time to test and ensure my composition will be working smoothly first.
I found myself in a very awkward spot if I stay with QC for this year because the composition I am running not only needs a computer with a very specific set of ports but also one able to run OS X 10.7 for QC feature compatibility which leaves me a very short list of Macs this composition can even run on. My 2011 17" MacBook Pro is dying a slow and painful death and so I am hoping to switch over to VUO which makes my list of possible compatible Macs much bigger. Here's hoping it's coming soon.
@Chris, it's scheduled for
Chris, it's scheduled for Vuo 1.2.6.
Alastair, it's been a while since we've done an alpha/beta release, but they are on https://vuo.org/releases .
Somehow I missed your
Somehow I missed your response but am SO glad to hear it's coming in Vuo 1.2.6... if you need any alpha/beta testers I can do a little testing with my planned project. :)
My initial need will just be to have my project read a datamatrix code with a camera, extract a 4-digit code from the delineated text string and then use that 4-digit code to find the appropriate replacement text in an XML file for displaying on screen.
@Chris and anyone else who'd
Chris and anyone else who'd like to be testers — we just published a Vuo 1.2.6 alpha release that includes XML/JSON nodes. We'd appreciate if you could try them out and let us know if they do what you need them to do.
The final 1.2.6 release will include much more stuff, including CSV nodes, but we wanted to get your feedback on the XML/JSON nodes sooner.
If you find any problems with the alpha release, please post a bug report. If you have questions/suggestions about the scope of this feature request (what the new nodes will/won't be able to do), please comment here. For other questions/suggestions, probably best to a new discussion thread so it's easier to follow the conversation.
Great!!! I'll try it out as
Great!!! I'll try it out as soon as I can! I'll try and set some time aside tomorrow to do that. I'm away next week so I'll really try and squeeze it in. This is amazing!
I'll have some time over the
I'll have some time over the next week (after election campaigning on Climate action) to check it out. I have a few interesting datasets I've been wanting to get into Vuo. I tend to use CSV out of spreadsheets but it's easy enough to use GREP to XMLify them.
I'm having a level beginner
I'm having a level beginner question about XML trees. I'm trying to get the Artist Name, Song Title and File Location for the iTunes songs from the iTunes Library XML.
I'm able to somehow locate the subtree of a song, but then I can't find how to locate the Name or Title because they all start with subtree "Key" as name, then have "Name" or 'Title" as content, but the title or name itself is yet another subtree with name "String", all the same again on the same line.
Should I be using Split Text from here on ? Can't I do this because the way the XML is written or because of the node not considering the same line as a subcontent ? Thanks
Joining the composition, it assumes your iTunes Library is in -/Music/
2017-03-10 13.42.48.png
iTunes Library XML Tree.vuo
I may be misunderstanding how
I may be misunderstanding how to use the
Find Subtrees using XPath
node. I've created an expression that I thought would allow me to find valuecode
and it's sibling valuesfullname
andschool
but it isn't working like I thought it would.The way I want it to work is to search by the program

code
and return with thefullname
andschool
values that are it's siblings.VUO XML screen grab.png
XML Data.png
XML Test.zip
@Chris, you need quotes
Chris, you need quotes around part of the XPath expression — replace
/diplomas/program[code=scan]
with/diplomas/program[code='scan']
.@Bodysoulspirit, the iTunes
Bodysoulspirit, the iTunes music library is in a special (Apple-specific) XML format called a property list — which is convenient for some purposes, but unfortunately not for locating items in the XML.
Possibly we could enable the
Make Tree from XML
node to detect if the XML is a property list, and if so, turn the keys into element names and the values into content.Anyone else interested in using the iTunes file or other property lists in Vuo?
@Bodysoulspirit, the iTunes
Thanks for the info about that !
Ok so either like that or as a specific
Make Tree from Apple Property Lists
node perhaps ? Would perhaps keep the regular node faster if less code ?Well ok if this is so specific to Apple I could imagine a new feature request though if it is more work !
So you mean all other .plist OSX files would work as well ?
Somehow related interest from vascoarmartins : Read iTunes playlists Feature Request
So you mean all other .plist
Good question — I meant that
Make Tree from XML
would handle property lists stored as XML text, like the iTunes file. But usually macOS stores property lists in a binary format (if you try to open the file in a text editor, it's just a bunch of weird characters), so that would be a separate node, beyond the scope of this feature request.Thanks for pointing out the existing feature request for reading iTunes playlists.
Good question — I meant that
Ok yeah as you said, I travelled the computer a bit and the iTunes library pretty much seems like the only xml property list, its icon also says XML where most other say PLIST. The funny thing is when you open all those PLISTS with Text Edit you get a binary like thing, but with a spacebar hit you actually see all those plist structured as XML even with the XML name at the top of it so that disturbed me ;)
But yeah if the iTunes library is the only XML Plist I would personally NOT put a constant extra port on the Make XML Tree node for that case only would you ? Don't know.
I was actually just testing the XML tree with the library but I personally have most of my music in Spotify so that would require DB which is another Feature Request anyway.
So perhaps either more have a separate "Make Tree from Plist XML" node which the user could use to build up some iTunes library like database for vuo, or a "Make Tree from Plists" as a separate Feature Request, or wait for that "Read iTunes playlists Feature Request" by vascoarmartins to be implemented.
Actually why I played with the iTunes database was because I wanted to retrieve the Name And Title from the songs which does not come up with the "Get Audio File Info". That could be added there, as a feature request to get ID tags, but I've read iTunes has a special ID behavior though and more plays with it's iTunes library so there would always be people to want ID tags and others to be able to read the iTunes library.
Not sure how much people in the community use iTunes tough ;)
@Chris, you need quotes
Thanks Jamie! This worked, looks like it's going to do what I need it to do. Fantastic work!
OK, Jamie... the first XML
OK, Jaymie... the first XML was working the way I expected however I was using the output of the first XML as an input for searching the second XML and ran into a stange issue where it's returning an empty list. I can't figure out why... it looks right to me. I've gone over it about 30x and made changes to the naming I was using in the XML to simplify things and everything and I can't figure it out. Any ideas what I am missing? It's probably simple. I'm really confused because after your suggestion above about using quotes I had no trouble getting the first XML to work and the second one is even more basic than the first.
XML Test.zip
@Chris, one of your XPath
Chris, one of your XPath expressions contains smartquotes, which our parser (currently) doesn't recognize as quotes.
In
(Loyalist )BackgroundGraphic.vuo
, replacewith
Thank-you!!!! I don't know if
Thank-you!!!! I don't know if I ever would have noticed that! It's working now!
We just released Vuo 1.2.6
We just released Vuo 1.2.6-alpha2, which includes Table nodes for CSV/TSV, plus fixes to issues people reported for the Tree nodes (thanks!).
Please let us know if you have questions or find any bugs!
Released in Vuo 1.2.6
Released in Vuo 1.2.6
Cool !
Cool !
Example composition for
Example composition for loading a CSV of data and scaling it all onto a graph which animates though different years of the data points. Option to make a spline or line graph.
https://vuo.org/node/2016