One of the advantage of using something like Julia is that you have instant access to good math and machine learning libraries. Iām afraid that at some point ML-based processing that is now mostly available in commercial software or hard to use research tools will become almost a requirement.
The CUDA stack looks good but not too sure outside of that. Another issue is the GUI.
For the pipeline side of things I think a node based approach (Ć la blender) works the best for graphics.
ā¦ in vkdt (as in any real time rendering framework, i think itās the first feature that people implement) you can hotswap your modules. itāll reload the changed shader code and rerun it in place, keeping your camera position/parameters/etc as they were. i know this is not what youāre asking but itās great fun for debugging.
If I understand correctly, a pipeline is a special case of a node system. If so, nodes seem a better idea.
Basically, a plugin takes as input image data (in some format, encoding, color space, etc) and a mask, and it outputs a new set of image data.
This seems unnecessarily restrictive. Perhaps a plugin could take any number of inputs, and create any number of outputs. Similarly, any number of UI inputs, and metadata input/output.
I suggest thinking carefully about version control, both of the core system and of plugins. For example, what happens when changes to one plugin breaks another plugin?
Do you want to restrict this to processing raws? Why not generalise to allow any image?
Node based editing is fun.
Natron, Nuke, Fusion all have communities of people who do develop nodes. Be it Nodes which contain node-sub-trees (its not a tree really but for brevity sake), or āplugin nodesā which do contain custom code for speed.
So if the user likes a particular assortment of nodes in a particular structure, you can save that and expose only parameters you want (also go in any time and change all parameters).
I bet a node with libraw could easily open those node-editors up for exactly what you want @mbs .
the guys over at acescentral develope their ānewā display-rendering-transforms partly in nuke iirc.
What I donāt like so much with node-based designed pipelines, is the fact that the usual constructs for flow control, that every programmer knows (tests, loops, conditional loops, breaks, ā¦) become a pain to use with nodes.
For instance, doing a simple loop with a break inside will be translated into a very complicated graph of nodes.
To be honest, I must admit Iāve never seen the interest of node-based pipelines, compared to the use of a basic script language, where flow control is often straightforward.
In the late 90ās, we were taught image processing with a node-based graphical system.
It became hell, as soon as you wanted to do more than connect simple processing nodes together.
Iāve seen plenty videos of people using Blender, who design quite complicated effects using nodes. They really get huge graphs. I admire them for their patience But I donāt understand how they can manage/maintain that in the long term
(not even sure if we can add comments associated to each node).
thatās the structure in vkdt too. i allow cycles to process animations/temporal feedback loops.
yes.
indeed. i think nodes are fun for illiterate users who just want to quickly bash something together. theyāll leave the actual programming (inside the node) for someone who knows what theyāre doing. that being said for a workflow-based application iād probably want to hide as much of the graph processing as i can from users too, and just work with very coarse presets/fixed blocks/a set of expert-curated graphs to pick from.
the node processing maps 1:1 to gpu processing. every vkdt node has exactly one compute/draw shader and can have multiple textures bound as input and multiple storage images as output. the gpu texture unit does all the work managing pixel formats. to implement even slightly interesting control flow, as you pointed out above, i need quite a bunch of nodes sometimes. i hide these behind āmodulesā which are the thing that is exposed to a user currently. that would be something like ālocal contrastā that actually consists of some 20 nodes internally. on top of that there are āpresetsā and āblocksā which consist of multiple modules with interconnections which can be inserted into an existing graph as a whole.
the thing with gpu programming is that control flow/branches/complex logic doesnāt really work well there in the first place (because of the heavy SIMT processing and branch divergence as well as incoherent memory accesses). so usually iād put my problem upside down to match a more simplistic processing logic before i even start putting it into code/nodes.
afaik most directed graph concepts let you put comments somewhere within/around the structure. Sure, code is more practical and efficient from a programming side.
I think there is a distinction between āregular usage of codeā for processing images and ādeveloping efficient codeā for processing images.
A directed graph is leaning more into the āregular usageā side with limited capabilities of ādeveloping codeā.
Since a node could/can (maybe must as a requirement) also contain code, I see no massive downsides to directed graphs and they would adress some/most of the stated issues that @mbs detailed.
A directed graph can be expressed in a structured text language, and a text language can be expressed as a DG (as a graphical image).
I last used DGs seriously about 30 years ago. They were invaluable for communicating complex technical relationships within engineering teams, and how those relationships changed as the project developed. They helped us understand how relationships could or should be rearranged for performance, system integrity and so on.
When flattened into stupidly large and complex graphics, we used them to impress, baffle and confuse senior management, ie project sponsors.
Last year I made a perspective matching system with Blender and Sverchok:
Keep in mind that both Blender and Sverchok are not my everyday thing.
Butā¦, it does function.
I could make this work without knowlege of complicated matrix calculations. I used a lot of nodes for translations and rotations ( especially the use of quaternions).
I would love to do something like this with GāMIC, but GāMIC is unfortunately too complicated for me to program in.
This is what audio video pipelines, like direct show, have been doing for years, and it works well! In this ecosystem, plugins are typically called codecs.
You have to be carefull on interfaces ācontractsā, maybe define categories for example to ensure a kind of consistency. A well-thought framework is mandatory.
Gnu Radio is another very good example.
Well in the example given of vkdt module, it seems that it is almost a plugin.
The glsl code can be compiled by the program itself and passed to th gpu.
The interface seems to be described by simple text file, so vkdt would be able to read it an create the interface for the module, may be with some limits in funcionality, but enough for making tests.
As you say it would be great to be able to test some contribution like sigmoid without waiting for developers to add it to main stream (in case the do).
As long as you are using not supported plugin, any problem it could cause would not be attributed to main stream modules.
A way to envelope that plugins in a security shell and catch exceptions to tell the user the problem comes from that plugin would be needed.