RawNeRF is able to combine images taken from many different camera viewpoints to jointly denoise and reconstruct the scene.
Sounds like a mean stack with some hugin alignment to me. 
Did you see the yt video ?
I can see how this could work really good in a cellphone camera. It can take multiple continuous raw captures and then process them for the denoise. Changing the focus was really cool to see.
Yeahā¦ A lot of the computational stuff feels like an effort to make up for having a tiny sensor that uses a high ISO all the time. I donāt need a lot of that with my large sensor camera.
these marketing department posts really trigger me.
i find it a stretch to call NeRF āgoogleāsā. it seems to be the hip thing to do for everybody for years now. also comparing a stack of 200 images to a single raw exposed up is not an acceptable comparison. if at all youād need to compare to (googleās) image stacking that is already implemented in pixel phones (and in vkdt). that said the raw rendition looks like it mainly profits from denoising the invisible negative values (below black point). averaging many independent samples is like a monte carlo estimator: noise levels reduce with sqrt(N)ā¦ means youāll get diminishing returns for higher sample counts.
i suppose you can just use this code (not from google) and feed it raw images. not sure what the technical contribution of the google project was, didnāt read the paper, i suppose linear data without black point subtracted has some subtleties to consider.
that said the nv code base is a bit heavy on the vendor specific dependencies for my taste, so i think running it separately is probably a better idea.
1 Like
new changelog:
- bugfixes (gamepad ui, db string lengths, supported image formats, etc)
- rate/label in dr mode too
- new bias parameter in film curve
- ui tweaks in the files view (wrt. import from card reader)
- straighten tool for image rotation
- ship bash script to create pure css html web gallery pages from exported images
2 Likes
Wow, itās so blazingly fast ! I already knew Iād impressed but on my ageing gear (i5 3470 + gtx 1650) the fluidity is unbelievable compared to DT 
Iāll have to get used to manipulate the pipeline (and the block concept) for the masking process but itās really giving me the wheel to explore it further more.
On the other hand the interface feels a lot more bare-bone and I didnāt stumbled across any convenient way to activate/deactivate a module other than removing it from the pipeline and re inserting it after, maybe Iāll have to dig a little bit more. I appreciate the many pop-up explanations.
On the technical side I had 0 problem following the git page to build and run it on my gentoo system.
Thanks for this really exciting software, hope youāll manage to go on working on it or that youāll manage to welcome contributors ! In the mean time Iām really looking forward to the moments Iāll manage to fit in my tight schedule to get used to using it and I hope Iāll be able to maybe transition from DT (after more than 10y of continuous amateur/hobby use and around 1800 individual edits) sometime in the future.
I think its very much a work-in-progress still in terms of the backend. The UI is barely there, but once the back-end is sound, Iām sure youāll start to see progress on the GUI.
Yeah, I did not meant it as a criticism but more as a disguised feature request as the very first thing I really felt was missing for me. But Itās already so new and ground-breaking for me that I did not dare to formally ask for anything
feature request noted. the thing is, in a graph where a module can have multiple inputs and outputs itās not super trivial to have a generic āswitch offā button that always works. switching the module off will maybe not generate that secondary output some other branch of the graph depends on. iāll probably introduce such a button for simple input/output chain modules only (i.e. has a main input channel and exactly one output channel).
Really weird
Starting program: /home/okke/build/vkdt/bin/vkdt -d all
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff7419640 (LWP 2716)]
[New Thread 0x7ffff6c18640 (LWP 2717)]
[New Thread 0x7ffff6417640 (LWP 2718)]
[New Thread 0x7ffff5c16640 (LWP 2719)]
[gui] glfwGetVersionString() : 3.3.8 X11 GLX EGL OSMesa clock_gettime evdev shared
[gui] monitor [0] HDMI-A-1 at 0 0
[gui] vk extension required by GLFW:
[gui] VK_KHR_surface
[gui] VK_KHR_xcb_surface
[qvk] error VK_ERROR_LAYER_NOT_PRESENT executing vkCreateInstance(&inst_create_info, NULL, &qvk.instance)!
[ERR] init vulkan failed
[ERR] failed to init gui/swapchain
[Thread 0x7ffff5c16640 (LWP 2719) exited]
[Thread 0x7ffff6417640 (LWP 2718) exited]
[Thread 0x7ffff7419640 (LWP 2716) exited]
[Thread 0x7ffff77e6740 (LWP 2713) exited]
[Thread 0x7ffff6c18640 (LWP 2717) exited]
[New process 2713]
[Inferior 1 (process 2713) exited with code 01]
(gdb) bt
No stack.
Huhu, no easy way out 
I though maybe some āpass throughā wires to patch in the pipeline view that would override the processing(s) (modules) in between itās input and output nodes but still let the module(s) in between function normally, except not delivering their output where the pass through wire is patched ?
I guess with my little experience of vkdt my thinking is not yet fully relevant but still it seemed an elegant solution to me
hmm missing the validation layers? the debug build uses them, the release build doesnāt.
right. leaving the graph connections in place and just calling this bypass connection during create_nodes with the input and output connectors of the module. the node level will then have bypass semantics (without paying for intermediate buffers). probably need to think about it for half an hour how to wire it in the core so itāll work for all modules which qualify.
Wow, that was fast, thanks !
haha in all fairness that function existed already for some similar purpose
Yes, thanks, vulkan validation layers were missing.
When I run gdb --args ./vkdt -d all which just shows a couple thumbnails (in ~/Pictures) and I close I see corrupted double-linked list as the last line (not sure if this is a problem). Though when running with gdb the window does close but Iām not sure if vkdt exits cleanly
vkdt_output.txt (3.1 KB)
with bt:
(gdb) bt
#0 0x00007ffff7a174dc in ?? () from /usr/lib/libc.so.6
#1 0x00007ffff79c7998 in raise () from /usr/lib/libc.so.6
#2 0x00007ffff79b153d in abort () from /usr/lib/libc.so.6
#3 0x00007ffff7a0b67e in ?? () from /usr/lib/libc.so.6
#4 0x00007ffff7a2126c in ?? () from /usr/lib/libc.so.6
#5 0x00007ffff7a21d34 in ?? () from /usr/lib/libc.so.6
#6 0x00007ffff7a21ec5 in ?? () from /usr/lib/libc.so.6
#7 0x00007ffff7a233d0 in ?? () from /usr/lib/libc.so.6
#8 0x00007ffff7a25a33 in free () from /usr/lib/libc.so.6
#9 0x000055555563c416 in dt_stringpool_cleanup (sp=0x5555556c4dd0 <vkdt+54072>) at ./db/stringpool.h:42
#10 0x000055555563bcfc in dt_graph_cleanup (g=0x5555556c2988 <vkdt+44784>) at pipe/graph.c:102
#11 0x000055555565fde4 in dt_thumbnails_cleanup (tn=0x5555556c2988 <vkdt+44784>) at db/thumbnails.c:183
#12 0x0000555555587759 in main (argc=3, argv=0x7fffffffdbf8) at gui/main.c:321
Not sure if this helps any. Iāll try to run with gdb to see if I see any direct crashes.
ah great, thanks for the update. the graph only holds a stringpool for debugging purposes. i just pushed a small patch that may or may not avoid this situation (double free, if thatās what it is).
btw temporarily disabling āsimpleā modules is now implemented:
as this button left of the expander, in the āpipeline configā tab. maybe i can relax what is āsimpleā enough a bit, for instance the secondary dspy output of the filmcurve module hinders this one to be āsimpleā.
3 Likes
Features works exactly as advertised The circle icon think is half drawn half masked by the next element containing the arrow and name of the filter.
oops, should be fixed now. didnāt notice on my screen res, as you can tell by the screenshot above.
btw other news are:
- an updated version of imgui that can actually display 32-bit unicode (where these on/off icons are from), so japanese characters should not be a hard thing to merge now (this was an earlier request at some point).
- came with a new key event system, so the gamepad/keyboard system is completely overhauled now. fixed a few things that went wrong in the process, let me know if you find anything else that seems weird.