discussion around collection-filters improvements

[AlicVB]

Hi all,

Note : Please apologize for the way too long message here. I wanted to make something more compact, but failed

First, I really thanks all posters in A calm, rational and respectul discussion about the new collection filters for the tone of the discussion. It’s pleasant, really ! Let’s hope we can continue on that way

To keep the discussion as clear as readable as possible, we propose to split the discussion in 2 parts :

• the collect-filter-history architecture HERE
• the design of the new widgets themselves in another thread : discussion around new filters widgets

Because, the time has come to go forward, in whatever direction… For that, we -the devs- really need to have a clear view of the good and bad points of the current implementation, after some time of use.

Reasons and workflows

Many legitimate questions have been raised about the reasons behind all this change… Here are some answers:

1. Collect module has limitations

• the combination of items is not obvious (hidden under the “triangle”) and the “mathematical” symbol are meaningless for many users
• The usage of a “text only” final parameters and the limitation of the design to text or lists make the interface quite hard to use for selecting range, multiple values, or some items.

2. Collections and filters are 2 separates modules on purpose (this was already the case before with “collect” and “topbar rating”).

• Changing things in a module doesn’t touch the other (resetting or changing directory in collect let rating unchanged)
• Idea behind this 2 steps is that collect allow you to get all the image of the subject you are working on. Then filters allow you to temporarily see only a subset of them for specific purpose (culling, batch edition, …)
• Internally they share the same code (that was not the case in 3.8)

Disclaimer : I’m not fluent enough in English to determine the best wording here (collect-collection-filters-filtering-…)

3. Relation between lateral filter module and topbar :

• There is no redundancy in the code : everything happens in the lateral filter module
• topbar widgets is just a container for lateral filters that are “pinned”

4. Widget changes :

• They are designed to reduce the number of click, and in some cases to allow more flexibility (range selection, specific values, …).

5. Presets

• now that filters belong to a “real” module (not just the topbar) they can benefit of the preset system.

6. History

As collect and filters are different actions and modules, their history are different too. We have envisaged multiple solution here :

• each module has it’s own history, integrated to the module panels (default for 4.0)
• keep a separated history module that would integrate both data. We can even imagine to have a tabbed interface or module options to choose to show only collect history, only filter history, both. This idea has received a so strong opposition, that we have not even started to try it…
• keep the previous history module for collect only (available in 1 click via the modules options). This put back the “1-click” and “direct visual” feature but it add some confusion for the collect-filter separation

Summary of users’ feedback

I’ll try to resume the ideas posted on darktable channels … Please apologize for all the missing information

For the new filter module itself :

++this allow more complex filters (more choice and combinations)
++this is easier to use than the collect interface (the widgets and the combination)
++this make some workflows easier to handle due to the new filters
++the ability to get back the history module

-the separation between collect and filters is not obvious
-it’s confusing to have parameters to gather images in 3 places
-the relation between the topbar and the lateral filter module is not obvious
-the changes about history is confusing

? multiple sorting feature
? historic / presets for the filters too

For the new widgets of the filters : see here : discussion around new filters widgets

As you can see, and that’s expected, we have contradictory returns in some cases, but that give us a good starting point to get a clear view (if you think that we have missed some important points, just say it now

And now ?

Next step will be to try to aggregate all this into something that would enhance the current implementation to be even better

What would be great is to have more feedback on the specific points :

• More workflow examples (tested ones…)
• How often do you change the items in the filters modules
• How many item do you use in the collect module, which ones
• How many items do you have in the topbar ? Which ones

And of course we would like to collect your opinion and ideas on how to make this architecture even better

Thanks for your support !

[AlicVB]

Let’s start for me
My workflows :

1. holiday photos : they comes from 3 different cameras (1 Reflex, 1 “compact”, and 1 phone) with different quality and resolution (Reflex are “top”, compact" are “good” but not enough for big prints, phone are just good for screen display). I usually start by culling the interest of the images (rating) and the type of images (color-labels, 1 for those with aesthetic interest, 1 for those with “souvenir” interest, 1 for “abstract” images, 1 for image part of a pano). Then I start editing them. Once done, I will pick up edited images for different purpose (slideshow, book, 5-6 best ones to send to family)
2. more “artistic” : I esp. love “abstract” images and b&w ones (they can combine) : If I want to edit some images, I just use the “abstract” color-label. If I need some for print, book, illustration , I just pick one the already edited + abstract image.

Collect module : I only use the directory item or no item at all (to show all images)
Filters : rating + colorlabels + filename (for the extension part) + history + ratio
Topbar : all the above filters

I have very rarely touched the filters modules in the left module itself once the item defined (once to add ISO, once to add search)

Considering the users returns, my usage and the initial goals, I have 2 main proposals :

1. the “enhance” one :

   • find a way to make more clear the “2 steps” process
   • simplify the filters modules in left panel (remove some features like multiple sort, item disabling, maybe even combination operator, …)

2. the more “aggressive” one :

   • consider that filters should be dead simple, available quickly, and their choice are a “one-time” parameter. So…
   • remove the filter module in the left panel
   • add a “configure” button to the topbar to select which widget to show there and in which order. Eventually in a second time allow presets and history.

[Terry (Terry Pinfold)]

Firstly I would like to express my appreciation to the developers for the improvements they have made to DT in the area of collection filters. I do not want the DT developers to create a free version of Lightroom, but what I miss now that I have moved over from LR is the ability to filter based upon tags, cameras and file extensions. With LR I used filters instead of collections to locate the images I wanted quickly. For instance I might want all my photos from Thailand (tag) and all the panorama images (tag). This strategy just worked so well for my needs. But I am happy to accept whatever the generous developers are willing to code for. Thanks

[g-man]

Right click on the tag and select “go to tag collection” OR go to the collections module and select tag instead of filmroll, folder.

[Terry (Terry Pinfold)]

Thanks for this advice. I will keep that in mind. However, I still feel having tag as an option in collection filters would be good.

[g-man]

Workflow

I also have 3 cameras when for family events and stuff. My workflow issues start at import. Since I can have 2 cards to import from the same event and I want a meaningful filename and folder structure. I like the copy & import feature to rename the images and place them into a folder structure. This means that the whole concept of filmroll is useless and confusing to me. I dislike that darktable defaults the collection module to filmroll.

Collect module: I use folders. Sometimes I use tags, but most of the time I go to the tag module and right click from there.

Filters: ratings and filename
Top bar: ratings, colors and filename. (but I dont use the colors)

I think there could be a 3rd proposal
3. move most of the collection items (metadata, capture time), into the filter module. The filter then applies to the entire database instead of just the collection. Lastly, make a preset for the filter module that is: “last imported” which is similar to filmroll today.

I think the last imported would be useful for the users that just want to use the darkroom to edit an image and not do any of the lightable functions. Maybe even have a database off option in the global preferences so only xmp are used.

[paperdigits (Mica)]

@AlicVB

topbar widgets is just a container for lateral filters that are “pinned”

Can you explain why this decision was made? I’d think that the top bar should have its own hamburger menu to control what is shown.

If I’m in the module and have a bunch of pinned widgets that I’m not using, that vertical space is occupied but not used. Vertical space in the sidebars is gold.

[AlicVB]

@paperdigits

Can you explain why this decision was made?

That has been done mainly to avoid having 3 stacked filters codewise. Especially for the ordering : it’s actually already quite confusing to have filters in many places in the interface (that’s one of the drawback of what we have imho) but if you add the fact that filters in the lateral bar are done before filters in the topbar (or the contrary) that may be worse !

@paperdigits

I’d think that the top bar should have its own hamburger menu to control what is shown.

I think so too (and for the record, that was planned as a possible future improvement of the topbar)

Just out of curiosity, as you speak of the lateral filter, I suppose that you use some filters in the lateral module, that are not pinned in the topbar… can you develop on this point ?

[paperdigits (Mica)]

Thank you for explaining it!

@AlicVB

Just out of curiosity, as you speak of the lateral filter, I suppose that you use some filters in the lateral module, that are not pinned in the topbar… can you develop on this point ?

I just updated to 4.0 about 2 weeks ago, so I don’t know that I’ve found the optimal usage pattern.

My topbar usage is star ratings, color labels, and sort order.

The collections are usually selecting a specific tag or selecting all images that have any color label applied.

Then it is some combination of color label or tag to narrow things down.

I generally like to have the image information module open and have customized that to my liking, but it still takes up a fair bit of space. So having the collect module take up space with unused pins for the top bar is not idea.

I think where my dt use might be slightly different is that I only have a bit more than 1000 images in my library. They are tagged and color labeled very specifically and I like to refine my collections down also very specifically. I’ll often have some specific tag for what I’m working on.

I use the collection history a lot, but really only between a handful of different views:

• newest images (filtered by color label/tag)
• recently modified (filtered by color label/tag)
• color label: all (filtered by some random one off criteria)
• tag(s) (filtered by some random one off criteria)

Getting to the history list is now an extra click compared to before, and augmenting the filtering criteria often pushes these four basic queries out of the history view.

Thank you for continuing to be open and collect use cases, that is super awesome of you and contributes to making our community and application stronger.

[Terry (Terry Pinfold)]

@g-man

The filter then applies to the entire database instead of just the collection.

This is a great suggestion and dare I say it would replicate one of the better features of Lightroom. As I have said before, I do not want the developers to create a free version of LR, but it doesn’t hurt to adopt some of the strengths of other programs into DT. In my case I have tens of thousands of images in the database and being able to filter the whole database would be so powerful.

[Terry (Terry Pinfold)]

@paperdigits

I’d think that the top bar should have its own hamburger menu to control what is shown.

I would support this idea if it could easily be implemented and maintained.

[wpferguson (Bill Ferguson)]

@AlicVB

Collections and filters are 2 separates modules on purpose

I’m glad to see this. My understanding (or misunderstanding) was that filters were being absorbed by the collection module and would just be additional parameters.

My workflows:

1. Sports:

• Import 1K - 9K photos shot with 1 or more cameras using RPD and a custom and per camera naming scheme into a directory. Import the directory into darktable. Filter the collection for the images rated in camera, select the ones I want to send to the newspapers, process and export them, then send them off.

• Use full preview and cull the collection, rating the images I want to keep as 1 star, After culling, filter the 1 star images the use the collection module to group them by ISO and apply the appropriate style, then start processing. After processing export and upload to web site.

2. Portraits:

• Import 150 - 1500 photos using RPD using a custom and per camera naming scheme. Import the folder into darktable. Use full preview and cull the collection. Select images that are similar, and may be scattered throughout the collection, and compare them side by side using a custom lua script named compare.lua that uses the lua filter API. So yes, my workflow is broken because the filter API has been removed. Once I select the best images I process and export them.

• The broken filter API hasn’t been an issue because I haven’t done any portrait shoots in the last several months. However, I have several coming up in the next month and it’s going to be a PITA working around this.

Last development cycle I was moving and didn’t have time to participate. This cycle I’m moved, but my house is a construction zone, so I may still not have much time to participate.

One last data point. My database contains 500K+ images. Making filters run against the whole database would probably not work well for me.

[Soupy (Tim)]

WORKFLOW

For portfolio/gallery images:
Rename photos date-originalname.raw, move to a folder based on category, import to darktable, apply a style to all, quickly view, rate 1 star those to be deleted and 2 those to be kept. Keepers then get additional editing, followed by another review, where worst images are rated 1 star and best 3 stars. 1 star is deleted, 3 is kept. If I want to keep an image that isn’t good it will stay 2 star. The absolute best of the best then get another round of editing for fine tuning - but there is typically no more culling unless duplicates.

For bulk/family images:
Same as above, but without the final round of fine tuning. Speed is the goal here.

For both: Some metadata is applied at import. Tags are applied after culling is finished.
For portfolio: Given title and description at export.

COLLECTIONS

I keep collection filters minimized and just use the old collections module. There, I switch between folder, film roll, capture date and tag. Why don’t I use collection filters? Because I don’t need all the different combinations of filters, so its better to minimize and save space. When I do need multiple filters at a time, it is typically one of the already mentioned four, plus rating in the top bar. I have been using range rating to give it a shot, and am liking it more and more - it is a faster workflow and I use it sparingly enough not to get too annoyed at the dragging. Normal rating is simpler and easier to understand, but that is perhaps only relevant for beginners. If I had the top bar open in darkroom while editing I would find the range rating brightness too distracting, but I hide it there, so again its not a problem for me. Sort is also good in top bar. I’ll sort by capture time, filename or rating.

To me, there should either be collection module, or collection filters module, but not both, as it is a duplicate of function.

What I strongly dislike is the removal of the old recent collections module. I used it a lot. Now it is an extra click on the history button - which means a lot of extra clicks for repeated use - very annoying.

[Terry (Terry Pinfold)]

@Soupy

Now it is an extra click on the history button

Maybe I missed something about working with the old recent collections module, but I find the history option works really well for me.

[Soupy (Tim)]

History (in the collections module) has the same data as the old recent collections, but the old recent collections was in a separate module you could keep open all the time - meaning you could switch from one history item to another in one click. Now it has no expandable state to always stay open, meaning it is always at least two clicks to get what you want.

[Pascal_Obry]

@Soupy

Now it has no expandable state

It is possible to get it in a separate module. See collection preferences under the hamburger.

[Soupy (Tim)]

Thanks!
IMO that should not be hidden away in some obscure menu, but solved the problem for me personally.

[Pascal_Obry]

@Soupy

IMO that should not be hidden away in some obscure menu,

Do you think calling the module preferences an obscure menu will add anything useful to the discussion?

[elstoc (Chris E)]

Those preferences (in the presets menu) are not very discoverable and I have had to point out their existence to a number of users who didn’t realise they were there. I understand the desire to reduce the number of options in the global preferences dialog but at least they were all in one place and easy to find before. But I agree that is a separate discussion from the collection filters module.

[Soupy (Tim)]

Yes, the discussion is about collection-filters improvements. Devs are asking our usage scenarios. If our scenario is to use the recent collections, and they are hidden away in a place unlikely to be found - then pointing this out could lead to a better design change, thus an improvement. OK, maybe it is about ‘collections’ not ‘collection-filters’ and therefore off topic - but as my original feedback was that we shouldn’t have duplicate of function in two different modules, it is not completely off topic. So long as there is duplicate of function, it should be expected discussions about one may inadvertently lead to discussions about the other - because improvements to collection-filters are not just considered in isolation (for one specific module, ignoring all the rest), but in relation to the whole program.