In the last year or so several ideas for new darktable modules have “hatched” in pixls.us. Some made it, some are still in the “hatching stage” and at least one didn’t end well.
The process should be pixls.us → darktable issue → darktable pull request.
Let’s use the scene referred local contrast module as an example (since it’s right in front of my face )
In pixls.us:
The idea is getting hashed out with possibly 2 implementations or a possible combination of the 2. At some point it will settle and the developer will be ready to go forward.
darktable issue
At that point an issue should be raised in darktable RFC: New scene referred local contrast module. It should contain:
what is being proposed (new module providing local contrast in scene referred space)
why - (because we don’t have one)
description - (the features distilled from the pixls.us thread)
reference - a link to the pixls.us thread for someone that wants to spend an hour (today as I write this but I’m sure it will be longer) reading the whole thread or has questions and is willing to search for the answer
Now the devs can chime in with questions/recommendations/suggestions/etc. It’s also a place that technical issues can be raised and addressed before a lot of work is expended.
If issues do arise, then the devs are there to offer help/advice/suggestions/etc.
If at some later date there is a question about the decisions made with the module the issue serves as a reference point to aid in understanding why the module is the way it is.
darktable pull request
When this point is reached, everyone should have an understanding of what’s happening and pretty much be on board with it. Now it’s just code and testing.
This isn’t the law and I don’t make darktable “policy”. But I also don’t ever want to see another pull request end badly, and following these steps will probably prevent that from happening.
If there is already a proof of concept implementation (or it is trivial) submitting it immediately as an RFC PR, rather than issue, has the advantage that the code is right there to look at and try. That makes it much easier to agree on what’s actually being proposed/discussed. Yes, in a GitHub issue you can insert a link to a commit in your personal repository, but making sure you’re all looking at the same/last one just is an additional challenge.
Plus it sounds like the same thing wil be discussed in 3 locations? People will be talking past eachother. Gatekeeping PRs so something “wrong” doesn’t get merged won’t be enough; now we have to stay on top of (all!) issues too, because if you don’t chime in early enough, it will be a “decided” fait accompli and the resulting PR could get merged in record time, before you’ve even had time to look at it and do all the background reading.
What I’d like to see more of is actually getting the respective devs with most knowledge in an area involved (@ed) at an earlier stage. Like, if you’ve put a whole UI together through trial and error, it is much less appealing for me to clean up afterwards than if you’d reached out for advice earlier on. Yes, I know from personal experience that it can feel magical to figure out from scratch how you can force the thing to do what you want, but you may not know to test all the other things that don’t automatically work or get broken, until you built a solid understanding of the whole framework. There’s a fine line between encouraging new contributors and accepting amateurism. And AI will make this worse, because people will submit large change sets they don’t at all understand or even have the knowledge to test, with the firm belief it is high quality and maintainable, because that’s what the AI told them. Requiring signoff from each of the “system maintainers” seems to work for Linux, but this is of course up to our maintainer.
If you think that it is not worth involving the devs at an early stage in your project, then consider whether your project is worth it at all. Because at some point “the devs” need to get involved; either before it gets merged or afterwards to fix the problems it caused.
I think that’s important, but is there a list somewhere to let us newly involved people know whom to add for each area?
The lack of documentation is also an issue. I have to admit I’ve used an LLM to try to gather patterns and create some sort of documentation in a back-and-forth with the tool (when it created something that wasn’t clear, or something I knew to be plain wrong based on my limited experience). It’s in a private GitHub demo so I don’t pollute others’ understanding with potentially wrong information. @dterrahe , would you like me to share that with you so you can review it, and if it’s useless slop, delete it, and if it can be made usable via review and corrections, make it public (later, when it passes review)?
The problem I saw was features getting proposed in pixls.us and then getting stuffed into an existing PR that started as one thing but kept growing with lots of feature creep until it became something totally different than the original stated purpose of the PR.
But, on the other hand AgX worked when done that way, but up front the purpose was create a new module and all the proposed ideas and changes fit into that boundary.
Another issue is how many devs are going to want to work an issue in pixls.us?
I’d like to say that I appreciate the effort to formalize the workflow for the transition from pixls.us to darktable very much.
Pixls is a great place to start module development because we can have exhaustive and informal discussions and brainstorming with the user base.
And I agree that the convergence towards a pull request may be troublesome for the module developer in the one hand. In the other hand, it is important to prevent additional effort for the dt core devs, which requires a streamlined workflows. In particular the phase where some issue or PR in the repo exists and discussions on pixls are proceeding in parallel can lead to confusion and unnecessary workload.
The core process of contribution is already described briefly and well here: Hacking on darktable · darktable-org/darktable Wiki · GitHub
So a workflow starting at pixls.us would probably be a modification / augmentation of this.
That can indeed be an (independent) problem. I guess the local contrast rgb is also heading into this direction
It is great to try out different approaches and explore possible features.
But in the end, the result must be streamlined and simplified to solve the problem it was designed for without unnecessary bloat.
This process is especially painful for us developers because it means giving up a part of our effort and our “baby”.
The sweet spot between simplicity and versatility varies, however. It strongly depends on the Software’s design philosophy and, most importantly, the target user group - it must be a strongly evidence-driven process in commercial product development, but can of course be more opinionated in FOSS projects that are not constrained by the need to survive in the market…
In this regard, maybe it would be very useful to have some guidelines regarding the paradigms of darktable to support streamlined module design and have some commonly accepted principles to refer to. All official information I was able to find is this chapter Developer's guide · darktable-org/darktable Wiki · GitHub
I don’t want to play AP and I know the benefit may not worth the effort - I just have the impression this could be useful
In this regard, I also see the danger that someone puts much effort into a module here on pixls, just to learn that it must be completely redesigned to meet the design and interaction principles of darktable when making a pull request.
Having some guidelines beforehand may reduce the risk.
My intuition is also that having a three-step process like pixls → issue → PR is making things much more complex with marginal benefit.
As an alternative, having clear guidelines how to decide / find out whether the new code is ready for a pull request (regarding functionality, usability and design) may be helpful.
I don’t think it is. From the outset it’s being proposed as a new module, so as long as the changes and ideas stick to the module then that should be fine. Now if someone suggests changing the exposure module as part of this effort, then we might have a problem.
The problem is limiting scope. Adding a new module gives you lots of scope. The problem is knowing when to draw the line and say this is what we are going with (and then sticking to it ). I like the idea of @dterrahe’s RFC: PR. Maybe we don’t need an issue in between.
My thought with this is that the devs already use the issue → PR interface and it’s all in one place on github. If dev’s have to go back and forth between pixls and PR that may be an issue of stretching them too thin, especially @Pascal_Obry. Especially when the conversation on pixls is hundreds of entries long and takes hours to read. My thought of going to an issue was to take the high points from the pixls conversation and condense it to a manageable amount for someone to comprehend.
There is also the link above to pull request guidelines trying to find guidelines/solutions to managing this process.
I’m not to have strict procedures for working on Darktable. I understand that some people are not ok with that.
My only point is this : If a dev come with a PR he should never ever expect it to be integrated. He should be ready to take into account the comments and eventually trash the work if not ok or redo part or all of it.
I’m not sure we should go against that. A dev must be free to “waste” its time if he feels that’s ok. Again he should just not expect the work to be integrated without discussion or even integrated at all.
And after all even in this case the time is not wasted as there is some knowledge learned while implementing the PR.
So let’s be open minded on this and keep the Darktable development fun.
In my opinion, I think that the balance is obtained by the mutual feeling of "respect“. This is harder to achieve in text only, and with strangers are a rapport as not previously established.
What I mean by “respect” is that in software development, I think it should always be okay to disagree on the "code“. But if a member of the discussion feels like the disagreement is made on a personal basis, that is the moment I think people walk away.
The tricky part is that this is a two-way understanding. I can voice an opinion that I believe is purely technical, but the one receiving it can interpret it in a different way. I think it is the responsibility of both sides to allow some room for misunderstandings. Discussions are not a noise-free channel, so some extra “error corrections” are needed for the correct understanding.
Not sure, people walking away because they are angry. That’s their right, what do you propose? There is people we can’t work with, on a company we can force people to work on something. On Open Source projects we cannot and we don’t want… A fun project and working with freedom delivers the best of everyone to me.
And we have to keep in mind that some projects can be very large & tricky. And we sometimes simply don’t know if results are up to expectations. Just two examples from the last years, agx and color equalizer. Both projects started as a proof-of-concept with a long list of problems to solve. @kofa did this in a local branch with a lot of user-decisions, i did it a bit different but for sure we did not exactly know where things would end. And in the end the decision about integrate or don’t is up to dt maintainer All we other devs can do is, do as good as we can. (I can’t count my ideas shown as PR’s that were simply bad after discussing, but so what? Keep going and try better.)
Re-reading this thread as someone who is not a dev here, it appears to me that for devs it would be frustrating to have to follow a pixls.us thread and a GitHub issue in parallel.
Recently, I had an idea that hatched from a frustration, where I wasn’t sure whether I should open an issue on GitHub or run it up the flagpole on pixls.us first. I chose the latter. I figured it was best to not pollute the GitHub repo with an idea that might be thrown out the door. Call it pre-validation.
I think pixls is a good way to get informal feedback on an idea before it more formally reaches the devs. But once it makes its way across the divide into an issue or an RFC PR, it would be good if there was a post in the pixls thread to say that the discussion has a new home.
the art is to develop a clear scope before flooding GitHub with immature pull requests.
It’s quite common that simple solution designs gets overloaded with several good ideas resulting in increased complexity and then hard to review stuff.
In these early phase (quantity over quality) it makes sense to announce the topic as an RFC issue in GitHub but maybe direct to pixels.us discussions to avoid parallel streams. Usually prototypes here are used to outline the feature and sharpen the requirements for the solution.
And sometimes it’s worth to remember the ‚keep it simple‘ principle and the darktable way to modularize.
Once a solution converges it makes sense to head over to GitHub with a pull request to start elaboration of the technical solution. Maybe the solution for the requirements can differ from prototype implementations.
At this stage, it is more important to avoid scope creepers – which occurs more frequently in discussions on pixls.us than on GitHub
But it’s not about rules or policies but about finding the proper way…
I too have occasionally sounded out the community on this forum to see if ideas I have are worth asking a developer to consider. A few polite no way answers and a couple of success stories.
But the degree of progress with DT is incredible. It is definitely not a stagnant project.
)
All we other devs can do is, do as good as we can. (I can’t count my ideas shown as PR’s that were simply bad after discussing, but so what? Keep going and try better.)