Lensfun: the past, the current state and the future

Software
,
1

Hi everyone,

this is a crosspost from the Lensfun mailing list and website but I wanted to share it here to reach a wider audience.

The Lensfun project was started by Andrew Zabolotny in 2007 to lay the foundation of a free and open source database for the correction of photographic lens errors. After Lensfun did not receive any more fixes and database updates, I more or less took over the maintenance from Andrew in 2012. Afterwards, my main focus was to continuously integrate the submitted profiles and database changes and to regularly release new versions to make these changes available for the application users.

From 2012 till today a lot of changes and improvements came to the Lensfun project. Torsten Bronger established the calibration webservice and GitHub repo where today most of the calibration work is done by a couple of people. He also dived deep into the maths behind the corrections and fixed many errors and inaccuracies.
Many other people contributed fixes, lens profiles and tutorials with instructions and scripts to facilitate the calibration process.

If we look at the number of lens profiles for each release, we can see that there is an increasing growth and the Git master database now nearly counts 1000 lens profiles.

0.2.6 350
0.2.7 374
0.2.8 441
0.3.0 580
0.3.1 667
0.3.2 714
Master 978

More and more image editors and even some commercial applications today use Lensfun and its database for their processing. That’s a great achievement and underlines the importance of the project.

In 2016 I started to refactor most of the library internals as it turned out that its structure cannot be extended, e.g. with the perspective correction now available in the alpha release, and was really difficult to maintain. A lot of code has been modernized and cleaned up to make Lensfun ready for the future. But unfortunately, the time I can spent in coding for the Lensfun project since then has decreased a lot and today the project is more or less stalled.

However, there is still a lot of work to be done. Just to name a few things that seem to be important to me:

  1. Create a new stable release

  2. Allow database lens entries that contain calibration data from various crop factors. Lensfun then just picks the closest one for each requested modification.

  3. Move the complete project to GitHub and modernize the project infrastructure and build system. Use Travis CI and Appveyor for testing.

  4. Develop a strategy how to decouple the database from the library source code to allow a faster release of new profiles to the user base. There is already the lensfun-updata-data script, but this only works reliably on Linux and not on OSX and Windows until now.

Currently, I do not have time to maintain the Lensfun project in way that is satisfactory for myself, for the community and for the users. This project really needs someone who takes a look at the bigger picture and has the vision and motivation to keep the project alive and in a good condition.

Therefore, I would like to ask the community to take over the Lensfun project management and maintenance in the next months. I will still be around and contribute from time to time. But to be realistic, none of the above listed changes will happen if no one else kicks in and does the job. Ideally, Lensfun is taken under the hood of a bigger project with experienced developers like e.g. Darktable or Rawtherapee that already makes use of Lensfun.

Let me know what you think and if you want to get involved!

Cheers,
Sebastian

16 Likes
2

well we can certainly offer help on all the infrastructure questions. like moving to a new hosting platform and CI.

And I would be happy to help to work on the data stuff. I find it super annoying how many places we need to update to support a single camera/lens.

5 Likes
3

that said @seebk will you be at the LGM in Saarbruecken in a few weeks? It would be really good to have you there I think :slight_smile:

2 Likes
4

@seebk we have already been talking about your mailing list post, but thank you for posting here as well, and 1000 thank yous for your work on the lensfun project.

3 Likes
5

@seebk just wanted to say thanks for all the effort. It’s easy to take it for granted when the program automatically adjusts for the lens’ shortcomings. Much appreciated! :smiley:

1 Like
6

I hope that someone will take over. I’m not able to do that but will keep working on the lens calibration script.

2 Likes
7

Thanks guys :slight_smile:

@darix I won’t be at the LGM although it sounds like a lot of fun :slight_smile: Did not follow the meeting for some while and wasn’t aware that it’s in Germany this year.
If you want to get in and help out in the Lensfun development that’s great. But the most important is to get a few people that make a plan and take care of the general maintenance tasks, coordinate the work and do this with a long-term scope. As I said on the mailing list, fixing a few things here and there and adding a new feature is one aspect. But making sure that it is all compatible between versions, everything works in the application context, that there are regular releases and fixes and that new features are maintained consistently is another story. And that’s where I see the biggest problem with my limited time.

1 Like
8

@seebk, sorry this took a while to post, I’ve been working/thinking about it and also been busy with work/life/etc.

I would appreciate any feedback you may have about these.

Community plans for Lensfun

Project management tasks (immediate need!)

  • Migrate away from Source Forge

    • Get project admin access to the SourceForge project
    • Make an announcement that we’re moving!
    • Open a Lensfun category for support/mailing list on discuss.pixls.us
    • Archive the SF mailing list mails
    • Shutdown the SF mailing list
    • Migrate the SF wiki
    • Migrate the SF issues
    • Migrate the SF source code repo (to GitHub or GitLab? Under the pixls.us organization?)
    • Migrate website (get a domain name? Or just use GitHub pages?)
    • Make an announcement with the changes made above

  • Recruit new developers

Longer-term plans

Community participation

  • Contact closed-source applications and ask them to contribute (we should make it easier to add camera support to applications like darktable and RawTheapee; currently we must touch muliple projects)
  • Contact lens manufacturers and ask them to contribute (particularly 3rd party makes)

Developer/DevOps tasks

  • Use continuous integration to build packages (GitLab/Travis/CircleCI)
  • Use continuous integration for testing (GitLab/Appveyor)

Developer tasks

  • Make a new stable release

  • Allow database lens entries that contain calibration data from various crop factors. Lensfun then just picks the closest one for each requested modification.

  • Develop a strategy how to decouple the database from the library source code to allow a faster release of new profiles to the user base. There is already the lensfun-updata-data script, but this only works reliably on Linux and not on OSX and Windows until now.

  • Figure out how to share data between projects that need/use lensfun

  • Make getting full camera support easier: exiv2, lensfun, darktable, rawtherapee, other photo applications. Possibly maintain a database and generate the necessary files for each application

7 Likes
9

@paperdigits: There is already a Lensfun organisation on GitHub which is used for Thorsten Bronger’s calibration service. I can get in contact with him and ask about his opinion to make this the official repo.

There are two more things to consider when moving away from Sourceforge:

  1. The lensfun-update-data script downloads the database from a Sourceforge http space. This has to be kept updated for a few years to make sure the script still works.

  2. The Doxygen generated API docs are also on the Sourceforge webspace.

Both could be hosted on GitHub Pages in the future. It is not an ideal solution as GH Pages is not intended for this kind of hosting. But I would prefer it compared to using a dedicated web server where only a single person has access.

The Lensfun website is based on Jekyll, so GitHub Pages is the ideal environment for this.

we should make it easier to add camera support to applications like darktable and RawTheapee; currently we must touch muliple projects

Make getting full camera support easier: exiv2, lensfun, darktable, rawtherapee, other photo applications. Possibly maintain a database and generate the necessary files for each application

Not sure if I understand this. As soon as the camera is in Lensfun database it is available in each of the applications… isn’t it? The problem in my opinion is to find a way to push the database updates to the users.

10

That’d be great. Thank you!

Ah, these are excellent points. Thanks. I think we can make github pages work for the doxgen generated stuff, or we can merge those docs into the jekyll site.

Sorry I wasn’t clear. Actually, both of those points are pretty much the same, with the emphasis on the second point-- it includes lensfun and other projects.

When a new camera/lens comes out, we need to make additions to quite a few projects so that the camera/lens is useable in applications like darktable and RawTherapee.

We generally need to:

  1. Collect a (raw) file
  2. dump the metadata and find the lens name.
  3. Submit the lens information to the exiv2 project to get the camera/lens name into exiv2 (darktable uses this to detect the lens name and match to the lens correction in lensfun)
  4. Shoot calibration images for lensfun, process them with hugin
  5. Submit the raw file to rawspeed/dcraw/other raw demoasicing libraries for support

I’m sure this isn’t even close to all the of the work that needs to be done, but it already touches multiple projects. I’d like to find a way to make all of that easier for people to contribute. Whether we just maintain a big database of cameras & lenses and other information necessary, it isn’t clear to me. But we need to do something in the future.

11

Amen.

The main reason why I want to push on this front is that right now it takes way too long from “We have the information available” to “users can use it easily”.

1 Like
12

Ok, I see, thanks for the clarifications. You are talking about all aspects of camera support including exif and raw decoding. Adding full camera support indeed is a long process right now in particular when considering that some Linux distributions like Debian are only updated in a 2-year cycle.

Regarding GitHub: I will move over the project from Sourceforge in the next weeks or so. Shouldn’t be too much effort I think…

1 Like
13

I would suggest gitlab over github. One point is their CI and the other is that moving gitlab instances is easy. You can export your data and import it again :slight_smile:

Btw. does lensfun already have support for Teleconvertors? I don’t own one yet but I wonder how the lens is reported.

1 Like
14

A suggestion: Split the code base and the database into two separate projects.

With this organization, a robust database community could contribute without messing with the developers maintaining the code. If the v2 database organization is stable, I think this’d be a good way to manage all this going forward. But, if/when a v3 database is posited, a new database project could then be started…

3 Likes
15

I would suggest gitlab over github.

I do not know GitLab very well, so it might have some benefits I do not know. But for Lensfun I think it is now best to be on a platform that is widely used to catch attraction of new developers. Once there is a stable team, moving Lensfun to GitLab or whatever platform is preferred shoudn’t be a big issue.

Btw. does lensfun already have support for Teleconvertors? I don’t own one yet but I wonder how the lens is reported.

Yes, but not with automatic lens recognition and you need to alibrate the combination of lens and converter. For example: lensfun/data/db/slr-nikon.xml at master · lensfun/lensfun · GitHub

A suggestion: Split the code base and the database into two separate projects.

That might make sense, but is a bigger step than one would imagine on first glance. It’s basically what I have summarised above as “Develop a strategy how to decouple the database from the library source code”. So I think that will be on the todo list but is one of the open items I never found the time to work on…

1 Like
16

gitlab allows you to login via your github accoutn e.g.

so there is not really a big hurdle for it as a platform.

17

@seebk any update on migrating to github?

18

I just moved the website and transferred the roadmap into a couple of GitHub issues with a next release milestone. Still have to look into transferring the issues and archives, but that’s nothing to worry about.

I think one of the most important issues and easy for someone from the outside would now be to setup Travis and AppVeyor on GitHub :slight_smile:

→ Setup continuous integration services · Issue #588 · lensfun/lensfun · GitHub

2 Likes
19

Awesome thank you!

20

How can someone get involved with Lensfun today? I’m having difficulties reaching anyone…

Thanks
Frederick Price