LLM Co-coding and AI Slop: differences explained

Software
, ,
1

[This is a long post, but it is not LLM generated or even LLM reviewed. My thoughts, my words - possibly wrong]

I am under the impression that some of the concerns raised in LLM-related conversations stem from a superficial understanding of how proper LLM-assisted development looks like.

I would like to clarify a bit how this works in real life, and why, I argue, code produced with the help of an LLM can actually be better than code produced by a developer programming solo.

The AI Slop problem

Thanks to (or rather, because of) LLMs, it is now possible for someone (who may or may not be able to code) to write a prompt, and then create a PR (or even let the LLM create the PR) with whatever the model spits out. This can also go further, and one can ask the LLM to decide what to implement. The LLM can be unleashed on a code base, and create any number of PRs that fix (what they think are) bugs, implement new features, and so on.

The result of this process is what we generally refer to as AI slop.

This is what FOSS projects are concerned about. They are afraid of being flooded with thousands of automatically generated PR requests which will be a burden for devs. Even if these sloppy changes can be identified quickly (which is not always the case) it still takes time to separate the wheat from the chaff, and for large enough numbers this can completely paralyze a project.

The concern is not that LLMs generate bad code (even though they can certainly also do that). The concern is that LLMs make it possible to spam FOSS repositories in a way that was not possible before and create an unbearable amount of overhead for maintainers.

[Edit: @Phemisters pointed out that there are also other concerns, such as the ethicality of how models are trained. Sure, it’s a complex topic and there are many angles. In this post, I am focusing on aspects pertaining the quality of code contributions]

LLM-assisted development is not AI Slop

AI slop is NOT what a developer who wants to do something meaningful uses an LLM for. Instead, you co-code (or pair-program, or whatever you want to call it) with an LLM.

Proper co-coding with an LLM is a very intense, hands-on task, which gives a developer the power to implement stuff much faster, and, if done correctly, better than they would do without the help of the LLM.

This is a summary of how you implement something with an LLM:

  1. Someone (the developer) has an idea, writes a design sketch, more or less detailed, and asks an LLM to prepare a plan. If any part of the initial request is very under-specified, the LLM will ask clarification questions.

  2. The model reads all the relevant parts of the codebase, if needed does its external research (e.g., color science documents), and it produces a plan, i.e., a full-fledged design doc which outlines what the model is planning to do, why and how.

  3. The developer reads the plan and comments on it, asking questions (e.g., “why this is done like this?”) or correcting the model proposal (e.g., “This should be done like this, not like that”)

  4. Steps 2-3 repeat several times, until the developer is happy with the plan.

  5. The developer asks the model to implement the plan.

  6. While the LLM is working, the developer can read the thinking process of the model (I don’t like the word, but it’s what it is called). The model explains in great detail what it is doing and how. The sequence of the actions that it will do, what is the starting point, the intermediate steps and the final goal. If it has any doubt, it will start thinking very hard about something.

  7. At any point, the developer can interrupt the model while it is thinking, and offer guidance or clarification. At any moment, if they see something wrong in the thinking trace, they can interject and steer the model in the right direction.

  8. When the code is ready, the LLM builds the project.

  9. From here on, it is an iterative process of the developer (1) testing out the feature, and (2) doing edits and fixes in the code, either directly or with the help of the model.

As I mentioned before, it is a very intense experience. The model thinks and edits fast, iterations are relatively quick and there is a huge amount of information that the developer needs to assimilate.

The whole process is very transparent, and the model is only in control to the extent that the developer lets it be. While the model will certainly make its own decisions and propose plans and implementation details, in the end it is the developer who has the last word concerning how things are done, what goes in and what does not.

Potential advantages of co-coding

And why LLM-assisted code changes can actually be of very high quality.

Codebase awareness

An LLM can keep all the relevant bits of a codebase in memory (the context). Thanks to this bird’s eye view of the code, it has a good understanding of where the best place to implement a certain functionality is. It may not know this better than the maintainers, who - thanks to years of experience - know the codebase almost by heart. But it certainly understands the overall code structure much better than a new developer who is not familiar with the code, and can make better, more informed decisions.

LLMs can easily find snippets of similar/related code to the one that the developer wants to implement and reuse that instead of reinventing the wheel. This leads to less and more maintainable code. A human that does not know the whole codebase cannot do this as efficiently or as accurately.

Code style and guidelines

An LLM analyzes the codebase and adapts its style to conform to the rest of the codebase. A developer strives to do that, but if you come from a different code base it is not so easy to start doing things differently.

More robust code

Humans are notoriously sloppy when it comes to identifying corner cases. It’s just that any code of non trivial complexity has a huge space of possible states, and keeping track of all those states is not something that our brain is wired for. Coding LLMs are trained on huge amounts of existing code, are specialized in identifying the patterns that can cause issues and know how to build effective defenses around them. They are a great help in finding subtle bugs that for a human would be very difficult to spot.

Cut through the boilerplate, focus on the substance

Writing code involves a lot of boilerplate and repetitive tasks. This is especially true for languages like C that do not offer a lot of high level abstractions. When you co-code, the developer can let the model take care of the “boring” stuff while they focus on the “meat” of a change (i.e., UI design, which transformations to apply, how to effectively decompose the problem, etc.).

Faster iterations => More iterations => Better results

Being able to iterate quickly, not having to spend too much time typing boilerplate, enables a more explorative, less dogmatic approach to coding.

The developer can try out different ways of doing the same thing, because trying an option is relatively cheap. A developer is not as likely to do that if any refactoring is going to require hours if not days of work to implement and then debug the result.

The developer is no longer affected by the sunk cost fallacy, i.e., they started doing something in a suboptimal way, but the amount of effort that already went into it makes it undesirable to revisit early decisions. Using an LLM frees the developer and allows them to recover from wrong early decisions (or to accommodate requirement changes) quickly and with comparatively little effort.

Documentation and learning

Coding LLMs are excellent at documenting the code that they produce. This is something that developers really do not like to do, and lack of documentation is one of the main hurdles for maintenance and a common cause of regressions (i.e., when someone modifies a bit of code that is not fully understood and things that used to work break).

When you co-code you learn a lot about the codebase. Reading the LLM plans and its thinking process gives the developer (especially if not very familiar with the codebase) a much better understanding of how the different parts fit together, and it’s an extremely valuable learning tool.

Conclusions

To summarize, these are my claims, based on my quite extensive experience with co-coding:

  • AI Slop and co-coding are two very different things, and they should not be confused
  • AI Slop is a threat for FOSS development, co-coding is not
  • Co-coding done well can actually produce better code, and contribute positively to the health of a codebase. This is especially true when the contribution comes from developers who are not too familiar with the codebase
  • Co-coding frees developers from having to worry about the less interesting stuff (code styles, boilerplate, mechanical tasks), which is what in real life takes most of the time, and allows them to focus on the core business logic

And here are some of the things that I do not claim (before someone implies that I do). The space of things that I did not claim is very large, and it extends to everything that I did not explicitly - well - claim, but some of these non-claims probably need to be spelled out clearly:

  • LLM-generated code is always good
  • Coding LLMs are perfect and make no mistakes
  • Anybody can effectively co-code with an LLM, regardless of their skillset
  • Everybody should contribute code, as co-coding makes it technically possible
  • LLMs write better code and understand codebases better than any human possibly could
  • Software projects should accept any LLM-generated submission because they are oh so good
  • Coding LLMs solve every possible problem
  • …
21 Likes
2

I fully agree with what is written here. Thanks @Masterpiga for taking the time to put words on this difference between slop ai and co-coding. I have far less expertise on this than you but this matches my experiences too.

9 Likes
3

Here another developer that agrees and has the same experience. Unguided LLM use creates AI slop. But controlled use helps with writing good code and help discover bugs early in process.

5 Likes
4

I agree also, thanks for writing this.
Co-coding with Claude allowed me to fix annoying bugs in G’MIC, for the better.

6 Likes
5

This is certainly one concern. There are, however, other concerns. For instance, and these are just my initial personal thoughts, and I’m thinking in very general terms and certainly not meaning to criticise anyone here

  • where did the training data for the LLM come from? Is it basically taking people’s work and re-using it without even an acknowledgement?
  • are we always sure that the code can be issued under whatever licence we choose? What if a company uses the LLM and generates essentially the same code and slaps a copyright notice on it just before you use it to generate essentially the same code?
  • does it encourage people to engage briefly and then disappear, leaving others to maintain the code - if people have to write it themselves they possibly have a greater sense of ownership.
4 Likes
6

I think your post is well-written and I don’t disagree with any of it. If proper LLM-assisted coding is implemented only, I have no doubt that the code in darktable and other projects, whether open source or not, will still be of a very high quality. Regardless, I do still oppose the use of LLMs for other reasons:

  • The “the power to implement stuff much faster” as you said, I think is not always a good thing. The speed at which code develops automatically can shift the priorities. For example, when I make YouTube videos, I edit my audio by hand. I could save at least two hours if I used the latest AI-assisted audio editing to remove clicks and so forth. But because I edit my audio by hand I am much more careful about the length of my videos (so I make them more concise) and what I put out there.

In other words, if I have a random idea, I really try and polish it and make the script understandable before I release it, because I know that 1 minute of fluff audio means another twenty minutes of editing.

I think that there is always a point where efficiency moves past the point where the beneficial it brings is outweighted by the negatives, and I think LLMs move programming (in general) past that point, even if there are specific instances that adhere entirely to your rubric.

Rapid development is not always a good thing; so too can this be the case with darktbale. If LLMs help you maintain a larger codebase, then they also permit a larger codebase. If they allow you to more efficiently implement modules, then they also help proliferate new modules.

And even if new modules are of very high quality and are easy to maintain, does that justify them? You can say that we can still decide whether to include them, but no matter what you think: the fact that they are easier to implement will sway your rational thought, no matter how rational you think your thinking is. Furthermore:

  • LLMs take up an enormous amount of energy. Companies right now are building massive datacenters that take huge amounts of energy even in the midst of an advancing climate crisis that threatens our home - the earth. By not supporting LLMs and using as little energy as possible, we also oppose this horrendous use of energy.

  • Sure, LLMs can be used in a proper way as you describe, but they are also very likely to cause many people to be worse coders as well. If the use of LLMs is discouraged anywhere, it also sends a signal to the future coders that they also need to learn without LLMs and I think that also contributes good to the world.

5 Likes
7

To address your first and second concern

  • For large LLMs that training data is the whole of GitHub and other services like that.

  • Currently with law makers and with judges the overwhelming majority are is arguing that LLM output directly has no copyright (so is by default public domain) when it is altered and curated by a human it is copyrighted and gets its license

8

Thanks @Phemisters, the points that you raise are not strictly related to the OP but still related and pertinent.

The LLMs trained by some companies have made the news because they were using unlicensed data (not specifically coding data, but still). This is not how things are done normally, and should not happen. No big corp wants to go under the spotlight for stealing data, especially given the huge amount of data that is legally available.

Incidentally, coding is one of the few domains (theorem proving being another one) where you can easily use an LLM to generate virtually infinite training data: give an LLM the tools to build and test the code. Then have the LLM generate tasks for itself and verify the results. It can go on forever.

Why is this a problem only if the code is LLM generated? At least, with an LLM one can have a reasonable expectation that the model is not trained on copyrighted data. I would be less confident assuming that human coders would refrain from using copyrighted code, or even remember that this code snippet that came to mind is actually coming from a copyrighted repo.

The attitude of the developer is a completely orthogonal aspect. And sense of ownership is not diminished because of the collaborative nature of the work. When it comes to sense of ownership, ideas and utility matter more than the amount of boilerplate written.

For the record, I wrote color_harmonizer (with the help of Claude) because I want to use it. I have no interest in submitting code for any other purpose, other than because it adds some functionality that I or others want to use.

1 Like
9

I disagree, completely. Big corps do things unethically all the time without compunction as long as they can factor it into the cost of doing business. Anything is on the table as long as it can be predicted reasonably enough that it can be handled by their legal team and the cost is outweighed by their perceived benefit. That’s the problem with modern capitalism these days as it has reduced everything to immediate short-term cost and benefit.

I believe the problem is potentially exacerbated by LLM code. You could also say, if someone buys a gun and kills someone, why is it a problem with just the gun? Because someone could have also done it with a knife.

They might matter more, but the amount of hard work on something does factor into ownership even if the boilerplate matters less. If one does more grunt work on a project, it WILL and DOES matter psychologically to the vast majority of people. Again, I do agree that ideas matter more but to dismiss the time that people put in to learning how to write even boilerplate is too much simplification.

I for one take great pride in my work because I try to do things small and by hand. And what about two carpenters, one who merely designs the chair and allows a machine to carve it, and one who designs and carves the chair with a chisel? Do you think the former will have as much pride in the work as the latter? Unlikely. To deny that is to deny a fundamental aspect of human psychology.

3 Likes
10

That’s what devs do everyday. Looking at code to learn and copy what has been done and working. LLM makes this easier, but using standard Web search you get access to code doing what you are trying to do. You copy the code, adjust and make it work in your environment.

That’s also basically what is done by everyone every day, looking at others and learning. That’s how we have been all raised. What is so different with LLM?

EDIT: Just before anyone take me wrong, I did not say that you always do that, but from time to time we do this (learning / copying others).

3 Likes
11

The fact that corporations can do unethical stuff does not negate my statement. I happen to know first-hand.

I didn’t dismiss that. I said that it’s not the main element that contributes to sense of ownership. I can provide countless examples of projects to which I contributed tens of thousands of lines of code (before LLMs existed) and that I do not care about, because my job required me to contribute but I did not have an interest at stake.

I didn’t. “A is not required” or “A is not the most important factor” does not mean “A does not matter”.

12

Like Meta torrenting the entirety of Anna’s archive? Anthropic doing the same thing? Youtube tightening its download protections due to companies trying to scrape the entirety of youtube? Come on :smiley:

Edit: Maybe your employer is more careful with some bullshit TOS that gives them permission to use user content so it’s “legal”, but evidence for other companies says otherwise

13

I don’t want to get drawn into a long debate, so I’ll reply to a few points and then leave it there.

To me it seemed the OP was saying there was a single reason to be concerned, and my point was that there are several.

Sure, but I expect people to be aware of the licensing of any code they copy. If, as @martinus says, the LLMs were trained on the whole of github, how does it handle the different licenses used in different projects?

Anyway, I’m going to leave the conversation now.

14

Fair point. I was focusing on the “quality/value of code contributions” aspect, I left the ethical aspects out of it by design.

Added a clarification to the OP, thanks!

1 Like
15

The magnitude itself is important. LLMs allow one to do what you refer to at a much faster pace, with much less modification. Therefore, it allows for much less learning. Of course, some people may have the discipline to use the LLM somewhat properly, but most will definitely become slightly lazier to learn certain things and become less well-rounded with an LLM.

It is the same in mathematics: if I have to prove a lemma to support my main theorem, then I might look up a similar lemma in the past and basically copy it but modify it. However, if I used an LLM to copy that small part (just the lemma) and run the code through a proof verifier such as LEAN and then just use it, it might be an identical result. But the average person is much less likely to check that lemma as thoroughly as if they had to verify it themselves. It’s just human nature.

It doesn’t matter if individuals in any corp don’t “want” to be seen stealing data, even if you know this first-hand. You said “No big corp wants”, which I took to mean what the corporation “wants” in terms of its emergent behaviour, which is a much more obvious interpretation. So we are talking about different things.

That is true, you did say that. But your implied conclusion was that there would still be a sufficient sense of ownership with LLM-assisted code to obviate any problems by feeling a lack of ownership, and to that I disagree (on average). And, the fact that you can do things WITHOUT ownership (as in your example) is irrelevantt to the argument that a sense of owership will be diminished when it exists in the first place.

In other words, I was arguing that when ownership exists but an LLM is used, then the sense of owernship will be diminished. Your example of not having owership in the first place then doesn’t give any counterexample.

16

No, and this caused some controversy. But current LLMs - in most cases - are not outputting any direct code they read (actually an LLM does not contain any complete script or what so ever)

So that is way law makers and judges are gravitating towards the reasoning that LLM output itself has no copyright.

If I would read the whole of GitHub and start writing an Operating System myself then there is no copyright infringement unless I start to copy parts of Linux for example and release under non GPL license

17

If you ask me, these are blunders. Very little to gain, a lot to lose.

That is very different. If you own some data that you can use to your advantage, you don’t want your competitors to use it for free.

1 Like
18

Of course, it’s obvious that everyone is fighting for training data, what I mean is that scraping youtube is probably against google’s TOS so it counts as “stealing data”

19

IMHO, AI is just a tool (not inherently good or bad): it depends on you the use you make of it…
As a consequence, there are certain open software softwares which might decide to use it and vice versa…
Nothing is written in stone: you could also start using it and later on stop it.

For instance, you could use a knife to kill someone, to threaten someone or, more simply, to cut your food while eating…
To lessen the danger of using knives you could force anyone to use only knives made of wood (not iron); you could force anyone to only buy knives from certain manufacturers etc etc but, in the end, you are only over-complicating things, in my view…

20

@Masterpiga I am unsure whether this is a darktable topic. Perhaps consider recategorizing it for a wider audience as well as tag ai.

1 Like