Yes but I mentioned that in my post. The current blackboxes can be understood by a team of programmers you can hire from the street. If we followed this path that all programming will be done by AI, there will be nobody left who will understand them but those AIs controlled by trillion dollar corporations run by shady people.
1 Like
These two statements are not necessarily consequential.
Yes, the natural evolution is that code will be bypassed directly. You only need programming languages if you need humans in the loop. Machine code is way more efficient and performant, so eventually models designed to work without humans in the loop will start generating that directly.
But that does not mean that the code will not be âunderstoodâ. The same algorithms that produce the code can be used to make it understandable to humans, if need be. Also, for all intent and purposes software products of corporations is already a black box for the average user, as they donât let you peek inside their code base. So, what does it matter if you can or cannot understand the actual code, when you cannot read it already?
As for the controlling part, as always it will be a struggle of powers. It used to be the case that only a few governments and corporations could afford to run a mainframe. Powerful entities have always add an edge over individual, and the consolidation of power and wealth into the hands of the few is an inevitable side effect of capitalism and corporativism. This is just another example of the same happening over and over, I am afraid.
Iâm not 100% convinced about that. Source code is an abstract representation thatâs easier to change, refactor, and to translate to machine code on various platforms (which, in the cloud, can be a real advantage â one day, you are on Intel, the other, on ARM, or on whatever in the future). Symbols carry semantic meaning, which is easier to translate to the specs given by the end user. I think compilers will remain much more efficient at turning that higher-level abstraction (which may not be in any of the languages in existence today) to binary than LLMs. Even if you remain on the same hardware, you may benefit from new optimisation at the fraction of the cost if you simply recompile your software than using an LLM to say âthis code was created by an earlier LLM version; see if you can make it more performantâ.
But thatâs tangential to the main worry: a few corporations monopolsing on the ability to turn specs into code.
I also share this worry, not only because of my own job, but because of handing key elements of our infrastructure to something we have no control over. Imagine if you depended on electricity or water without control over it. Itâs no wonder that in most places such basic services and infrastructure are non-profit state monopolies to some extent, are controlled strictly by laws, and companies are obliged to maintain open access for everyone.
3 Likes
Yes, these are all good points. Maybe a better way of putting it is that LLMs will develop their own internal representation that combines the best of both worlds. My point being that whatever they come up with does not need to be human readable IF humans in the loop are not a factor.
I guess it is a matter of accountability, really. We currently need men in the loop because (1) the tech is just not good enough for these things to be completely autonomous, and (2) they are not legal entities, and the legal framework around them is still fuzzy, so everything that they do needs to be signed off by a human who takes responsibility for their actions.
But a point may be reached (I am not completely sold on this, yet) where specialized models become good enough that human visibility is only required at some well designed intersection points, and whole subsystems can run in complete autonomy. And if a legal framework exists so that they (or the companies that provide them) can be held accountable in case of errors or failures.
Which is what Doctorow calls the Reverse Centaur (someone whoâs only there to take the blame if something goes wrong; who is not helped by AI, but rather is serving it).
Iâm quite sure there will be companies that work like current software companies, creating custom products for their customers, or providing outsourcing/staff augmentation, where the main difference to today will be that much more work will be performed by machines; and probably there will be companies providing the compute cloud, LLMs and all. Even with traditional software companies, there are contracts that deal with who is responsible for what. I think that as long as gross negligence or outright malice can be proven, such companies are (and should be) protected from consequences of shipping buggy code (bugs being unavoidable). There will be the market, based on a price / performance ratio, based on reliability, speed, cost (as usual, choose any 2 
).
2 Likes
The population collectively has far more power. I think thatâs why some systems like to push rugged individualism so hard. It makes the people easier to control.
Radicalization has the same effect. Divide et impera. It works.
My guess is that LLM based coding services will be enshittified long before that happens. The providers know your full usage in detail. Every single byte. Once a company fired most of the good engineers who can actually code, the temptation to tweak pricing plans to extract all almost value added will be too great.
From the perspective of the providers, the black box is a feature. Make is as black as it gets; the harder the code becomes for humans to refactor, the better.
Yes, I know that you can use LLMs to write clean code with careful planning and iteration. But not every user will be so careful or put in the effort when they can cut corners. I routinely get PRs on FOSS repos that were produced by a single-sentence prompt that adds a feature at the cost of a disproportionate technical debt. And then the âauthorsâ get offended when I donât merge.
3 Likes
A lot of how this plays out will be determined by how easy it is to replicate these models and by the availability of compute. So far some companies are ahead, but Iâve not seen a huge mote protecting them.
1 Like
I just skimmed this but it something like this legal⌠it may also be a puff pieceâŚ
3 Likes
IANAL, but AFACT Adobeâs blanket license does not prohibit this, so why not.
LLMs are really good at this. Saw a post the other day where a guy used agents on loop to figure out ways to unlock the bootloader on old amazon fire tablets. A human could do this, sure, but it would cost hundreds of reverse engineering hours, where an LLM works tirelessly at it till it finds the way in. Honestly not sure how to feel about this
2 Likes
OK now do Nikon embedded lens correction.
8 Likes
If we used Claude to decipher the lens correction, will you use it?
1 Like
Proof would be in the pudding, so to speak. Nikon NEFs contain in-camera corrected JPEGs, so the corrected output of whatever algo Claude came up with could be compared against that.
I guess Iâm going to do my exiftool-ingest-to-rawproc sooner than laterâŚ
Yes. Its one of the original intents of the LLM: launder that IP! Its transformative!!
3 Likes
I find using LLMs perfectly adequate to automate tedium. Just make sure you keep creativity a human activity.
There is a argument that LLMs âstrip awayâ the accidental complexity of software development, and âexposeâ the inherent complexity. I donât fully buy that, as they tend to add a lot of accidental complexity to the code, even where they hide it from the programmer. Nevertheless, the ânext step on the ladder of abstractionâ has some merit; I do tend to think at a higher level when coding with coding agents. But I guess the jury is still out on whether that actually results in better software.
2 Likes
I agree with that, with some remarks.
- it depends on language and code based. In PHP and Python this happens a lot
- In javascript frontends the LLM suggest to add another third part package
- with the Fable moddel van Claude I have seen this behaviour much less
- If I include in my initial prompts to KISS and that I donât want any verbosity the outcome is much better
3 Likes
I now also regularly add âKISS, YAGNI, surgical changes, donât overengineer, donât chase highly unlikely theoretical possibilitiesâ.
5 Likes
I use LLMs to write code in languages I use too rarely to be proficient in them. I know precisely what I want, but I would have to wade through hundreds of pages of docs to figure out how to do it.
For example, Bash is useful (simple, fast to start, ubiquitous), but it has a ton of quirks once I go beyond simple âassign variable, use variable, occasional loopâ semantics.
I can then ask to have the output explained to me if I have questions. My use case is like a search engine 2.0: I could find everything out myself, it would just take lots of time.
I think LLMs are an amazing technology because of how they handle input: they operate (or emulate operating, depending on how your think about it) on a layer that is one or two steps more semantic than literal text, so I no longer have to find the right synonym as I would in a search query.
I still have to know what I want, as precisely as possible. The more open ended the query, the worse the output.
I still donât use LLMs to author code I really care about in the long run. They do ok work, but again, I have to ask the right questions. For me, those questions come from actually working with the code.
But LLMs are amazing for the tedious parts: setting up tooling, CI, dependency chains, whatever. This is again about miniutiae of domain-specific languages, usually declarative in a .yaml or .toml file, I would have to read tons of ever-changing documentation to be up to date with this.
Oh and donât forget algebra. They very, very good at tedious algebra involving (for me) obscure identities I would have to look up.
4 Likes