Trying Claude.ai

A few days ago, I decided to test the Claude.ai chatbot, because I heard it was quite capable when it came to code.

Context

I’ve been regularly testing the capabilities of other chatbots (ChatGPT, Mistral, and Gemini) for several months now, for coding scripts in G’MIC, and unsurprisingly, overall it doesn’t work at all.

There are clearly not enough functional examples of code written in the G’MIC language (which is a bit esoteric, it must be said :slight_smile: ) to expect an LLM to have enough information about this language in its training set, and to be able to write code “fluently” in G’MIC.

The result is often a lot of hallucinations, the use of G’MIC commands that don’t exist, obvious syntax errors, etc. Nothing usable, therefore, for someone who wants to “tinker” a little in G’MIC language without knowing too much about it.

And I’m not even talking about having a functional code on the first try. Even after a lot of iterations and making suggestions to the LLMs, we usually get nowhere: the chatbots add new errors to a script while trying to fix other errors, and so on, in an infinite loop.

Trying Claude for implementing a rotating 3D cube

And there, with Claude, I must say I was quite impressed.

I asked him to generate a G’MIC script that displays a rotating 3D cube. No, it didn’t work at all the first time: there were obvious syntax errors in the first script generated. But unlike other chatbots, Claude was able to correct the errors with a few well-thought-out suggestions. And while making the corrections, he didn’t introduce (too many) new errors. After a few iterations, I actually had working source code for a 3D cube.

And I had fun helping him improve his code until he was able to generate a spinning dice, with rotation and zoom controlled by the user, and inertia in the rotation.

Here is the final G’MIC script he generated:

die_face :
  128,128,1,3,240
  rectangle. 2,2,125,125,1,0,15,15,15
  r=14
  if $1==1
    circle. 64,64,$r,1,200,200,200
  elif $1==2
    circle. 32,32,$r,1,200,200,200
    circle. 96,96,$r,1,200,200,200
  elif $1==3
    circle. 32,32,$r,1,200,200,200
    circle. 64,64,$r,1,200,200,200
    circle. 96,96,$r,1,200,200,200
  elif $1==4
    circle. 32,32,$r,1,200,200,200
    circle. 96,32,$r,1,200,200,200
    circle. 32,96,$r,1,200,200,200
    circle. 96,96,$r,1,200,200,200
  elif $1==5
  circle. 32,32,$r,1,200,200,200
  circle. 96,32,$r,1,200,200,200
  circle. 64,64,$r,1,200,200,200
    circle. 32,96,$r,1,200,200,200
    circle. 96,96,$r,1,200,200,200
  elif $1==6
    circle. 32,32,$r,1,200,200,200
    circle. 96,32,$r,1,200,200,200
    circle. 32,64,$r,1,200,200,200
    circle. 96,64,$r,1,200,200,200
    circle. 32,96,$r,1,200,200,200
    circle. 96,96,$r,1,200,200,200
  fi

go :
  512,512,1,3,0 => background
  w[background] 512,512,0,0,"Dé 3D - trackball"

  repeat 6 { die_face {1+$>} } image6cube3d[-6--1] center3d. n3d. *3d. 200 => cube

  prev_x=-1
  prev_y=-1
  prev_y2=-1
  zoom=0
  iax=0 iay=1 iaz=0
  iangle=0
  friction=0.95

  do
    mx,my,mb={*,x,y,b}

    if $mb==1
      if $prev_x>=0
        nx:=($mx-256)/256
        ny:=($my-256)/256
        ox:=($prev_x-256)/256
        oy:=($prev_y-256)/256
        nz:=sqrt(max(0,1-$nx*$nx-$ny*$ny))
        oz:=sqrt(max(0,1-$ox*$ox-$oy*$oy))
        nl:=sqrt($nx*$nx+$ny*$ny+$nz*$nz)
        ol:=sqrt($ox*$ox+$oy*$oy+$oz*$oz)
        nx:=$nx/$nl  ny:=$ny/$nl  nz:=$nz/$nl
        ox:=$ox/$ol  oy:=$oy/$ol  oz:=$oz/$ol
        dot:=min(1,max(-1,$ox*$nx+$oy*$ny+$oz*$nz))
        angle:=180/pi*acos($dot)
        ax:=$ny*$oz-$nz*$oy
        ay:=$nz*$ox-$nx*$oz
        az:=$nx*$oy-$ny*$ox
        iax=$ax iay=$ay iaz=$az
        iangle=$angle
        rotate3d[cube] $ax,$ay,$az,$angle
      fi
      prev_x=$mx
      prev_y=$my
    else
      prev_x=-1
      prev_y=-1
      if $iangle>0.1
        rotate3d[cube] $iax,$iay,$iaz,$iangle
        iangle:=$iangle*$friction
      fi
    fi

    if $mb==2
      if $prev_y2>=0
        zoom:=$zoom+($my-$prev_y2)*1.5
      fi
      prev_y2=$my
    else
      prev_y2=-1
    fi

    +object3d[background] [cube],50%,50%,$zoom,1,5,1,1
    w.
    rm.

    wait 20

  while {*}"&&"!{*,ESC}

  w[] 0

And honestly, that’s not so bad. Particularly if we compare to what other chatbots were able to generate during my previous tests!

Well, to be honest, it was mostly a test for fun. I don’t think I’ll ever need Claude to code anything in G’MIC language! But anyway, I imagine that for someone who doesn’t know G’MIC as well as I do, it could still be useful for creating functional scripts after a few iterations.

A G’MIC mode for Emacs.

So I thought to myself: if Claude is capable of creating functional things in such an exotic language, why not trying to get him to implement a G’MIC mode for my favorite text editor, Emacs? (what else? :sweat_smile: ).
It’s something I’ve wanted to do for a while, but honestly, seriously learning Lisp just to do this one particular thing didn’t seem worth the time investment.

Well, I finally succeeded with Claude.
Please note: it wasn’t a walk in the park. I had to iterate a lot over several days (I’m limited by the number of prompts because I use the free version of Claude). It was sometimes painful: it added errors when I asked for a new feature, etc. But in the end, it worked. And let’s be clear, at the end, it was much faster than if I had had to learn Lisp and the Emacs API to create an editing mode.

Here is the current version of the G’MIC mode for Emacs:

I use it since yesterday, and I admit I find it very nice: it does automatic coloring, but above all, it automatically applies the correct indentation to lines of G’MIC code, and that is priceless for me.

So, that was my experience using the Claude.ai chatbot.
My impression is the same as always with this kind of chatbot: it can be really useful if you understand what you’re doing and what you’re asking it to do. Without that, there’s a good chance you’ll end up hitting a wall!

But I was still impressed, especially by the difference in its code generation capabilities compared to its competitors, in languages that are not necessarily very popular (G’MIC and Lisp).

3 Likes

Interesting; thank you for the summary.

From what I have been reading, a major advance in Claude Code has just occurred. Now, multiple agents may be deployed as a team, distributing (and limiting) the workflow by agent, with another agent to supervise. Agents can communicate with the user (still a human at this stage) and each other to coordinate and align progress, check code, etc. With permission, Code can edit files, call APIs, make git pull requests, and do independent research. I’m no coder, but I’m glad that I’m not a new graduate in computer science.

Here is a case study from Anthropic: Building a C Compiler

We live in interesting times. I’m not sure that’s a good thing.

1 Like

I recently gain access to clause.ai but the full enterprise version at work. It’s impressive.

A reply to their post https://www.youtube.com/watch?v=6QryFk4RYaM

…now teach it that the opposing sides should sum to 7.

1 Like

Haha funny, I was expecting this remark :slight_smile:

Actually, I didn’t notice this issue at a first glance, but this is Claude who asked me (in French, this is a copy/paste of his answer):

Est-ce que l'orientation des faces est correcte, c'est-à-dire les faces opposées font bien 1+6=7, 2+5=7, 3+4=7 comme sur un vrai dé ?

translated in English as:

Is the orientation of the sides correct, i.e., do the opposite sides add up to 1+6=7, 2+5=7, 3+4=7, as on a real die?

And he proposed to try to fix this if it was not the case. So in this case, he was smarter than me :slight_smile:
I didn’t try to correct it because it was fine as it was, but it’s quite interesting that he suggested I check this point and correct it if necessary!

This is mostly over-hyped to prop up their value pre IPO.

For this C compiler example the program they wrote (by humans) for the agents to work with each-other works only for this particular compiler example. It also didn’t write the assembler and linker. They also relied on testing with GCC since their agents also didn’t write any tests, etc.

@David_Tschumperle using a model that provides more free context, before asking for it to generate you gmic code have you tried giving it a few of your personal gmic scripts to see if that improves its output?

No, I didn’t provide additionnal information. Just asked him to code a G’MIC script from scratch.
During the iterations, I just suggested him some things when I saw he was going in the wrong direction when trying to fix a bug (like telling him things like “command object3d does not consume the provided 3D mesh in the image stack.”). But that’s all.

As most of my G’MIC scripts are actually available on the Internet, I supposed it was also available for Claude.

1 Like

Of course it is over-hyped. That’s Silicon Valley for you. :wink:

But if the trend continues companies will conclude that it is cost-effective to automate portions of code development. It may be starting already, The latest iteration of ChatGPT may have been developed this way, and there is some buzz about a Japanese company Rakuten. Of course I have no idea what they do or how successful they are. Likely more hype here too.

Rakuten press release

On a smaller scale, there is also the potential for “personal software development,” where individuals use AI to help automate and streamline personal workflows. This has the potential to impact larger companies focused on productivity software. Personal code is much easier as I’m sure you know.

I’m not concerned about senior devs and people with experience and vision in the software space. But I wonder if the pipeline of junior devs will be impacted over the coming years.

1 Like

Same. At least for the near future. It was kinda the same thing when IDE’s came out, then autocomplete (or any other tool ever that made writing code less complicated).

AI (at least in it’s current state) is a tool you still need to know how to use it. You still need to have knowledge about what you are trying to build with it. It helps a ton if you know to use it and makes tedious tasks simpler. Let it do some tasks while you do something different and then quickly check the output and fix some things up.

It fucks up quite often, focuses on the wrong things etc. and if you don’t know how something should look/work you won’t see if it doesn’t do that.

a little example:
I had a friend (no dev experience at all) tell me how he was able to build that awesome service by just telling an AI to do it. It even did authentication… but… only hid things, there wasn’t actually any check if the user is allowed to do what he wants to do… there where several other things like that. you can fix that, even the AI can but it doesn’t double check if nobody says so.

i think the focus will shift. They’ll use AI early and integrate it more in their flow. You still need to teach them the “vision” and “experience” part. And of course they need to understand what they as well as the AI are doing. i guess the biggest impact will be: there won’t be needed as much junior devs in parallel.

1 Like