Llm

Dan Luu, who is now fast becoming for me an unmissable blogger, has written-up another excellent study in using coding agents. In this post he explores how well coding agents make use of various testing and verification techniques when instructed to do so. It’s a fascinating read, highly recommended. The sort of tl;dr is: Agents are not really very good at using testing and verification techniques and when doing so their coding accuracy (on hidden tests) is not a significant upgrade on no particular instructions. In fact, most of the techniques under-performed the default of no instructions, and some increased the cost whilst providing no major benefit. ...

Mojo is now open source, via Simon Willison. I was also struck by the same quote as Simon in the Mojo road map: As Mojo matures through phase 3, we believe it will become increasingly compatible with Python code and deeply familiar to Python users, but more efficient, powerful, coherent, and safe. Mojo may or may not evolve into a full superset of Python, and it’s okay if it doesn’t. ...

The Google developers blog has published a post Why Go is an Ideal Language for AI-Assisted Software Engineering. How to say this nicely, I don’t think much of this post. First of all, I think this is a great area of active research. Just as we aren’t terribly sure about what makes a good programming language for humans, we also do not know exactly what makes a good programming language for LLMs. It does seem that it is useful to have a lot of code from a programming language in the training data for an LLM, but I think that’s a slightly different (if more pragmatic) question than what makes a fundamentally good programming language for an LLM. But note also, if we determined some kind of fundamentally good programming language (even a new one) for an LLM we could probably get over the lack of training data in several ways. ...

In a somewhat non-specific titled blog post: What’s the best programming language for coding agents?, Dan Luu takes the time to actually investigate the question of which programming languages are more efficient in their token usage. Well technically the token usage of LLM models when writing in those programming languages. Primarily he’s interested in the claims that dynamic programming languages are more token efficient because the model needn’t write any type annotations. He is skeptical and I think the skepticism is warranted: ...