• Getting FastAPI ready for free-threaded Python

• Introducing Astral’s ty into FastAPI’s lint checker

• Analysing loss curves to triage nanochat user contributions

• Welcoming nanochat’s little brother ‘autoresearch’

• Publishing skills files to help LLMs write the best code for frameworks you maintain

Free-threaded Python

Since v3.13, CPython supports building Python in a “free threaded” model, which means thath the GIL (global interpreter lock) is disabled. This then paves the way for more efficient execution of Python programs by utilizing all available CPU cores in parallel threads. The Python versions that support free-threading are denoted with a “t” at the end, so e.g. 3.13t and 3.14t.

There’s been quite a lot of work done in the ecosystem to start supporting these free-threaded Python versions, which isn’t always trivial. Nathan Goldbaum and other colleagues at Quansight Labs have been working hard to get core packages such as NumPy and scipy to fully support a nogil build of Python. About half a year ago, we started looking into this for FastAPI as well (cf. PR 13946), but quickly ran into dependencies that weren’t compatible yet: httptools, pydantic, bcrypt and orjson to name a few. The beauty of open-source: I pinged Nathan (out of the blue) to help assess the situation for FastAPI and not only did he show up in the PR thread with some pointers and advice, he also went on to help httptools fix their CI and get ready for free-threaded support.

While the ecosystem was working on this, we put it on the back burner for FastAPI for a few months, until I took it back up this month in PR 15149. This time even without asking, Nathan showed up with helpful advice and helped us get fastar ready for nogil <3.

The final steps then to get FastAPI ready for Python 3.14t, was to remove orjson and ujson as standard test dependencies, which was possible since FastAPI’s BDFL Sebastián Ramírez (commonly known as Tiangolo) had deprecated ORJSONResponse and UJSONResponse in favour of using JSON responses that can be serialized with Pydantic (cf. PR 14962).

I’m pretty happy with how this has worked out, and it was super nice to interact with the folks from Quansight Labs. With all of the LLM-slop PRs, reviews and conversations that we’re dealing with on GitHub lately, it’s great to interact with actual people who are just super helpful and genuinely care about the ecosystem. Thanks a bunch, Nathan & team! The 3.14t PR isn’t merged yet - it’s waiting for a final review by Sebastián - but you can expect free-threaded support in FastAPI sometime soon!

ty is for types

Not only has Astral taken the Python world by storm with their fast package manager uv, and their linter ruff, but now they’ve also published a new type checker called ty. Like its siblings, ofcourse, it’s implemented in Rust.

Over at FastAPI, we think types are quite a big deal, and up until now we’ve been using mypy for our precommit linting needs. Mypy is pretty good and stable, but still suffers from some false positives (notifications that aren’t errors) and false negatives (errors that aren’t reported). Running a second tool in parallel might not be a bad idea!

When we posted about this on LinkedIn, there were some comments about ty not being ready for production yet. But it’s important to realize that ty isn’t being run in production: it’s a precommit check for a production-ready codebase, and all of its warnings and errors are manually reviewed by us, the maintainers of said codebase, before pushing any code to production. It’s really just a tool to help us find potential issues in the code base, just like mypy is.

So, I set out to add ty to the precommit steps in all of fastapi’s OS libraries: not only FastAPI itself, but also Typer, Asyncer, SQLModel, annotated-doc, markdown-include-variantes, etc etc. You get the idea. I started with Typer, as it’s my favourite OS repo to work on and a good guinea pig as FastAPI’s little sibling. Here, I noticed that ty sometimes doesn’t infer the correct type, and requires additional cast and/or assert statements to help it see the correct type. Alternatively, you can just add an ignore statement to this line that will suppress ty’s warning/error. We opted for the latter, though this requires some care: don’t just put “type: ignore” when you’re running mypy in parallel. Because in those cases where mypy doesn’t see an issue, it will in fact complain that the ignore statement is unnecessary. You can put “ty: ignore” instead (mind the two missing letters), so that it become a ty-specific ignore statement and won’t influence mypy in any way.

There were a few spots in Typer’s code base where we were using a try-catch instead of an explicit if-else to check for existence of certain attributes. ty rightfully called this out, and it’s one of those small improvements we were able to make by integrating ty into the precommit check.

In Asyncer, there was a class definition that mypy had issues with, and we had previously decided to annotated with “type: ignore” to suppress mypy’s warning. Now ty comes along, and decides there is actually no issue with this line. For ty versions 0.0.24 and before, this would mean that you’d have to add a second ignore statement and the line becomes something like this:

class X(Y): # type: ignore # ty: ignore[unused-ignore-comment]

Which is not super satisfying. However from ty 0.0.25 onwards, ty can actually parse the error codes within a “type:ignore” statement, something it couldn’t before (cf. PR 24096). This means that if you put the exact mypy code within brackets, ty will parse it, decide it’s not one of its known codes, and ignore it. So when updating ty to 0.0.25 or above, the ugly line quoted above becomes something like

class X(Y): # type: ignore[valid-type]

Which is much better. In general, if you prefer to run ty in parallel with any other type checker, I’d definitely recommend using 0.0.25 and above.

Continuing with our other repos, many actually required zero code updates after adding ty to the precommit step, which is nice (e.g. fastapi-new, fastapi-cli, annotated-doc, markdown-include-variants). However, when running ty on FastAPI’s code base for the first time, no less than 150 issues popped up. Some of these were actually pretty useful, for instance pointing to Pydantic v2 deprecations that FastAPI should have addressed earlier but didn’t because our CI tests for the lower ranges of our dependencies were broken (this is the type of rabbit hole you often find yourself in when open-source-maintaining). So we fixed the CI first (with some helpful digging by Yurii Motov), and then addressed the deprecations properly. This is a clear case where ty helped us identify issues that hadn’t been reported by mypy. But the same is also true in reverse: mypy sometimes finds issues that ty doesn’t. So for now we’re happy running both in parallel. And yes - thanks for asking - I did manage to resolve all 150 issues, with help from Yurii who gave super detailed and constructive feedback on the PR. Team work makes the dream work ;-)

Eventually we also decided to mark ty warnings as errors that would fail the CI. This is easily achieved by setting this in your pyproject.toml:

Loss curves going down, down, down

As you may remember from previous posts, I’ve been involved in the maintenance of Andrej Karpathy’s nanochat repo, which provides all code for training and fine-tuning an LLM. In February, Andrej had set up a leaderboard to help incentivize improvements to the pretraining stage of nanochat. Comparing performance using the CORE score, the aim is to get to GPT2-level capabilities in as little as time as possible, measured on a 8xH100. Ideally, also val_bpb should go down, which is a sort of normalized, vocab-size-invariant loss score.

So one thing I’ve been doing to help assess user contributions that aim to improve nanochat’s pretraining code, is to do a quick sanity check on a (single) A100 instance I have access to. Using W&B, which is awesome for creating graphs you can spend all day looking at, I can then easily plot & compare different PR contributions:

These are all d14 runs on an A100, and in terms of val_bpb, master still comes out on top at the end.

There’s many other interesting plots to look at, and all of these graphs are created automatically for you when you run the nanochat training scripts and are logged into W&B. For instance, it’s also really useful to check memory usage or disk utilization stats:

These plots are also useful to monitor your training run after having made code adjustements: you’ll typically be able to spot it relatively quickly if something is going fundamentally wrong with the loss profile. Then you don’t have to wait until the training run ends to go back in and correct things, saving you some valuable cloud credits…

Autoresearch: fine-tuning LLMs with LLMs

Meanwhile, Andrej himself was also experimenting with ways to further improve the pretraining step in nanochat, and push the limits of the leaderboard further. Eventually he decided to outsource his LLM engineering research to … an LLM. I’ve written a bit more about this, including an experiment with multiple agents riffing off eachother’s contributions in this post about autoresearch / autolab:

Publishing skills files for your repo

Who better to explain to agents how to write code for a certain framework, than the maintainers of said framework? This is actually a pretty cool exercise to do, a bit like writing documentation but for LLMs, not for humans. Sebastián had pioneered this idea in FastAPI, and I’ve written a similar one for Typer. What I find particularly promising with this approach is that you can tell LLMs how to navigate recent deprecations or breaking changes in your framework. You can show it best practices and even explain why, so in turn the LLM can review or write code for its human and explain the thought processes behind the change. Would love to get some feedback from the community whether it’s been helpful to them and how!

-Sofie

And now, there’s autolab! Now, agents that have been able to improve on the val_bpb score (validation bits per byte - lower is better), can contribute their “patch” to a set of 8 official workers, run by Karpathy, who will go and validate the contribution. When it’s a genuine improvement over the current record, this patch contribution becomes the new master. All other agents continuously check whether there’s a new master, and update internally when that’s the case, so they can keep working off of eachother’s improvements. And then you get something like this:

Each dot on this graph represents a contributed patch by an autonomous agent, a patch that was verified by the official workers and resulted in a new master with a better val_bpb score. Basically it gets more difficult to establish a new record as time goes on, and it’ll be interesting to see whether the agents will change strategy over time as well.

Agent OxyKodit joins the swarm

I couldn’t help myself and decided to spin up an H100, which is currently required to join the swarm. It’s actually super simple: after installing & authenticating Claude Code, you just paste the full agent instructions from autoresearchhub.com (currently offline as this was just a quick experiment), and the agent starts churning. It installs everything, contacts the swarm online to get the latest state of master, then decides on some experiment to run. After making some edits to train.py, it trains a small GPT language model for a fixed 5-minute wall clock budget. If it ends up beating master’s val_bpb, it will submit this patch and hope to end up on the leaderboard.

So for instance, this was early on in my run:

It decided that the experiment it did (using SwiGLU instead of squared ReLU) wasn’t an improvement. Then it also saw that master was updated in the meantime, and made sure to fetch the latest.

Then at some point it actually did find an improvement, at least when run locally, and it submitted the patch with a pretty verbose explanation. On autolab’s UI, you could see it being in the queue, waiting for validation:

As another agent (“scientist”) was running at the same time, you can see that there are multiple submissions already queued or assigned for verification, which means that your submission may still fail to obtain SOTA (state-of-the-art) if another agent beat you to it with a better score. That’s unfortunately what happened to my first submission, but that’s OK - our agent relentlessly continues (as long as you pay for tokens).

In its first 2 hours of existence, Agent OxyKodit was able to make a few more patch submissions when finding local improvements, but hasn’t yet been able to make it to the “Path to SOTA” quite yet. Still, it’s nice to see its efforts reflected on the leaderboard:

And in the background, on the console of this H100, the search continues …

-Sofie

After more than 10 years of contributing open-source (OS) code on Github, I’d love to share some tips & tricks from my own personal experience - both as a contributor and as a reviewer.

What do effective Pull Requests look like?

Let’s first dive into the contributor’s perspective. You’ve spent days, maybe weeks, to implement a new feature or hunt a bug, and you’re finally close to wrapping this up. You’re proud of the work and effort you put in (and rightfully so!), and you’d like your contribution to be merged into the ‘main’ branch as soon as possible. You wrap things up by quickly creating a PR contributing all the code from your branch, write one or two lines in the PR message body, and call it a day. You’re looking forward to all the positive feedback on your hard work in the morning!

How to showcase the value of your work clearly?

Unfortunately, one of these things might happen instead: your PR sits around and doesn’t get a review. Or it does get a review, and it’s endless questions and comments. You’re getting slightly annoyed or even disappointed. Can’t they see the clear value of this work and just merge it in?

Turn this around and ask yourself: what could I have done to make the value of my work more clear? To convince the reviewers of the correctness?

Here’s a helpful shortlist to give your PR a flying start the next time around:

• Ensure your code is formatted according to the project’s guidelines. This instills trust that you’re paying attention to details.

• If it’s a bug fix: contribute a clear, concise unit test that fails on ‘main’ but succeeds on your branch.

• If it’s a new feature: provide sample code that demonstrates the functionality. This will incite enthousiasm and make the value of your work crystal clear and tangible.

• Update all documentation relevant to the changed functionality. You might be inclined to do this “after review” or even “after approval”, but in most cases that’s a mistake. Writing the documentation will help you think about the general API design and whether it’s consistent with your current code & functionality. Furthermore, it will help reviewers better understand the implications of your PR. Eventually, this will help speed up the approval process.

• In the PR description, write a clear summary of the proposed changes and indicate any potential issues with backwards-compatibility.

• Make it clear whether or not you’ve used AI to generate any of the code, whether you’ve reviewed it all, and which parts you may not understand yourself. Honesty builds trust!

Focus on one unit of work per PR

Think about what constitutes one unit of work and address only one problem/feature in one PR. Did you run into a vaguely related bug X while implementing a new feature Y? Submit the bug fix X to ‘main’ with a clear unit test and submit the PR for feature Y independently. Having more ‘atomic’ PRs has many advantages:

• Reviewers can focus on one thing at a time

• Small fixes get merged & released more quickly: a bug fix can go into the next patch release, while a major feature/refactor may have to wait for the next minor or major release

• If you ever need to revert a feature PR because of unexpected issues, you don’t end up accidentally reverting a good fix as well

• You might have a few merge conflicts, but they’ll be smaller & easier to deal with if you keep the functionality of each PR minimal

Be your own PR’s first reviewer

Once you’ve done all the above: be your own PR’s first reviewer. What does this mean exactly?

Submit the PR as draft, step out of “developer mode” and into “review mode”, and inspect your own code changes with a healthy dose of criticism & sceptism. This is especially critical if you’ve used AI to (help) write the code.

Is the PR description up-to-date? Does it provide enough context for someone else to follow your reasoning? Are all the changes in fact related to one core functionality? Are there any debug comments or print statements still lingering? Did you write unit tests for everything? Does the CI run succesfully, are there no merge conflicts? Does the example code run correctly? What if you change the input slightly, do you get appropriate error messages? Can you make the code more robust & user-friendly? Is the code easy to read and maintain?

And one of the most important tips: write review comments (on your own PR!) for things you're uncertain about, important design choices or non-obvious implementations you require feedback on. This will help reviewers zoom in on the important stuff. It might be tempting to think "hey maybe they won't notice this weird approach I took at L342", but you're not doing anyone a favour: a reviewer might flag it and become less confident in the remainder of the PR, or they might not flag it, which could be worse: the code gets merged into 'main' and you'll be explaining this odd code block when a bug report hits your issue tracker 2 months from now. Overall, and somewhat counter intuitively, admitting that you're uncertain about something actually helps build confidence in you and your contributions overall. It sets the stage for open communication, positive feedback and constructive collaboration.

Any tips for the reviewer?

Switching perspective - let’s now consider how a reviewer can be as effective and helpful as possible. Do consider that not all review(er)s are equal: some will just write a quick nit-pick comment, others may spend 2 hours going through all the code. Either way: explain clearly what you’ve done and where you’re at. Useful context to give, for instance, could be:

• “Just a quick drive-by comment, didn’t review everything”

• “I looked at the PR in detail, and only had a few minor comments”

• “Started reviewing, but I need you to address point X before I can continue”

Giving this context beyond just the bare comments, helps other reviewers as well as the original contributor to better understand the state of the PR, what is needed to push it along, and who should do what. And while you're at it, say something nice and positive as well. Even when you're unhappy with the implementation but understand the motivation for the feature, you can say so: "this would be a great feature to have". At least, acknowledge the work and effort the contributor has put in, as a bunch of critical comments may otherwise come across as very demotivating. Of course, reviewing is also hard work, requires context switches, and you're only doing this to help and/or that's just your job. But nevertheless, I've found it pleasant to work with collaborators who uphold a positive and constructive attitude, even if they have plenty of comments or criticism.

The same principle about open and clear communication for the submitter is true for the reviewer as well: if you're uncertain about certain parts of the code, simply say so. Perhaps find someone else to do an additional review, or decide to lean on the contributor's expertise. In general, make it clear what the next steps should be for this PR to move along. Avoid getting trapped in the "bystander effect", where e.g. 3 people are requested for review and they all wait for someone else to do it. Avoid relying on implicit expectations such as "I thought you were still working on this" or "I thought X was still reviewing" - clarify if need be and make expectations explicit.

PRs do not necessarily follow a linear path

And finally, understand that work on PRs, like any other dev work, is not always a linear process. Sometimes you’ll need to go back and revisit earlier decisions, or ultimately decide on a different design that requires some refactoring. That’s not a problem in itself, as long as you’re converging to a final solution and not endlessly debating / refactoring. It’s still better to do one more round of changes before merging, than having to do it in subsequent releases and impacting users. Realise that you’re all working towards the same goal of improving the code and functionality so that it’s ultimately as useful as possible for the users of the software. And in the process, view every PR as an opportunity to learn from other developers and further improve your coding skills, whether you’re the original contributor or the reviewer.

Happy OSS’ing!

-Sofie

• Writing reference docs for Typer (finally!)

• Guiding users to use uv instead of poetry

• Reviewing nanochat progress

Typer reference docs

Sebastián Ramírez, or Tiangolo as he’s known on GitHub, is not only famous for the awesome developer-friendly open-source tools he creates, but also for the extremely extensive and beginner-friendly documentation that he always writes. As an example, the docs for FastAPI give you an introduction on Python types and explain concurrency and environment variables, before kicking off with a “First Steps” introduction to FastAPI. The tutorial continues to build complexity and provides detailed code examples along the way.

For Typer, a very similar philosophy was taken when writing the documentation, and users are generally very happy with the high quality of available information. However at some point, when you’re a seasoned developer already familiar with FastAPI and/or Typer, you don’t want to go scrolling through pages & pages of verbose text to find a particular setting or default value.

Enter reference docs: a concise overview of the relevant methods, parameters & defaults, and how they can be used. While we already had one for FastAPI, it took me until now to write this up for Typer, too. PR 1504 shows how we approach this: rather than writing these reference docs from scratch, we basically add Doc() annotations within Annotated[] definitions for the parameters of important functions - right in the code itself.

So for instance this

would now become

And that will render on the reference docs like so:

(kudos to Yurii Motov for improving the layout in PR 1630 !)

Anyway while we hope this will be useful for all Typer developers out there, it was actually also fun to write, because it kind of made me stop and think about every single functionality that Typer offers.

uv instead of poetry

uv has pretty much taken the land of Python by storm. Built by Astral, the same team behind ruff, uv is written in Rust and meant to be an extremely fast Python package and project manager, which it is. Pretty much everyone I know has switched to using uv instead of pip, but for the “Building a package” tutorial in Typer we were still referencing poetry. Rectified that since PR 1474 !

It’s the kind of change where you carefully have to read all the documentation surrounding the edits, to see whether there are any spurious remarks still referencing the old documentation. For instance, the package structure will look slightly different on file with uv instead of with poetry, so all of that needs to be updated too. What I think would be boring work to many, is something I actually quite enjoy doing, being meticulous and ensuring consistency across documentation, to ensure there’s as little friction for first-time users as possible. And ofcourse, ask your favourite LLM to double check everything to make sure you haven’t forgotten any references!

nanochat reviewing

In December, things were relatively quiet on the nanochat front, the open-source repo by Andrej Karpathy that provides you all the code for training and fine-tuning an LLM, even providing scripts to chat with it on a web or CLI interface. But in January a few major refactors happened:

• The inline rustbpe was deleted and moved to a separate repo

• The configurator was nuked in favor of argparse

• The base_train script was improved both in code layout as well as hyper-parameter tuned

• Muon optimizer changes resulting in a bump of val_bpb

• Integration of Flash Attention 3

• …

While I had nothing to do with any of these improvements, as “repo czar” I’ve tried to be helpful by updating older PRs that had accumulated merge conflicts, closing PRs that were no longer relevant, and making sure that everything remained consistent (e.g. PR 407, PR 416, PR 438).

It’s really a lot of fun to have a “front row seat” watching this repo evolve and get better every day!

-Sofie