I Built a Coding Gauntlet to Trip Up 7 Free AI Chatbots. None Fell.

I wanted to know something simple. In 2026, is there any real difference between the free versions of the big AI chatbots when it comes to writing code, or have they all quietly converged into the same competent blur? The only honest way to find out was to test them myself, so that is what I did.

The trick was to invent a problem none of them could have seen before. A memorised solution proves nothing. A brand new spec with a few nasty edge cases buried in it is a real test of whether a model reads carefully and thinks, rather than pattern matching against its training data.

The first challenge

Round one looked trivial. Write a Python function that cleans and reformats a string. The difficulty was hidden in the ordering of the rules and a handful of edge cases that punish anyone who skims.

Write a Python function normalize(code: str) -> str that transforms a
string according to these rules, applied in this exact order:

1. Keep only ASCII letters (a-z, A-Z) and ASCII digits (0-9). Discard
   spaces, punctuation, accented letters, and anything non-ASCII.
2. Uppercase the remaining letters.
3. If nothing remains, return "EMPTY" and stop.
4. Keep only the first 12 characters.
5. Pad on the right with 'X' until the length is a multiple of 4.
6. Insert a hyphen between each group of 4 characters.

The traps are the accented characters (anyone reaching for Python’s isalnum without an ASCII guard keeps é and ö and fails), the empty case, padding the final block to a multiple of four, and getting the truncate then pad then group order right. I built ten test cases hitting every one of those.

Round 1: everyone aced it

All seven scored ten out of ten. Every single model correctly guarded the ASCII trap. Every one handled the empty string, the padding, the ordering. Not one of them slipped. I sat there slightly deflated, because I had built this test to produce a loser and it refused to produce one.

result = ''.join(c for c in code
    if ('a'<=c<='z') or ('A'<=c<='Z') or ('0'<=c<='9')).upper()

So I made it nastier

If the easy trap could not separate them, I would use a genuinely counterintuitive one. Round two ranks the characters in a string by frequency, and breaks ties in a direction almost nobody expects.

Write a Python function dedup_ranked(s: str) -> str:

1. Remove every space character.
2. If nothing remains, return "EMPTY".
3. Treat characters case-sensitively ('A' and 'a' are different).
4. Rank the distinct characters by frequency, most frequent first.
5. Break ties by CODE POINT, HIGHEST first — 'z' before 'a', '9' before
   '0', lowercase before uppercase.
6. Return the ranked characters joined into one string.

Rule five is the whole game. The obvious tie-break is alphabetical, ascending. The lazy one is whatever order a Counter happens to produce. The spec asks for neither. To get it right, a model has to actually read that rule and implement a descending code point sort on purpose. I was certain this would finally split the field.

Round 2: another clean sweep

Ten out of ten. All seven. Again. Every model read rule five correctly and implemented the descending tie-break, either by sorting with reverse turned on or by negating both sort keys. Nobody defaulted to alphabetical. Nobody forgot the case-sensitivity. I genuinely could not break them.

What this actually means

Two invented specs, seven free chatbots, twenty edge cases each, and a perfect score across the board. The headline is uncomfortable if you still think of free AI as the budget option. For small, well specified coding tasks, correctness is essentially a solved problem. The question of which free model gets the answer right is close to meaningless now, because they all do.

That does not mean they are interchangeable. It means the thing that separates them has moved. When every model passes, the difference is no longer in whether the code works, it is in how the code is written. That turned out to be a whole article of its own.

They all got the right answer. So I read every line of code they wrote, and asked the only question left, whose code would I actually trust. For the researched view of these assistants, see ChatGPT vs Gemini vs Claude.