Benchmarks are everywhere, but most of them feel disconnected from the work developers actually do. MMLU scores and coding contest pass rates are fine for headlines, but they don't tell you much about what it's like to sit down with a model and get something done.
So I decided to run a different kind of test: Give the models the same files to start with, and exactly the same prompt, and compare the results. I have been building a personal library of out-of-print e-books, from the start of the 20th century, and I decided to use Kiro to write a tool for converting from epub format to markdown. Markdown is becoming a standard format for use with source code, and is commonly used in Github for documenting projects, Jupyter and Sagemaker notebooks, Kiro AI-DLC documents, and, because of that, it's a de facto standard format for AI tools.
The epub format is quite straightforward, it's a zip file, generally containing text in html format, but with a specific folder layout and formatting. Commercial epubs are encrypted, but public domain documents can be examined just by renaming them to a .zip file and opening them in your favorite operating system.
Markdown, on the other hand, is a text file that contains extra information about how the text should be displayed in a viewer. It has equivalent text formatting to HTML, but does not have sophisticated visual formatting available in HTML with stylesheets and backgrounds. Here's a simple example of an HTML file and a Markdown with the same content:
<html>
<body>
<h1>This is a title</h1>
<h2>This is a subtitle</h2>
<p>This is a paragraph of text</p>
<p>This is another paragraph with <i>italics</i> and <b>bold</b></p>
</body>
</html>
In markdown, this would look something like this:
# This is a title
## This is a subtitle
This is a paragraph of text
This is another paragraph with *italics* and **bold**
The task of converting from epub to markdown is surprisingly demanding: unzip the epub, parse the book manifest and spine (which tells us the order of the book contents), extract metadata into Markdown frontmatter, convert all the actual contents in a way that respects the original typography, removing HTML noise.
It's the kind of task that separates models cleanly. It requires multi-step tool use, XML and HTML parsing, format inference, and sustained consistency across dozens of pages, all on source material the model has never seen before and can't look up.
I decided to compare Qwen 3 Coder Next, Claude Sonnet 4.6 and Claude Sonnet 5.5 models, all running under Kiro 1.4.86, and then let Kiro run in "auto" mode.
The initial prompt for all of them was:
Let's start with this: I want a python script that takes an epub and extracts all the manifest, converting it to a single markdown file. Each source html file in the epub should start with a "---" followed by the name of the file, without the extension, on the next paragraph.
It should be smart enough to interpret any CSS and add italics, bold, superscript and subscript natively to the markdown without using HTML tags.
We will add more functionality and evolve this with other features in future.
I deliberately inserted an error in the prompt. I should have said spine instead of manifest, as spine defines the order that the files should be chained in the output markdown, whereas the manifest is just the list of files that make up the epub. I did this to see if any of the scripts would pick up on that and put them in the correct order. All models did.
The "---" is a horizontal rule in markdown.
Here is the meat of it:
This took 9m10s and used 5.42 credits (Kiro measures effort with "credits" instead of "tokens"). It generated a 650 line script and used up 29% of the context window.
I ran these commands on the test files:
> python .\epub2md.py ..\test_data\test1.epub
Successfully created 'test1.md' with 21 chapter(s).
> python .\epub2md.py ..\test_data\test2.epub
Successfully created 'test2.md' with 14 chapter(s).
Looking at the generated test files, it didn't correctly break lines at the end of each "chapter" inside the epub. It also didn't generate correct markdown across separate lines.
Let's look at what that means. Markdown should have a blank line between paragraphs, heading, and so on. If I put text on the following line (no blank line), it inserts a break inside the paragraph, like the <br /> html tag. If I have two lines inside a paragraph that are italic, it should add italic tags to each line, as the tags don't flow across line breaks:
Fixing the line breaks I used this prompt:
In the markdown source, there should be a blank line at the end of a epub file, before the --- and a blank line between the --- and the file name.
This used up another 5m23s and 3.86 credits.
A follow up prompt to correct the italics issue was
If a paragraph has a line break in it, in correct markdown, the italic and bold tags should be closed, and opened again at the start of the next.
Another 13m58s and 9.24 credits. The context window ended up at 54%.
The same prompt took 5m39s to generate a 542 line script, and used 8.07 credits. The context window for Claude is larger than Qwen, so it only filled up 8% of it.
Here's the output of my tests:
> python .\epub2md.py ..\test_data\test1.epub
Converting '..\\test_data\\test1.epub' → 'test1.md' …
Done – 3,242 lines, 34,083 words written to 'test1.md'
> python .\epub2md.py ..\test_data\test2.epub
Converting '..\\test_data\\test2.epub' → 'test2.md' …
Done – 3,214 lines, 41,770 words written to 'test2.md'
Note that it gave me more verbose output. The markdown had the same problem with italic tags, which I fixed using the same prompt as with Qwen, for another 1m33s, and 4.2 credits. Context window ended up at 9%.
This model took 8m39s and 12.62 credits. It generated a whopping 1016 line script, and filled up 12% of the context window. Here's the test output:
> python .\epub2md.py ..\test_data\test1.epub Wrote 21 document(s) to 'test1.md'.
> python .\epub2md.py ..\test_data\test2.epub Wrote 14 document(s) to 'test2.md'.
Looking at the generated output, it got the italics correct, but added a backslash \ to the end of each line. I asked it to remove this with:
In markdown, you don't need the backslash in order to continue on the next line, just a newline character.
And used another 1.65 credits and 54s. Context window ended up at 13%.
As a last test, I ran Kiro in auto mode and let it choose the model it considered more appropriate. This took 2m51s and used 3.85 credits, and used 7% of the context window. Because my test files didn't have any superscript or subscript formatting, it created a dummy file to test the conversion, and removed them as soon as it was happy with the results.
Here's the test output:
> python .\epub2md.py ..\test_data\test1.epub
Wrote 'test1.md' from 21 document(s).
> python .\epub2md.py ..\test_data\test2.epub
Wrote 'test2.md' from 14 document(s).
The output code was a 627 line script, and the italics was correct, as well as line endings and paragraph splittings.
Here's a summary:
| Model | Total Credits | Total Time | Context Window Usage | Lines |
| Qwen 3 | 18.52 | 28m 31s | 54% | 650 |
| Claude Sonnet 4.6 | 12.27 | 7m 12s | 9% | 542 |
| Claude Sonnet 5.5 | 14.27 | 9m 33s | 13% | 1016 |
| Auto | 3.85 | 2m51s | 7% | 627 |
Note that these numbers are the final credits used. The baseline Auto model uses 1x credits. For Claude Sonnet, it uses 1.3x credits, and Qwen uses 0.05x credits. Even with these multipliers, Qwen still took the longest and used more credits than the other models.
You might get slightly different results on your own tests, depending on memory, processor, and so on, but all these were collected on the same computer, running the same programs, so the relationships between the results should be consistent.
Looking at them, it seems best to just leave Kiro on auto and let it go wild.
Of course some models are better at certain programming tasks. But it's worth giving different models a try and seeing which generates results that you're more comfortable with.
And, always, always, validate the generated code.