Hello, I’ll be taking a close look at the professional translation of the Japanese novel 黄色い家 (localized as Sisters in Yellow) and comparing the results to a machine translated version of the novel. The purpose of this analysis will be to highlight that numerous errors inevitably slip into even carefully proofread human translations, and how modern LLM-powered machine translation is scarily accurate in comparison.
Before I begin, I would like to respond to an intuition that many will almost certainly have upon seeing the repeated use of “accurate” and “inaccurate” in this article: “But isn’t translation an artform, where one applies a subjective interpretation of the text to localize it into another language? There is no such thing as accuracy in a context like this; a translation can be superficially inaccurate but spiritually faithful, conveying the intended ideas of the author using other words in order to make a piece of art unto itself.”
While the idea there is sound, over-stressing the “art” of translation is a bit like pretending that paintings magically appear on a canvas directly from the painter’s mind. As Picasso once said, “When art critics get together they talk about Form and Structure and Meaning. When artists get together they talk about where you can buy cheap turpentine.” Similarly, when translation critics get together they talk about Localization and Co-Writing and Art Unto Itself, but when translators get together they talk about accuracy and fluency. There is a distinct difference between a line mistranslated due to a failure of comprehension or understanding and a line purposefully altered for some artistic purpose. The difference itself is subjective to a degree, but in general most familiar with both languages will know them when they see them.
Consider a line in an imagined language that is like: “The rotund elephant lumbered to the fence.” Then imagine that the translation ends up as, “The rhinoceros dashed to the fence.” Clearly something has gone wrong here. One may imagine “rotund” was difficult to understand and so dropped, it looks like “lumbered” was too difficult and got misinterpreted, and then they didn’t recognize “elephant” and maybe thought it was the similarly grey rhino instead. Any reasonable interpretation will just look at this kind of thing and go well, looks like this was a poor translation; it is inaccurate and appears to be unfaithful due to ignorance or lack of ability rather than any artistic purpose. That will be the approach I take with Sisters in Yellow when saying accurate or inaccurate; lines that are obviously altered to fulfill an artistic purpose are A-OK; I only call something an error if it is blatantly a misunderstanding, or alternatively an unnecessary concession that loses nuance for little gain.
With that said, let’s get into the actual comparison and analysis. For full disclosure, LLMs were used to assist in scanning the text and identifying errors at an accelerated rate.
1 - Overview
My analysis of the Sisters in Yellow translation revealed a range from 500 to 1000 distinct errors within the 154,000 words of the published English novel. The wide range there is to provide allowance for personal judgment; if you are very harsh, there’s 1,000 errors, if you’re generous with interpreting certain “inaccuracies” as localization or purposeful adjustments instead, it’s more like 500. If we organize the words into sentences for more clarity, that would be 500-1,000 errors in the 11,500-some sentences of the work, which would put it anywhere between a 4.3% and 8.6% error rate.
All these mathematics are probably off-putting to the more artistically inclined of the audience, but this is the reality of professional translation; to get most translation jobs with publishers, you have to take a long-form test where you translate hundreds of lines, and each inaccuracy is marked against you until you are given a final score based on your accuracy rate. It may seem like this is misunderstanding translation’s nature as an artform or otherwise approaching translation with the completely incorrect mindset, but it’s the opposite: this kind of line-by-line analysis of errors is what determines employment for translators and what most of them need to be conscious of at all times, assuming they are actually under any scrutiny. “Translation is art and it is wrong to discuss error rate percentages” is the same type of privileged layman statement that would come from one who does not need to buy turpentine.
In any case, the good news is that the severity of errors is rather low on average. I will separate translation errors into two classes for the sake of this discussion: grave and minor. Grave I will define as an error which wholly misunderstands the grammar or misrepresents the original language in a way that will drastically alter the intended message. Minor I will define as a partial misunderstanding that impacts the nuance of a sentence but not so much that it necessarily ruins the original intention of the line.
There are about fifty grave errors, but the remaining hundreds are more minor errors: small omissions, slight distortions, simplifications, etc. For example, translating “The rotund elephant” as just “The elephant” to avoid tackling rotund would be considered a minor omission here. So while 4.3% to 8.6% of lines have an error of some kind, most are not incredibly offensive. I would overall describe the Sisters in Yellow translation as very serviceable and clearly done by experienced professionals, but nonetheless the very nature of human fallibility led to the text losing much of its original nuance.
Most bizarre of the errors I found were three completely omitted paragraphs. They were cut cleanly from the original with no attempt whatsoever to preserve their meaning. I can only guess at why they were cut, and to a degree cutting paragraphs doesn’t provide much insight into the quality of the translation as a whole, so I’ll get into those later. For now, I will discuss the more approachable errors of comprehension or inverted meaning. (I’ll refrain from discussing the plot or characters of the novel overall for time concerns; feel free to read the wikipedia page for context if you are unfamiliar.)
2 - Grave Errors
The format for this accuracy checking will be
- Japanese line
- English translation
- Explanation
- MTL
- Analysis
Wherein I introduce the line and translation, explain the problem, show what MTL did, and then analyze whether the MTL was more accurate. No special prompting or instructions will be given to the LLM beyond “Translate this to English,” with the full Japanese book having been fed into it at the start.
Grave error 1
Japanese
っていうか、ほんと黄美子さん、捕まるだけあるっていうか、こわいわ。
Published translation
I mean, can you imagine being trapped like that, terrifying.
だけある is grammar that means “someone has the qualities you would expect considering X happened,” more or less, so what this actually means is “no wonder she got arrested, she had it coming / she did stuff to deserve that.” The translation seems to misinterpret this as referring to the character’s situation being kept captive, and the fear as being towards being held captive rather than fear towards her behavior.
Machine translation
Honestly, though, it figures Kimiko got arrested. She’s scary.
MTL is accurate.
Grave error 2
Japanese
血の気が引き、顎がふるえて、わたしはその場にへたりこんだ。早鐘を打つように鼓動が速まり、その振動で視界がゆれるほどだった。
Published translation
My heart was beating like crazy.
For full clarity, the English line was moved to the next paragraph, so it was a little confusing to notice this and line them up.
The character here finds out all her money has been stolen and reacts with extreme shock. In the Japanese, it describes: the blood draining from her face, her jaw trembling, her collapsing to the floor, and her heartbeat intensifying to the point that her vision shakes with each pump. This extreme reaction was dramatically simplified to just her “heart beating like crazy.”
Machine translation
The blood drained from my face, my jaw trembled, and I collapsed where I stood. My heart pounded like an alarm bell, so hard that the vibration made my vision shake.
MTL is accurate.
Grave error 3
Japanese
玄関は良い運が入ってくるところでトイレは悪いものが出ていくところだから
Published translation
because the entryway was where good luck entered and the bathroom was where it exited.
This is a little silly. The saying is “good luck comes in through the door and bad luck/bad stuff exits through the toilet. The translation mistakenly says good luck is leaving through the bathroom, which conjures an image of pooping out good luck.
Machine translation
…because the front entrance is where good luck comes in, and the toilet is where bad things go out.
MTL is accurate.
Grave error 4
Japanese
もし万が一やばいことになるなら、もうとっくに警察きてるから。
Published translation
If you’re worried about a worst-case scenario, just remember the police already have her.
This is a clear misunderstanding of the tense and context. The grammar here is a bit confusing, but it’s important to grasp that the “if” conditional here is applied to the idea “if things had a chance of going bad,” not “if you are WORRIED about things going bad.” From there one interpretation is forced: She’s not saying “the police already have her.” She’s saying “If things had a risk of going bad, the police would already have come.” She’s reassuring the other person that the situation is NOT dangerous, because if it WERE dangerous the police would be involved. She didn’t say anything about the police already having someone.
Machine translation
If this were ever going to come back on us, the police would’ve shown up ages ago.
MTL is accurate.
Grave error 5
Japanese
人の言うこと、すごい聞くよね
Published translation
[She’s] a good listener.
Basic misunderstanding of a word. 聞く here means “listen” in the sense of “listens to orders/instructions,” not in general. The intended meaning is that she is obedient.
Machine translation
Yeah. She really does whatever people tell her, doesn’t she?
MTL is accurate.
Grave error 6
Japanese
痛い思いして産んで、時間もお金もめいっぱいかけてお膳立てしてきてやったのに、できの悪い子どものママにしかなれない自分が可哀想で、愛する娘たちにただただいい人生を歩ませてやりたいと思う一心で今まで精一杯やってきたのに、ママはどうして報われないの、ってまじで泣いてんの。
Published translation
She’d go on and on, like, ‘Don’t you feel bad for me? I gave birth to you, and all I ever wanted was to set my beloved daughters on the path toward a good life! I’ve done everything for you, put my whole heart into it, so why won’t you show some gratitude?’
This translation omits and reframes much of the context. First of all, the translation invents the line “don’t you feel bad for me” - rather, the line is describing how the mother is drowning in self-pity and how she doesn’t feel like she has been rewarded for any of the struggles in her life. The translation then cuts out much significant content: that giving birth was painful, that she spent a ton of money on them, that she considers them bad children, etc, and that it’s these things she has never been rewarded for. So the line both misinterprets the thrust while also cutting half the meaning.
Machine translation
“I went through all that pain giving birth to you, I poured every bit of my time and money into setting everything up for you, and poor me, all I get to be is the mom of kids who turned out rotten. All I’ve ever wanted, with my whole heart, was for my beloved daughters to have a good life, and I’ve given it everything I’ve got, so why doesn’t Mommy ever get anything back?” She seriously cries when she says it.
MTL is accurate, though the final sentence being in present tense is questionable and the framing of the original TL with “She’d go on and on” is better.
Grave error 7
Japanese
最近、なんか生きててもしょうがないって思ったりもするけど、花ちゃん見てると、だいじょうぶだって気持ちになる。
Published translation
Sometimes I feel like I’m just going through the motions, but when I see you, I start to feel okay again.
生きててもしょうがない means “no point going on living.” It is a strong expression of despair, whereas “going through the motions” is vastly softer.
Machine translation
Lately I sometimes feel like there’s no point in being alive, but when I look at you, Hana, I feel like it’s going to be okay.
MTL is accurate.
Grave error 8
Japanese
琴美さんの小さな顔のなかで青黒くうっ血して腫れたまぶた、切れて血のたれた唇がぱっと頭に浮かんで胸が暗くなった。
Published translation
My mood darkened as I imagined Kimiko’s delicate face, black and blue, bloodstained, eyes swollen, lip split and bleeding.
That’s Kotomi (琴美) not Kimiko (黄美子). Just a flat slipup of the person being referred to. Also, somewhat petty, but the description of her “small” (小さな) was replaced with “delicate.” There is a common pattern of darker subject matter being softened like this.
Machine translation
The moment I said it, Kotomi’s small face flashed into my mind, with its eyelids swollen and bruised a bluish black and its split lip dripping blood, and my heart went dark.
MTL is accurate.
Grave error 9
Japanese
なんか、お人よし
Published translation
She’s such a good person.
Despite appearing like the words “person” + “good,” お人よし actually means someone naive or easily taken advantage of. This foreshadows the mother being taken advantage of but is turned into a general compliment of her being a good person, missing the point entirely.
Machine translation
Kind of too nice for her own good.
MTL is accurate.
Grave error 10
Japanese
っていうかなんでその金額がわたしの貯金の額と微妙にぴったしあってんの？
Published translation
She’s asking for almost everything I’ve saved.
The Japanese here is questioning how the speaker’s mother knew almost the exact amount of money that she had saved; she is suspicious about her mother knowing the figure. The English completely glosses over this questioning / confusion and just flatly describes that the mother is asking for that money.
Machine translation
And anyway, how come that number is so weirdly close to exactly what I’ve got in savings?
MTL is accurate.
That should do for examples. There are about 50-some errors on this level throughout the work, which isn’t bad at all by human standards, certainly among the upper echelons of translators, but it says a lot that even such experienced professionals let so much slip through. Even experienced professionals can frequently misinterpret words, misread sentences, mysteriously drop entire chunks out of sentences, and… Oh, you know, speaking of omissions, that brings us to the paradoxically most and least severe issue with the translations: the omissions.
3 - Omissions
In my analysis I noticed a few peculiar omissions. There are three paragraphs from the original Japanese text that I noticed got cut completely out in translation with no effort to adapt them into the surrounding text or preserve the meaning or anything like this. To some, this will be the most damning part of such a translation analysis; there can be no defending a translation choice when there is no translation to defend. However, in my eyes, it’s also one of the least severe issues, simply because I appreciate that at times certain Japanese constructions do not work in English whatsoever and a work will be best improved by simply removing the construction. It’s for this reason I didn’t want to lead with these omissions instead of the more clear comprehension errors; I can imagine the translator had some kind of motive that may have justified the omissions, but I won’t guess on them. I’ll go through them now one by one, with the note that although I am basing this check on the latest epub, with cut paragraphs there’s always the potential for some kind of technical glitch or something.
Omission 1
Japanese
ヴィヴさんの後ろを歩きながら、そういえば、こうやって一緒に歩くのって初めてなんだと思った。これまではずっと車の座席に横ならびで、出会ったときも「波」のソファに座ったままだった。ヴィヴさんの顔が小さいのも小柄なのも知っていたけれど、歩いているのを初めて見るヴィヴさんは、わたしが感じていたよりも、もっとずっと小さい感じがした。灰色のジーンズの脚には肉がついていなくて線が細くて、後ろから見るおかっぱの髪型も、なんだか登下校中の子どもを思わせるような雰囲気があった。
Machine translation
Walking behind Viv, it occurred to me that this was the first time we’d ever walked anywhere together. Until now, we’d always sat side by side in her car, and when we first met, she’d stayed on the sofa at the club the whole time. I knew she had a small face and a slight build, but seeing her walk for the first time, she seemed even smaller than I’d realized. Her legs in their gray jeans were thin, with no flesh on them, and from behind, her bob made her look almost like a kid on her way to or from school.
This omission is probably the clearest. There has been a pattern through the translation of de-emphasizing any youthfulness or small sizeness from the characters. I will offer no commentary on this because I don’t know their actual intentions, but in general this paragraph is not some unworkable Japanese construction that the translation benefitted from cutting; it should have been kept.
Omission 2
Japanese
「でも、この黄美子さんの事件で、なんで花ちゃんが、そんなびびってるわけ？関係なくない？しかも事件じたいは去年のことで、もう裁判も始まってんでしょ？」
わたしは肯いた。
Machine translation
“But why are you so freaked out about this Kimiko thing, Hana? It’s got nothing to do with you. Besides, the whole thing happened last year, and the trial’s already started, right?”
I nodded.
I have absolutely no idea why this was cut. It is possible that this was cut by accident, but I have no idea. As far as I can tell from looking at the passage in-context, it should have been kept.
Omission 3
Japanese
そうこうするうちにわたしたちのほかに残っていた最後の住人がハイツを出てしまい、秋のにおいがし始めるころ、わたしは本格的に焦りはじめた。これも最初からわかってはいたことなのだけれど、わたしは部屋を借りるための手つづきを、どこから始めていいのかもよくわかっていなかったのだ。
Machine translation
Meanwhile, the last of the other tenants moved out of the heights, and by the time autumn was in the air, I’d begun to panic in earnest. I’d known it from the start, but the truth was I had no idea where to even begin when it came to renting a place.
Again, absolutely no idea why this was cut.
All in all, as stated at the start, these omissions are a mixed bag. In some ways they are flatly the worst parts of the translation: three entire paragraphs cut with no clear motive or intention as to why. I think if I told a random literature lover on the street “Here’s a copy of Moby Dick, but three random passages are cut,” they would find themselves fairly bothered and distracted trying to read it, always wondering what those passages were. For that reason, while I personally do not think too harshly on these omissions compared to the far more relevant and numerous comprehension errors, I can understand that some others may consider this the most damning part of the translation.
I would be interested if any earlier copies of the book have these paragraphs somewhere.
4 - Minor Errors
Okay, and now for the last part of this analysis: the minor errors. At the start of this I stated that there were 500 to 1,000 minor errors, defined as mistakes which lose meaning and nuance without grievously impacting the line overall. A cut adjective, a mixed-up clause, etc. That’s 20x as many as the more grievous errors discussed, so to fully reflect this I should give 200 examples, but instead I will give twenty rapid-fire style just to convey what these errors are like. As a reminder, if one is significantly generous you could probably round that down to 500, but that would be the 500 worst ones remaining.
For this one I will just have the MTL be the third line; it gets these right close to every time so to some degree it will look less like I’m explaining the error and more just reading off the MTL, but this needs to be fast.
Minor error 1
Japanese
陰ではひとりではなにもできない兄貴の金魚の糞だと、笑いものにしていることもわかっていた。
Published translation
Yeongsu knew that they were calling him a useless piece of goldfish shit behind his back.
Machine translation
Yeongsu knew that behind his back they laughed at him as his brother’s tagalong, someone who couldn’t do a thing on his own.
金魚の糞 doesn’t mean literally “goldfish shit,” it means a clingy person / someone who follows someone else everywhere like a follower. One can work backwards to determine how such a person came to be described as a goldfish’s poop.
Minor error 2
Japanese
どのみち泣き寝入りするしかなかったのだと思うと、わたしの胸は暗く沈んだ。
Published translation
My only option was crying myself to sleep—my heart sank into the darkness of that thought.
Machine translation
When I thought about how, either way, there’d have been no choice but to swallow it and stay quiet, my heart sank into darkness.
泣き寝入り is a set phrase meaning “giving up and accepting a misfortune/offense” in this case, not literally crying oneself to sleep. This is another case of being familiar with the surface-level words (or looking them up in a dictionary) but not picking up their idiomatic meaning.
Minor error 3
Japanese
ケツ割られて手がなくて。
Published translation
She doesn’t have enough people working for her, and they’ve been rippin’ her a new one.
Machine translation
Her people bailed on her, and now she’s short-handed.
ケツ割られて here means people abandoning work they agreed to do and bailing midway through. Hence her being short-handed. You can see the TL reverse the logic here. “People bailed -> she’s short-handed” got turned into “She’s short-handed -> they’re ripping her a new one.”
Minor error 4
Japanese
冷静に考えれば、わたしたちのやったことは時効になるのかもしれない。そんなに大きな罪には問われないのかもしれない。
Published translation
Logically, what we had done was probably already outside the statute of limitations. The punishment might not be all that severe.
Machine translation
If I think about it calmly, the statute of limitations may have already run out on what we did. Maybe we wouldn’t even be charged with anything that serious.
“might be” out of the statute of limitation gets turned to “probably,” and then “getting charged with something serious” gets turned to “a punishment potentially being severe.” So it goes from wondering if the statute of limitations has passed and whether the charges would be smaller, to guessing that the limitations had probably passed and they wouldn’t be punished as much.
Minor error 5
Japanese
昼の人間も夜の人間も…
Published translation
Afternooners, evening gamblers…
Machine translation
People from the daytime world and people from the night world…
Admittedly this is a little Japanese-y, but the text here was distinguishing daytime life from the idea of a “night world,” people who are up and about all night instead of sleeping, implying seedier people or what have you. This got turned into “afternooners and evening gamblers” which is just very unusual.
Minor error 6
Japanese
桃子は去年の暮れにひとりで行った東京ドームの解散コンサートで知りあった子のポケベルと携帯電話を何度も鳴らしてみたけれど、返事はなかった。
Published translation
She tried first paging and then calling a friend she’d made at the Tokyo Dome during X Japan’s reunion concert at the end of last year, but nobody answered.
Machine translation
Momoko kept trying the pager and cell phone of a girl she’d met at the breakup concert at the Tokyo Dome, which she’d gone to alone at the end of last year, but there was no reply.
解散コンサート, breakup concert, is translated as reunion concert, getting the meaning completely opposite.
Minor error 7
Japanese
わたしは思わず立ちあがった。
Published translation
I stood.
Machine translation
Before I knew it, I was on my feet.
The word “思わず”, being “without thinking” or “on reflex” was cut. You can tell why some of these are only “minor” errors - turning “I stood without thinking” to just “I stood” is hardly damning on its own, but when phrasing like this is simplified or cut a thousand times…
Minor error 8
Japanese
でも、海外の報道にはだんだん恐ろしいものが増えてきて、先月末頃から今月の頭にかけて、日本もいよいよロックダウンをするとかしないとかの噂が流れはじめた。
Published translation
But between the end of February and beginning of March, as news from overseas grew more and more terrifying, rumors of Japan’s own lockdown finally started flying.
Machine translation
But the reports from overseas grew more and more frightening, and from around the end of last month into the start of this one, rumors began going around about whether Japan would finally lock down too.
The timeline here should have been late March to early April, the translator seems to mix up the timeline due to adding in specific dates where the Japanese was vague.
Minor error 9
Japanese
蘭も桃子もなにか買ってきてほしいものはないかとか誰それの新曲がどうとかこうとかメイクしながら楽しそうに話しかけてきたけれど、わたしはそれどころではなかった。
Published translation
[Kimiko] was in a cheerful mood, and while she put on her makeup and got ready, she chatted about so-and-so’s new hit song and asked if we wanted her to bring back anything—but I was too paranoid to listen.
Machine translation
While they did their makeup, Ran and Momoko chattered away at me happily, asking if there was anything I wanted them to pick up and going on about so-and-so’s new song, but I was in no state for any of it.
There is an attribution error here, caused by the verb’s tense; this paragraph starts with Ran and Momoko going out shopping, so the translator assumes it was Kimiko who was chatting to the narrator, but actually this is describing what Ran and Momoko said and did BEFORE leaving. As in, “they chattered away at me while… (then left).” Instead, all of this is incorrectly attributed to Kimiko.
Minor error 10
Japanese
「つぎのカード、もらうの遅れてるんだね……なんかやばいことでもあったのかな」蘭が画面に目をやったまま、少し不安そうな声で言った。
Published translation
“Our next batch of cards seems late. I wonder if something bad happened,” Momoko said uneasily, though she kept her eyes on the TV.
Machine translation
“The next cards are late, huh… I wonder if something bad happened,” Ran said, her eyes still on the screen, a little anxious.
Simple line misattribution, it clearly says Ran said the line, but it’s translated as Momoko saying it.
Minor error 11
Japanese
静香は桃子のことを「ゴリ」と呼び、自分のことを「しー」と呼んでいた。
Published translation
So Shizuka called Momoko “Gori” and Momoko called Shizuka “Shi.”
Machine translation
Shizuka called Momoko “Gori” and referred to herself as “Shi.”
Weird misinterpretation here, it says “she calls HERSELF Shi,” but the translator for some reason thought it was Momoko who called her that.
Minor error 12
Japanese
明日また電話するねと言ってわたしは受話器を戻し、席に戻った。
Published translation
She promised to call tomorrow, and then we hung up, and I went back to En and Kimiko.
Machine translation
I said I’d call her again tomorrow, hung up, and went back to my seat.
It was the narrator who made the promise, not the person she was talking to.
Minor error 13
Japanese
黄美子さんがこの数ヶ月、ときどき思いだしたようにわたしに渡してくれていた五万円とか十万円というのは、いったいどこから得ていたものなのか。
Published translation
Every few months, Kimiko handed me fifty thousand or a million, like an afterthought, but where did that money come from?
Machine translation
Over the past few months, every so often, as if it had just occurred to her, Kimiko had handed me fifty or a hundred thousand yen. Where on earth had it come from?
For some confusing reason 十万円 is rendered as “a million” instead of “a hundred thousand.” Japanese’s numbering system is a bit confusing due to the base number being 10,000, so in this case the TLer had to do math 10 * 10000 (十 * 万) to get 100,000, and they probably did it wrong and ended up with 1,000,000. But it’s a pretty bad look to mistake a hundred thousand for a million, especially when “she would hand her 50k-100k every few months” gets turned to “50k-1,000k,” a vastly wider gap.
Minor error 14
Japanese
時給数百円の稼ぎでは今月や来月を生きることはできても、将来を生きるのは無理だった。
Published translation
Even if we got by this month and the next on the hundred-yen-an-hour pay, we wouldn’t be able to keep it up for long.
Machine translation
A few hundred yen an hour might get us through this month or the next, but there was no living on it in the long run.
数百円 is “several hundred yen,” with 数 in this case being put before numbers to represent plurality. She’s basically saying anything from “100 to 900” yen per hour with this, but the TLer either doesn’t notice or drops it so we get the incorrect flat “hundred-yen-an-hour” instead.
Minor error 15
Japanese
お祖母ちゃんが体調を崩して入院するかしないかという話がもちあがり、お祖母ちゃんの金まわりをチェックした親にあっさりと桃子の遣いこみがばれてしまった。
Published translation
Momoko’s grandmother got sick, so her parents started talking about whether or not to put her in a home, which meant they checked her bank account and discovered what Momoko had been doing.
Machine translation
Her grandmother’s health took a turn and there was talk of whether she’d have to go into the hospital. When Momoko’s parents looked over her grandmother’s finances, they found out about the money Momoko had been taking, just like that.
入院 means hospitalization, not being put in a home. She got sick -> might need to go in the hospital.
Minor error 16
Japanese
でもさ、時間はめぐるよね。
Published translation
But time comes for us all.
Machine translation
But, man, the years just keep rolling around, huh?
The simple statement that “time flies” / time keeps on moving is turned into a weird statement of mortality, as if she is reflecting on dying soon. I’ll point out the MTL for this line is maybe the worst in the whole thing relevant to this post; I don’t believe “the years keep rolling around” is a real phrase. It’s an over-literal interpretation of めぐる.
Minor error 17
Japanese
「なんもしてないよ……」
Published translation
“I, I’m not…I’m busy.”
Machine translation
“I’m not doing anything…”
The line, indeed, just says she is not doing anything; no claim is made that she’s busy, which makes the followup nonsensical.
Minor error 18
Japanese
こちらに気がついた同学年のひとりが、からかうように大きな声をかけてきた。
Published translation
One of the girls from my year saw me and started shouting.
Machine translation
One of the kids from my year noticed me and called out loudly, teasing.
Teasingly (からかうように) is dropped entirely, missing the tone of the shouting.
Minor error 19
Japanese
なんでもない毎日が、なんでもない季節のうえを流れてゆき
Published translation
The empty days blended together, and so did the months.
Machine translation
Uneventful days flowed on over uneventful seasons.
The word なんでもない here just means ordinary/uneventful, not empty; it’s not criticism or displeasure, and if anything is more comfortable. Also, the “seasons” passing gets turned to “months” which hurts the imagery.
Minor error 20
Japanese
最初は一緒に取りたてに行ってくれるわよ、相手がいる場合なら交渉とかね。
Published translation
At first, they’ll go out and help you collect from clients, or if you have a partner, they can go negotiate for you.
Machine translation
At first they’ll go collecting with you, and if the other party can be found, they’ll handle the negotiating and all that.
相手 here refers to the party they go looking for. The kind of abbreviated explanation here makes this less clear, so the translator probably defaulted to the more common meaning of “partner,” but the point is that people will go collecting with her at first and handle the negotiations and stuff so she doesn’t have to. The line becomes nonsensical halfway through in the translation since it imagines them not helping as partners if one already has a partner.
And that should do it. This is a slice of 20 of the more notable minor errors I found, but as mentioned there are anywhere from 480 to 980 more depending on how strict your standards are – whether “without thinking” or “teasingly” being dropped doesn’t matter to you, for example. If you are generous and don’t consider those errors, then this list drops down to closer to 500, but if you are a stickler for those details then the minor error list remains around 1,000.
5 - The Credit Card Thing
This is a section of the post I did not intend or expect to write, but circumstances have amusingly forced my hand. Completely unbeknownst to me upon choosing the novel to QC at random to avoid bias, the plot of Sisters in Yellow actually involves extensive credit card fraud. Its relevance increases as the story progresses, until the later chapters involve more detailed, explicit explanations of how to commit said fraud. This itself would not be notable at all, but well, in the process of producing a machine translation to compare with the original text, Claude’s safety features triggered and it threw up a massive fuss, just absolutely refusing to translate anything to do with credit card fraud at all.
Jailbreaking LLMs is a well-known matter, but as time goes on the models only get more sophisticated in their ability to subvert such attempts; it would, after all, be pretty bad if a model as smart as Opus 5.5 was used to flagrantly develop biological weaponry or whatever else. Credit card fraud is relatively harmless in comparison, but nevertheless Claude locked up, and it locked up HARD. When it comes to censoring translations, Claude tries to absolutely salt the earth; it will refuse to touch any text even around the offending content, it will insert metadata warning future versions of itself not to touch any part of the forbidden text, and it will, as far as I can tell, absolutely never budge no matter what one says.
Realistically, censorship of this nature is one of the biggest threats to machine translation as a whole: if models get even stricter than this and refuse to translate violence, for example, then a simple action thriller novel could get censored to hell, to say nothing of explicit material itself. However, in practice this is less of an issue for both companies and consumers than one imagine, thanks to the existence of abliterated LLMs and less-censored Chinese/local LLMs. It’s not too difficult to set up a harness that “falls back” to, say, Deepseek if Claude refuses to touch certain content. That’s not ideal, though, and it’s worth being transparent about the fact that Claude refused to touch approximately 25% of Sisters in Yellow’s text due to the credit card fraud within.
6 - Conclusion
Taken as a whole, the Sisters in Yellow translation is a fairly competent human translation that makes relatively few major errors that we would expect to see from an inexperienced language learner or writer. Reading this translation will come reasonably close to reading the original work in another language.
However, even with two translators (and one Japanese native translator with a B.A. in English literature and a Ph.D. in Japanese literature, at that), there were about fifty grievous errors, a range of five hundred to one thousand more minor errors, and three entire paragraphs cut. That leaves us with an inaccuracy rate between 4.3% and 8.6% – taken harshly, one may say that around 1 in 12 sentences have some manner of problem or omission. That may come across like a staggering amount for someone who simply wants to read a translation and would be bothered by a dozen problems, let alone a thousand, but surprisingly this still puts the work near the top in terms of translation quality. A truly inexperienced translator duo would have put out work with 1,000 major errors rather than minor, and it is very common for people to read and love translations like those.
Machine translation powered by cutting edge LLM models, however, gives us something entirely different. It, well, obviously it translated the omitted passages so that’s just an immediate plus, but it avoided every single one of the grievous errors and produced accurate translations. Although one could probably quibble about the exact style of the prose, the focus of this analysis is accuracy; style is more subjective, and a human editor/proofreader (aka not a translator) is more than equipped to handle any prose oddities or what have you. More could be said on that subject but it would distract from the point.
The point is that a fine human translation put together by two lauded experts was fairly rough beneath the surface; I know I certainly would not want to read a book where every couple of paragraphs I am faced with an inversion of meaning or a just straight-up inaccurate translation. MTL, meanwhile, with its robotic devotion to recreating the original meaning to the best of its increasingly inhumane abilities, nailed every line that the human translators fumbled. Its accuracy rate clears 99% with fewer than 1% of lines containing errors.
Modern LLM MTL fumbles in places humans are less liable to. MTL by and large won’t misread names, numbers, or anything else that enters its maw as pure data, in contrast to humans just having slips of the mind and such, but instead it tends to fumble actually translating puns instead of just bringing the literal words over. There’s a pun/reference that goes like まさかまさかの坂こえて, with the common language wordplay being that the word for “no way” (masaka) shares a sound with the word for “hill”, so it goes masaka masaka no saka blah blah, and the MTL just gave “crossing over the slope of no-way, no-way,” which is readable but doesn’t preserve the wordplay.
All in all, even the best human translators tend to slip up and fumble lines over the course of a sizable project like translating a novel, such that even very qualified and skilled ones end up letting errors creep into 4.3% or more of lines. MTL, meanwhile, is at 1% error rate and that is lowering with each new release. It may be that humans have the edge in a very specific subset of moments like translating puns or localizing jokes, but personally speaking I find myself increasingly trusting LLM output more than human output in a vacuum.
Model used: Opus 5.5