400 Bad request errors with Japanese texts

[ttreder-explorance]

Describe the bug
We are getting 400 errors when calling POST /v2/translate with the DeepL-node library translateText() method when using Japanese comments from the following file.

To Reproduce
Steps to reproduce the behavior:

• Load all the comments from the provided xlsx file into an array.
• Batch comments with payload size < 120KiB.
• Call translateText() with arrays of comments, no sourceLang param and 'en-US' as targetLang.
• The first 4 batches should fail with this response:
  DeepL API response POST /v2/translate 400 Response details:, content =
• The 5th batch should succeed with a 200 OK response.

Expected behavior
The /v2/translate endpoint should have a consistent behaviour when translating Japanese texts.

Environment:

• NodeJS: 18.0
• DeepL-node: 1.13.0
• macOS, Windows or Linux

Additional context
We did try the translation API with a lot of different languages, so far Japanese seems to be the only one causing this issue.
Also worth noting that when we use the /document DeepL API translate the document correctly.

[daniel-jones-dev]

Hi @ttreder-explorance, can you please share the code you used, or the array indices in each batch? I was not able to reproduce the issue so far, using the first 60 or 100 array items.

[ttreder-explorance]

const translator = new Translator(YOUR_API_KEY, {
  maxRetries: 5
  minTimeout: 20000,
});
log.getLogger('deepl').setLevel('debug');
let searchParams = new URLSearchParams();
let commentsToTranslate: AnalysisMappedComment[] = [];
comments.forEach((comment) => {
        searchParams.append("text", comment);
        const commentsByteLength = searchParams.toString().length;

        if (commentsByteLength > 122880) {
          const result = await translator.translateText(
               commentsToTranslate,
               null,
               "en-US",
          );
          // Do what you want with the result

          searchParams = new URLSearchParams({ text: comment });
          commentsToTranslate = [comment];
        } else {
          commentsToTranslate.push(comment);
        }
      });
}

if (commentsToTranslate.length > 0) {
      const result = await translator.translateText(
            commentsToTranslate,
            null,
            "en-US",
       );
       // Do what you want with the result
}

[daniel-jones-dev]

Hi, thanks for the code. I was not able to reproduce the problem. I'll ask some other engineers if they have an idea.

Does the issue happen consistently for these inputs? I'm wondering if it is the input content triggering the issue.

If you lower the batch size limit (to say 100KiB), is the problem resolved? That wouldn't be ideal, but might be useful to understand the problem.

[ttreder-explorance]

Hi @daniel-jones-dev,

The issue is happening consistently with the same file. It's also not happening consistently with other languages files (even arabic).
I limited the batch size to 100KiB. This time I got 6 batches that all failed for the same reason.

[paschaldev]

Hi @daniel-jones-dev

I might be experiencing something similar. i tried translating a Markdown text from Spanish -> English. We're in production and wanted to ship this feature but this is limiting at the moment and I do not know how to resolve.

export const translator = new deepl.Translator(env.DEEPL_API_KEY);

const result = await translator.translateText(
    text,
    "es",
    "en",
  );
  return result;

When I used this as English -> Other language, things worked fine even with Markdown. HOwever, translating from other language back to English was throwing an error: DeepLError with little to no other information so I do not know what I am doing wrong.

I tried translating same text on the web using this https://www.deepl.com/en/translator and it worked fine

[paschaldev]

@daniel-jones-dev I found the problem. I used en instead of en-US but I'll suggest this should be automatically handled. When I pass en it should automatically fallback to en-US or en-GB as the case may be except I pass a full locale.

[ttreder-explorance]

Hi @daniel-jones-dev, I just got the same 400 bad request with some French texts this time. It not happening all the time anymore. It looks very random. Sometimes it's working, sometime not. Not sure what to think about it anymore.

[ttreder-explorance]

I'm using
this file's réponse column First batch fails, second works.

[mgagliardo91]

Also seeing an issue with receiving a 400. In my case, it was related to the amount of requests that were being issued in parallel from the client. Sending the same requests sequentially resulted in no error. I think there may be some issue either in the client or on DeepL's side related to load in some way? Haven't dug much deeper to see where the issue lies.

[ttreder-explorance]

@daniel-jones-dev by digging deeper into this I managed to solve this problem that I wish was documented properly. So for future devs that stumble onto the same problem, here it is:

We sometimes have very large files of texts to translate and have to batch calls to the DeepL API. For security reasons, the api has a throttling algorithm implemented, but the configuration for it is undocumented.

From our early tests, we realized that having a leaking bucket of 10 requests + a refresh of 1 per second afterward was the best to maximize the batches speed while not getting any 423 Too Many Requests error.

What a surprise when we discovered that you can also get 400 undocumented errors when you send large payloads (size will vary depending on the language of your texts, probably due to encoding) at a fast rate.

To solve that issue, we removed changed our leaking bucket algorithm to only allow 1 request every second. With this, we can send size maxed payload requests without getting that 400 Bad Request error (that should really be a 423).

I think those details are fairly important for developer that wants to use your API and I would suggest adding them to your Usage and Limits page.

[daniel-jones-dev]

Hi @ttreder-explorance, thanks for the follow up and the detailed report.
The team responsible for this component confirmed this is not expected behaviour. Yes our API should respond with 429 Too Many Requests in this case. We're going to look into it more, I'll update here when I have more information.