Allow `model_max_tokens` to be set to whatever the LLM maximum is

[iuliaturc]

Description

I'm trying to use GenerateREADME and maximize the underlying LLM's context window. But unfortunately I can't figure out easily what that magical value is, because model_max_tokens isn't the length of the final input sent to the LLM.

For instance, I'm trying to consume the entire 128k context window. And I'm doing a bunch of trials:

1. patchwork GenerateREADME ... model_max_tokens=128_000 ===> Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, you requested 255511 tokens
2. patchwork GenerateREADME ... model_max_tokens=64_000 ===> Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, you requested 191511 tokens
3. patchwork GenerateREADME ... model_max_tokens=30_000 ===> Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, you requested 157511 tokens

So I need to keep guessing.

Proposed solution

Have an option to e.g. set model_max_tokens=-1, which would mean the maximum window allowed by the underlying LLM, once all the other tokens you're sending under the hood are accounted for.

Alternatives considered

n/a

[iuliaturc]

Uhm silly me I could have just calculated the extra token count is 255k - 128k = 127k to realize that the maximum I can set model_max_tokens is 1k. But anyway, would still be nice to have something like model_max_tokens=-1 to not have to go through a failure and then do the calculation.

[iuliaturc]

Which also makes me realize that passing the entirety of the codebase to the LLM (which I think is what code2prompt is doing) will eat up most tokens so you can't really get good documentation in 128k tokens. End of rant :)

[patched-admin]

@iuliaturc I would recommed using the Gemini models for this patchflow. But we'll take a look at the tokens too! The template patchflow was meant to be a naive implementation - we do have some avenues to compress the code2prompt output.

[iuliaturc]

@iuliaturc I would recommed using the Gemini models for this patchflow. But we'll take a look at the tokens too! The template patchflow was meant to be a naive implementation - we do have some avenues to compress the code2prompt output.

Nice, do you have any docs or official recommendations on how to compress the code2prompt output?