Problem / pain point
I am using local LLM's for confidentiality reasons. They are too slow for the timeouts offered in the GUI, 2 hours. I would like timeouts like 10 hours, or perhaps even unlimited. I usually run those generation jobs over-night, and I would prefer them to complete correctly than being stopped unnecessarily.
Proposed solution
Either add more timeout options or way to disable the timeout.
Alternatives considered
I could buy faster hardware, but since it will likely get old and too small too fast, I would prefer to use the hardware I have.
Scope
Generation capability
Hard-constraints check
Problem / pain point
I am using local LLM's for confidentiality reasons. They are too slow for the timeouts offered in the GUI, 2 hours. I would like timeouts like 10 hours, or perhaps even unlimited. I usually run those generation jobs over-night, and I would prefer them to complete correctly than being stopped unnecessarily.
Proposed solution
Either add more timeout options or way to disable the timeout.
Alternatives considered
I could buy faster hardware, but since it will likely get old and too small too fast, I would prefer to use the hardware I have.
Scope
Generation capability
Hard-constraints check