Describe the bug
When I stop a streaming chat, llama.cpp continues to generate tokens (and may take a long time to finish the response). It looks like Zoo is not properly canceling the stream / closing the connection to the backend.
To Reproduce
Steps to reproduce the behavior:
- Ask Zoo a question that leads to a long answer or thinking process
- Stop after some time (e.g. to edit the question and ask again)
- See the backend still generating (and possibly blocking when no other slot is free)
Expected behavior
Stopped responses cancel the corresponding stream
What version of zoo are you running
Version: 3.60.0 (a62a652)
Describe the bug
When I stop a streaming chat, llama.cpp continues to generate tokens (and may take a long time to finish the response). It looks like Zoo is not properly canceling the stream / closing the connection to the backend.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Stopped responses cancel the corresponding stream
What version of zoo are you running
Version: 3.60.0 (a62a652)