feat(jobs): filter job listing by namespace and add job deletion

[SusannaShu]

Why

A multi-tenant proxy (our keyboard-backend) fans every end user's uploads into a single Knowhere account, separated only by namespace. Two gaps made that proxy unable to own its own view of a user's files:

• GET /v1/jobs had no way to scope results to one namespace, so the proxy had to keep its own job-id list and issue one GET /v1/jobs/{id} per file.
• There was no way to remove a job at all. DELETE /v1/jobs/{id} returned 500 Method not allowed, so callers had no delete path and had to hide jobs client-side, which meant they reappeared on the next sync.

What changed

Namespace filtering

GET /v1/jobs, GET /v1/jobs/page, and the v2 equivalents accept a namespace query parameter, matched against job_metadata ->> 'namespace'.

Three behaviors worth calling out:

• The filter is normalized the same way creation normalizes it. Job creation runs the namespace through normalize_retrieval_namespace, so an unnormalized filter value would silently match zero rows. ?namespace= and ?namespace=%20%20 now mean the default namespace.
• Omitting the parameter still lists every namespace. Absent means "no filter", not "default".
• Jobs predating the metadata key are treated as default. Rows whose job_metadata has no namespace key are matched by the default filter rather than being invisible to every query.

This also fixes a latent break in the same code path: JobReadModel.list_jobs_for_user did not accept namespace, so read_service forwarding it raised TypeError on every list call. That method now accepts and forwards it.

Job deletion

DELETE /v1/jobs/{job_id} and DELETE /v2/jobs/{job_id}.

Deletion is soft, via a new jobs.deleted_at column:

• The row is retained, so in-flight workers, billing records, and state audit history are untouched. Deleting a running job is safe — the worker finishes normally, the job is simply no longer listed.
• Soft-deleted jobs are invisible to GET /jobs, GET /jobs/{id}, and confirm-upload. check_job_permission raises NotFound after the ownership check, so deletion never leaks whether a job exists.
• Deleting twice returns 404 the second time, same as an unknown job.

The linked document is archived by default so the content leaves retrieval, with ?archive_document=false to opt out. Archiving is skipped when another live job still targets the same document, so re-parsing a document and then deleting the older job does not wipe the current content.

Response:

{
  "job_id": "job_abc",
  "deleted": true,
  "document_id": "doc_xyz",
  "document_archived": true
}

Indexes

Two partial indexes for the new query shapes, both WHERE deleted_at IS NULL:

• idx_job_user_active_created_at on (user_id, created_at) — the default listing.
• idx_job_user_namespace on (user_id, (job_metadata ->> 'namespace')) — namespace-scoped listing.

Migration

fce1d2e3f4a5, on top of fbe1c2d3e4f5. Adds the nullable deleted_at column and the two indexes. No backfill needed — existing rows are active by definition.

The repo already had two alembic heads before this change (f0d85d209e68 and fbe1c2d3e4f5); this branch does not change that count, and tests/deploys use upgrade heads. Worth a separate cleanup, out of scope here.

Tests

• tests/contract/test_job_delete_contract.py — delete hides the job from GET and list, other jobs survive, double delete is 404, unknown job is 404, archive_document=false is honored.
• tests/contract/test_job_read_contract.py — namespace filter isolates tenants, omitting it lists everything, blank means default.
• tests/migrations/test_schema_contract.py — deleted_at and both indexes exist after upgrade heads.

Verification status

make lint and make typecheck both pass clean. Route registration, the generated namespace SQL, and the alembic revision graph were verified directly.

The contract tests were not executed locally — this machine has no libpq or Postgres server binaries, so pytest_postgresql cannot start, and that blocks collection of the whole suite (pre-existing, not introduced here). They need a CI run to confirm.

The delete service's control flow was exercised against stubbed repository and document services, covering all four paths: archive, skip-archive-when-shared, already-deleted → NotFound, and wrong-owner → PermissionDenied.

Compatibility

Additive. New optional query parameter, new endpoint, new nullable column. No existing response shape or default behavior changes.

🤖 Generated with Claude Code

[suguanYang]

Thanks for explaining the keyboard-backend use case. We understand the underlying requirement: the backend needs to list the files belonging to one tenant and remove a file from that tenant’s document collection without maintaining its own list of Knowhere job IDs.

However, We do not think Job is the correct resource for this behavior.

In Knowhere, a job represents a document-processing attempt. It is part of the permanent operational ledger and preserves processing state, failures, billing, timing, provenance, and audit history. It should remain available even after the resulting document is archived or deleted. Soft-deleting a job still hides part of that history and conflates two different lifecycles.

namespace is meaningful for the persisted document corpus, not for managing processing attempts. A job may record the namespace as immutable provenance, but clients should not use GET /jobs?namespace=... as their document-listing interface.

We believe the requested client workflow should use the document resources instead:

• GET /documents?namespace=... to list the tenant’s processed documents.
• DELETE /documents/{document_id} or the existing archive operation to remove a document from the active corpus and retrieval.
• GET /jobs/{job_id} only to poll or inspect a specific processing attempt.
• If processing history is needed, expose it under the document, for example GET /documents/{document_id}/jobs or /revisions.

For uploads that have not produced a document yet, keyboard-backend should retain the returned job ID while processing is in progress. If it needs durable reconciliation of pending submissions, we should
design that explicitly—possibly around the existing external data_id—rather than turning the job ledger into a file registry.

Therefore, We agree with the business requirement, but We do not think namespace-filtered job listing or job deletion is the right interface. Could you revise the change around the document model and
describe any remaining pending-upload reconciliation requirement separately?

[gdccyuen]

Agree, please reject this PR, thanks.