LCORE-3048 : okp_rag e2e test

.konflux/artifacts.lock.yaml

@@ -3,6 +3,6 @@ metadata:
   version: "1.0"
 
 artifacts:
-  - download_url: https://github.com/lightspeed-core/lightspeed-providers/archive/8cd1b3d3bdd841ea99d31b334ae00a275581661c.zip
-    checksum: sha256:6e660a30babcd1c00eef1686b251f9368410e03e08a7c0d4a680db00693ac448
+  - download_url: https://github.com/lightspeed-core/lightspeed-providers/archive/c095177e7d6d58a4009556251bbbca96ebf3dcb4.zip
+    checksum: sha256:2146f5b1d4850c437db024498ef4ec2a49ea760a413d25269ca4641f11f4b093
     filename: lightspeed-providers.zip

.tekton/integration-tests/pipeline/lightspeed-stack-integration-test.yaml

@@ -212,6 +212,10 @@ spec:
           - name: quay-aipcc-password
             secret:
               secretName: quay-aipcc-password
+          # Red Hat registry pull secret (docker-registry type) - will be copied to test namespace
+          - name: redhat-registry-pull-secret-source
+            secret:
+              secretName: redhat-registry-io-pull-secret
           - name: credentials
             emptyDir: {}
         steps:
@@ -251,6 +255,9 @@ spec:
                 mountPath: /var/run/quay-aipcc-name
               - name: quay-aipcc-password
                 mountPath: /var/run/quay-aipcc-password
+              # Red Hat registry pull secret (docker-registry type)
+              - name: redhat-registry-pull-secret-source
+                mountPath: /var/run/redhat-registry-pull-secret
               - name: credentials
                 mountPath: /credentials
             env:

Makefile

@@ -1,6 +1,9 @@
 SHELL := /bin/bash
 #TODO: We need to rename all those python and config files as well
 
+# Define comma for use in $(if) expressions (where commas are argument separators)
+COMMA := ,
+
 ARTIFACT_DIR := $(if $(ARTIFACT_DIR),$(ARTIFACT_DIR),tests/test_results)
 PATH_TO_PLANTUML := ~/bin
 
@@ -16,6 +19,7 @@ OGX_CONFIG ?= run.yaml
 OGX_CONTAINER_NAME ?= lightspeed-ogx
 OGX_IMAGE ?= lightspeed-ogx:local
 OGX_PORT ?= 8321
+LIGHTSPEED_PROVIDERS_DIR ?= $(shell [ -d ../lightspeed-providers ] && cd ../lightspeed-providers && pwd)
 CONTAINER_RUNTIME ?= $(shell command -v podman 2>/dev/null || command -v docker 2>/dev/null)
 
 # Doc tools configuration
@@ -89,6 +93,9 @@ start-ogx-container: build-ogx-image ## Start OGX container
 		-v $(PWD)/$(CONFIG):/opt/app-root/lightspeed-stack.yaml:ro,z \
 		-v $(PWD)/scripts/ogx-entrypoint.sh:/opt/app-root/enrich-entrypoint.sh:ro,z \
 		-v $(PWD)/src/ogx_configuration.py:/opt/app-root/ogx_configuration.py:ro,z \
+		$(if $(LIGHTSPEED_PROVIDERS_DIR),-v $(LIGHTSPEED_PROVIDERS_DIR)/lightspeed_stack_providers:/opt/app-root/providers/lightspeed_stack_providers:ro$(COMMA)z) \
+		$(if $(LIGHTSPEED_PROVIDERS_DIR),-v $(LIGHTSPEED_PROVIDERS_DIR)/resources/external_providers:/opt/app-root/src/.llama/providers.d:ro$(COMMA)z) \
+		$(if $(LIGHTSPEED_PROVIDERS_DIR),-e EXTERNAL_PROVIDERS_DIR=/opt/app-root/src/.llama/providers.d) \
 		-e OPENAI_API_KEY \
 		-e BRAVE_SEARCH_API_KEY \
 		-e TAVILY_SEARCH_API_KEY \

deploy/lightspeed-stack/Containerfile

@@ -37,7 +37,7 @@ COPY ${LSC_SOURCE_DIR}/pyproject.toml ${LSC_SOURCE_DIR}/LICENSE ${LSC_SOURCE_DIR
 
 # lightspeed-providers:
 # Fully hermetic — uses prefetched artifact or pinned commit from GitHub
-ARG LIGHTSPEED_PROVIDERS_COMMIT=faf6a89a3ad7856e2e7a934324f31d146108acdb
+ARG LIGHTSPEED_PROVIDERS_COMMIT=c095177e7d6d58a4009556251bbbca96ebf3dcb4
 RUN set -eux; \
     ZIP_PATH="/tmp/lightspeed-providers.zip"; \
     EXTRACT_DIR="/tmp/providers"; \

docs/testing/e2e_scenarios.md

@@ -115,6 +115,20 @@
 * Check if models can be filtered
 * Check if filtering can return empty list of models
 
+## [`okp_rag.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/okp_rag.feature)
+
+Konflux only (`@konflux-only`) in CI. Targeted local Behave (match YAML to OGX/LCS): [Running OKP RAG tests locally](e2e_testing.md#running-okp-rag-tests-locally).
+
+* Offline mode query with inline RAG returns rag_chunks and referenced_documents
+* Online mode streaming query with inline RAG returns referenced_documents
+* Query with inline RAG with dynamic semantic filter returns rag_chunks and referenced_documents
+* Offline query API with OKP tool RAG has rag_chunk and referenced_documents returned
+* Online responses API with OKP tool RAG has rag results returned
+* Query succeeds with empty rag_chunks when OKP server is unavailable
+* Streaming query succeeds with empty referenced_documents when OKP server is unavailable
+* Query returns no rag_chunks and no referenced_documents when OKP is disabled
+* Streaming query returns no referenced_documents when OKP is disabled
+
 ## [`proxy.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/proxy.feature)
 
 * LLM traffic is routed through a configured tunnel proxy

docs/testing/e2e_testing.md

@@ -9,15 +9,16 @@ This guide describes how to run, extend, and understand the Lightspeed Core Stac
 1. [Overview](#overview)
 2. [Directory Layout](#directory-layout)
 3. [How to Run E2E Tests](#how-to-run-e2e-tests)
-4. [Environment Variables](#environment-variables)
-5. [Deployment Modes: Server vs Library](#deployment-modes-server-vs-library)
-6. [Tags and Hooks](#tags-and-hooks)
-7. [Configuration Files](#configuration-files)
-8. [Feature Files and Steps](#feature-files-and-steps)
-9. [Gherkin Keywords in Feature Files](#gherkin-keywords-in-feature-files)
-10. [Choosing the Test Layer: E2E or Integration?](#choosing-the-test-layer-e2e-or-integration)
-11. [Writing New Scenarios](#writing-new-scenarios)
-12. [Troubleshooting](#troubleshooting)
+4. [Running OKP RAG tests locally](#running-okp-rag-tests-locally)
+5. [Environment Variables](#environment-variables)
+6. [Deployment Modes: Server vs Library](#deployment-modes-server-vs-library)
+7. [Tags and Hooks](#tags-and-hooks)
+8. [Configuration Files](#configuration-files)
+9. [Feature Files and Steps](#feature-files-and-steps)
+10. [Gherkin Keywords in Feature Files](#gherkin-keywords-in-feature-files)
+11. [Choosing the Test Layer: E2E or Integration?](#choosing-the-test-layer-e2e-or-integration)
+12. [Writing New Scenarios](#writing-new-scenarios)
+13. [Troubleshooting](#troubleshooting)
 
 ---
 
@@ -136,6 +137,73 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
 
 ---
 
+## Running OKP RAG tests locally
+
+`okp_rag.feature` is `@konflux-only`. CI deploys OKP as a pod; `make test-e2e` skips the feature. Do not set `E2E_KONFLUX_E2E=1` locally (that path is Kubernetes). On a laptop, start OKP in Docker, enrich and run OGX and LCS as host processes, then run **one scenario that matches the YAML you started**.
+
+Needs `registry.redhat.io` login, `OPENAI_API_KEY`, `uv sync --locked --group ogxlibdev`, and `../lightspeed-providers`.
+
+### Prerequisite: OKP in Docker
+
+```bash
+docker login registry.redhat.io
+docker run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
+```
+
+Wait until Solr answers (not only the portal page):
+
+```bash
+curl -sS -m 15 -o /dev/null -w "%{http_code}\n" \
+  'http://localhost:8081/solr/portal-rag/select?q=*:*&rows=0'
+```
+
+### 1. Create the enriched run file
+
+Use the same Lightspeed YAML the scenario will load (`-c`). Example: offline inline RAG.
+
+```bash
+export PYTHONPATH="$(cd ../lightspeed-providers && pwd)${PYTHONPATH:+:$PYTHONPATH}"
+export EXTERNAL_PROVIDERS_DIR="$(cd ../lightspeed-providers && pwd)/resources/external_providers"
+export RH_SERVER_OKP=http://localhost:8081/solr
+
+uv run python src/ogx_configuration.py \
+  -c tests/e2e/configuration/server-mode/lightspeed-stack-okp-offline.yaml \
+  -i run.yaml \
+  -o run_enriched.yaml
+```
+
+### 2. Start OGX
+
+```bash
+export PYTHONPATH="$(cd ../lightspeed-providers && pwd)${PYTHONPATH:+:$PYTHONPATH}"
+export EXTERNAL_PROVIDERS_DIR="$(cd ../lightspeed-providers && pwd)/resources/external_providers"
+
+uv run ogx stack run run_enriched.yaml --port 8321
+```
+
+### 3. Start LCS
+
+Same `CONFIG` as `-c` in step 1. `make run-ogx` starts LCS only (`CONFIG=`, not `make -c`).
+
+```bash
+export RH_SERVER_OKP=http://localhost:8081/solr
+export E2E_LLAMA_HOSTNAME=localhost
+LIGHTSPEED_STACK_LOG_LEVEL=DEBUG make run-ogx \
+  CONFIG=tests/e2e/configuration/server-mode/lightspeed-stack-okp-offline.yaml
+```
+
+### 4. Run the matching Behave scenario
+
+`okp_rag.feature:24` is the offline inline scenario (`lightspeed-stack-okp-offline.yaml`). Comment `@konflux-only` on the Feature, `OKP(Solr) server is running` in Background, and `OGX is restarted` / `The service is restarted` on that scenario (OGX and LCS are already up). Do not commit those comments.
+
+```bash
+uv run behave tests/e2e/features/okp_rag.feature:24
+```
+
+For another YAML (`lightspeed-stack-okp-online.yaml`, tool RAG, and so on), repeat steps 1–3 with that file, then run the scenario line that uses it. Product setup: [OKP guide](../user_doc/okp_guide.md).
+
+---
+
 ## Environment Variables
 
 | Variable                        | Default     | Description                                                                                                                               |
@@ -151,6 +219,8 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
 | `E2E_DEFAULT_PROVIDER_OVERRIDE` | —           | Override default provider id (e.g. `openai`).                                                                                             |
 | `FAISS_VECTOR_STORE_ID`         | —           | Vector store id for FAISS-related scenarios.                                                                                              |
 | `RUNNING_PROW`                  | —           | Set in Prow/OpenShift; enables Prow config paths and pod/container ops.                                                                   |
+| `E2E_KONFLUX_E2E`               | —           | `1` in Konflux only. Unskips `@konflux-only` and deploys OKP as a pod.                                                                     |
+| `RH_SERVER_OKP`                 | —           | OKP/Solr URL (local default `http://localhost:8081/solr`).                                                                                |
 | `OPENAI_API_KEY`                | —           | **Required.** Used by the app and OGX for LLM calls (e.g. OpenAI). The E2E tests and the stack will not run correctly without it. |
 
 
@@ -176,6 +246,8 @@ All tag behaviour is implemented in **`features/environment.py`**: the hooks (`b
 | Tag                             | Effect                                                                                                                                                  |
 |---------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `@skip`                         | Scenario is skipped (reason: "Marked with @skip"). Use for broken or WIP scenarios.                                                                     |
+| `@konflux-only`                 | Skipped unless `E2E_KONFLUX_E2E=1`. Used by `okp_rag.feature`.                                                                           |
+| `@cfg_okp`                      | OKP Solr RAG. Konflux deploys OKP in `before_feature`. Local: [Running OKP RAG tests locally](#running-okp-rag-tests-locally).            |
 | `@skip-in-library-mode`         | Scenario is skipped when `E2E_DEPLOYMENT_MODE=library`. Used for tests that require a separate OGX (e.g. connection disruption).                |
 | `@local`                        | Skipped unless running in "local" mode (context flag).                                                                                                  |
 | `@InvalidFeedbackStorageConfig` | Before scenario: switch to invalid-feedback-storage config and restart container. After: restore feature config and restart.                            |
@@ -246,6 +318,7 @@ The feature files below are run in the order given in `tests/e2e/test_list.txt`:
 | `rest_api.feature`               | REST API: OpenAPI endpoint.                                                                                                             |
 | `mcp.feature`                    | MCP (Model Context Protocol): tools, query, streaming_query with MCP auth (required, token, invalid token).                             |
 | `models.feature`                 | Models endpoint: list models, filter, empty result; error when OGX unreachable.                                                 |
+| `okp_rag.feature`                | OKP Solr RAG (`@konflux-only`). Local: [Running OKP RAG tests locally](#running-okp-rag-tests-locally).                                  |
 
 
 If you add a new feature file, add it to **`tests/e2e/test_list.txt`** so it is included when you run the full E2E suite (e.g. `make test-e2e`). The order in that file is the run order.
@@ -396,6 +469,6 @@ The integration side of this boundary is described in
 - **Readonly database (SQLite) in OGX**: If the RAG KV DB is on a bind-mounted path that becomes read-only (e.g. after restart), move it to a named volume (e.g. via `KV_RAG_PATH` in docker-compose) so writes succeed.
 - **ChunkedEncodingError on streaming_query**: The step for streaming_query uses `stream=True` and consumes the stream; if you add new streaming steps, avoid reading the full response with `response.content` and use the same stream-reading pattern so a server close after an error event does not raise.
 - **Event loop is closed (httpx/AsyncClient)**: In E2E, any code that creates an `AsyncOgxClient` (e.g. for shields) must close it (e.g. `await client.close()`) in a `finally` block before the event loop is torn down (e.g. before `asyncio.run()` returns).
-- **Scenarios skipped**: Check tags (`@skip`, `@skip-in-library-mode`, `@local`) and `E2E_DEPLOYMENT_MODE`; ensure the scenario is not excluded by `--tags=-skip` (or the opposite if you intend to run only skipped scenarios for debugging).
+- **Scenarios skipped**: Check tags (`@skip`, `@skip-in-library-mode`, `@local`, `@konflux-only`) and `E2E_DEPLOYMENT_MODE`; ensure the scenario is not excluded by `--tags=-skip` (or the opposite if you intend to run only skipped scenarios for debugging).
 
 For more on test structure and commands, see the main project guide (`CLAUDE.md`) and `tests/e2e/features/steps/README.md`.

docs/user_doc/okp_guide.md

@@ -51,12 +51,22 @@ validating that queries return referenced chunks.
 
 ## Step 1: Launch OKP
 
-Start the OKP RAG service with Podman:
+Start the OKP RAG service with Podman or Docker:
+
+**Using Podman:**
 
 ```bash
+podman login registry.redhat.io
 podman run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
 ```
 
+**Using Docker:**
+
+```bash
+docker login registry.redhat.io
+docker run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
+```
+
 > **Note:** Remove `-d` to run in the foreground.
 
 * The service listens on **port 8081** on the host (mapped from 8080 in the container).  Lightspeed Stack itself listens on `8080`, so this avoids port conflicts.
@@ -105,41 +115,61 @@ Edit your Lightspeed Stack config file (e.g. `lightspeed-stack.yaml`) and add
 the following top-level sections so that OKP is used for either inline or tool
 RAG:
 
-Inline RAG:
+**Inline RAG:**
 
 ```yaml
 # RAG configuration
 rag:
-  inline:
-  - okp
-okp:
-  rhokp_url: ${env.RH_SERVER_OKP}
-  offline: true
+  retrieval:
+    inline:
+      sources:
+        - okp
+      max_chunks: 3      # Final tight cap on chunks returned to user
+  okp:
+    rhokp_url: ${env.RH_SERVER_OKP}
+    offline: true
+    max_chunks: 5        # Initial generous fetch from OKP
 ```
 
-Tool RAG:
+**Tool RAG:**
 
 ```yaml
 # RAG configuration
 rag:
-  tool:
-  - okp
-okp:
-  rhokp_url: ${env.RH_SERVER_OKP}
-  offline: true
+  retrieval:
+    tool:
+      sources:
+        - okp
+      max_chunks: 10     # Tool RAG limit
+  okp:
+    rhokp_url: ${env.RH_SERVER_OKP}
+    offline: true
+    max_chunks: 5
 ```
 
-* **`rag.inline`** and **`rag.tool`**: Enable OKP as the RAG source for inline context injection and for the RAG tool.  Tool rag means the LLM will be provided a search tool it can choose to invoke to find relevant content and augment the user prompt.  The tool may or may not be invoked.  Inline means a rag search and prompt augmentation will always occur.
+* **`rag.retrieval.inline`** and **`rag.retrieval.tool`**: Enable OKP as the RAG source for inline context injection and for the RAG tool. Tool RAG means the LLM will be provided a search tool it can choose to invoke to find relevant content and augment the user prompt. The tool may or may not be invoked. Inline RAG means a RAG search and prompt augmentation will always occur.
 * **`okp.offline`**: When `true`, source URLs use `parent_id` (offline/Mimir-style). When `false`, use `reference_url` (online).
+* **`okp.search_mode`**: Default Solr search strategy for OKP queries. Controls *how* Solr ranks chunks, not *which* documents are eligible (that is `chunk_filter_query` and per-request `solr.filters`). Allowed values:
+  - **`hybrid`** (default when the field is omitted): combines vector similarity with keyword (BM25) text search. Requires an embedding model. This is the global default.
+  - **`semantic`**: pure vector search against Solr embeddings. Requires an embedding model. Best when the query is conceptual rather than matching document wording.
+  - **`keyword`**: BM25 text search only. No embedding model is needed, so it is the right choice for air-gapped or embedding-free deployments, and it is typically faster than hybrid.
+* **`max_chunks` behavior**:
+  - **Inline RAG**: `rag.retrieval.inline.max_chunks` is the final tight cap on chunks returned in the response, while `rag.okp.max_chunks` is the initial generous fetch from OKP before filtering and ranking.
+  - **Tool RAG**: `rag.retrieval.tool.max_chunks` sets the limit on chunks available to the tool.
 
 If you want to filter the docs to a specific product, you can include a static query filter such as:
 
 ```yaml
 okp:
   offline: true
+  search_mode: keyword
   chunk_filter_query: "product:*openshift* AND product_version:4.21"
 ```
 
+A per-request `solr.mode` on `/v1/query` (or streaming/responses) overrides this YAML default for that call. Per-request `lexical` is treated as `keyword`.
+
+> **Important:** When running OGX in **Server mode**, changes to static OKP settings such as `chunk_filter_query` and `search_mode` require an OGX restart to take effect. They are applied when Lightspeed Stack enriches OGX configuration at startup and will not propagate automatically during runtime.
+
 When you launch Lightspeed Stack it will augment the OGX configuration (the synthesized run.yaml in unified mode, or your external run.yaml in the deprecated legacy mode) with
 configuration for OKP.
 
@@ -258,7 +288,7 @@ curl -sX POST http://localhost:8080/v1/query \
 - **Static filters preserved:** The configured `chunk_filter_query` (e.g., `"product:*openshift*"`) is always applied
 - **Dynamic filters added:** Request filters are combined with static filters using AND logic
 - **String escaping:** Special Solr characters in filter values are automatically escaped
-- **Works with all search modes:** Filters apply to `semantic`, `hybrid`, and `lexical` search modes
+- **Works with all search modes:** Filters apply whether the search mode is `semantic`, `hybrid`, or `keyword` (per-request `lexical` is an alias of `keyword`).
 
 ---
 
@@ -312,13 +342,15 @@ Example response excerpt:
 ],
 ```
 
-> **Note:** The first time you query the system the response may take
-> additional time because it must first download the necessary embedding model
-> to perform the vector search.
+> **Note:** The first time you query the system, `hybrid` or `semantic` search may take
+> additional time because OGX must download the embedding model used for vector
+> search. `keyword` search skips that download.
 
 If you see no RAG context, verify:
 
 1. OKP is up at http://localhost:8081
 2. `lightspeed-stack.yaml` has `okp` under `rag.inline` and/or `rag.tool` as in Step 4
 
+The Behave suite `okp_rag.feature` (`@cfg_okp`) runs in Konflux in CI. To stand up OKP + OGX + LCS on a laptop and run a matching scenario, see [Running OKP RAG tests locally](../testing/e2e_testing.md#running-okp-rag-tests-locally).
+
 ---