You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: scripts/user-config/README.md
+22Lines changed: 22 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,6 +34,28 @@ All failures are collected with their source and reason into the `IngestionResul
34
34
- Printed in the **INGESTION COMPLETE** banner (so you can see what failed and why at a glance)
35
35
- Returned as JSON from `POST /api/v1/data/load-references` for programmatic inspection
36
36
37
+
## Git incremental ingestion (optional)
38
+
39
+
You can ingest **only files that changed** in a git work tree since the last run, instead of scanning the whole directory every time.
40
+
41
+
- In your profile YAML, set **`guide.git-ingestion.enabled: true`** and **`guide.git-ingestion.state-file`** to a JSON path (often under **`scripts/user-config/`** so it stays local and gitignored if you prefer).
42
+
- The runner records the **git HEAD** per configured **`guide.directories`** entry. If HEAD is unchanged, that directory is skipped on the next startup ingest.
43
+
- To **force a full re-ingest** of one directory (for example after a bad partial run), remove that directory’s entry from the state file, or call **`POST /api/v1/data/git-ingestion/revision/reset`** with a JSON body **`{ "directory": "~/path/to/repo" }`** while Guide is running (same path style as in YAML). See **`scripts/README.md`** for curl examples and security notes.
44
+
45
+
## Shared Neo4j (Embabel Hub or custom Bolt)
46
+
47
+
Ingestion scripts can target **compose Neo4j** (default) or **another Bolt endpoint** (Hub, remote, etc.):
48
+
49
+
-**`append-ingest.sh`** is for **adding** content; use **`USE_EMBABEL_HUB_NEO4J=1`** for the common Hub layout (Bolt on **27687**, credentials preset in the script header). Override **`NEO4J_BOLT_PORT`**, **`NEO4J_PASSWORD_HUB`**, etc. if your setup differs.
50
+
-**`fresh-ingest.sh`****wipes**`ContentElement` data in the connected database. Do **not** point it at a **shared Hub** or production graph.
51
+
- For MCP against the **same** Bolt DB as ingest, you can use **`scripts/run-mcp-guide-against-hub.sh`** (see **`scripts/README.md`**).
52
+
53
+
Embabel Hub and the default Guide stack both use **384-dimensional** local ONNX embeddings (`all-MiniLM-L6-v2`) for vector indexes; mixing embedding models against one graph requires a deliberate re-index / re-ingest strategy (see operator docs in **`scripts/README.md`**).
54
+
55
+
## Operator API (purge and resync)
56
+
57
+
With Guide running, you can **preview or delete** ingested chunks by **URI prefix** or by **directory** (resolved to a `file:` prefix), and reset **git** revision state for one directory. Endpoints are **`POST /api/v1/data/content-elements/purge-preview`**, **`.../purge`**, and **`.../git-ingestion/revision/reset`**. Full table, examples, and safety notes (**prefix length**, **`confirm: true`**) are in **`scripts/README.md`** under **Operator API (shared Neo4j)**.
58
+
37
59
## MCP tools
38
60
39
61
All ingested content -- both URLs and local directories -- is immediately available through the MCP tools (`docs_vectorSearch`, `docs_textSearch`, etc.). The MCP tools and the ingestion pipeline share the same Neo4j store, so there is no separate sync step. Once ingestion completes, MCP clients (Cursor, Claude Desktop, etc.) can search the content right away.
0 commit comments