From 4929e34531d79c8451ec4d811780403ff529ae0f Mon Sep 17 00:00:00 2001 From: mvansegbroeck Date: Fri, 21 Aug 2026 12:29:29 -0700 Subject: [PATCH 1/2] docs: add Word document generation Colab notebook Adds docs/colab_notebooks/8-generating-word-documents.ipynb, a standalone notebook showing how to turn generated rows into real .docx files: one Pydantic model serving as both the LLM output_format and the renderer's input contract, sampler controls that label the corpus by construction, and python-docx rendering of headings, tables, footers, and Word core properties. Follows the 7-nemotron-personas pattern: a self-contained Colab notebook with an Open in Colab badge and the standard install/API-key setup cells, rather than a numbered entry in the docs/notebook_source tutorial series. It is referenced from an upcoming dev note. Because it is not part of docs/notebook_source, it is not executed by make test-run-tutorials and adds no dependency to the notebooks group; python-docx is installed by the notebook's own Colab setup cell. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: mvansegbroeck --- .../8-generating-word-documents.ipynb | 700 ++++++++++++++++++ 1 file changed, 700 insertions(+) create mode 100644 docs/colab_notebooks/8-generating-word-documents.ipynb diff --git a/docs/colab_notebooks/8-generating-word-documents.ipynb b/docs/colab_notebooks/8-generating-word-documents.ipynb new file mode 100644 index 000000000..1f3c9c4cc --- /dev/null +++ b/docs/colab_notebooks/8-generating-word-documents.ipynb @@ -0,0 +1,700 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "22bade12", + "metadata": { + "nemo_colab_inject": true + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "id": "a55d1f46", + "metadata": {}, + "source": [ + "# ๐Ÿ“„ Data Designer Tutorial: Generating Word Documents\n", + "\n", + "#### ๐Ÿ“š What you'll learn\n", + "\n", + "Your document pipeline probably does not ingest parquet. It ingests `.docx`.\n", + "\n", + "This notebook builds a corpus of synthetic corporate policy documents and writes each row out as a real\n", + "Word file โ€” the kind of dataset an enterprise RAG index, a document classifier, or a DLP scanner needs to\n", + "be developed against, and the kind you are never allowed to copy out of a customer's SharePoint.\n", + "\n", + "- ๐Ÿงฑ **Structure, not prose**: use a Pydantic model as the LLM's `output_format` so you never parse markdown\n", + "- ๐Ÿ’ฌ **Field descriptions are prompt text**: why `Field(description=...)` is the fastest quality lever you have\n", + "- ๐ŸŽฒ **Labelled by construction**: sampler controls become dataset columns, so no annotation pass is needed\n", + "- ๐Ÿ–จ๏ธ **Rendering**: turn each row into a `.docx` with headings, tables, footers, and Word core properties\n", + "\n", + "> **Prerequisites**: This notebook uses [build.nvidia.com](https://build.nvidia.com/models) and\n", + "> [`python-docx`](https://python-docx.readthedocs.io/). The setup cells below install the dependencies\n", + "> and pick up your API key.\n", + "\n", + "This notebook builds on the [structured outputs tutorial](https://docs.nvidia.com/nemo/datadesigner/tutorials/structured-outputs-jinja-expressions-and-conditional-generation).\n", + "If this is your first time using Data Designer, start with the\n", + "[first notebook](https://docs.nvidia.com/nemo/datadesigner/tutorials/the-basics) in this series.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1491d940", + "metadata": { + "nemo_colab_inject": true + }, + "source": [ + "### โšก Colab Setup\n", + "\n", + "Run the cells below to install the dependencies and set up the API key. If you don't have an API key, you can generate one from [build.nvidia.com](https://build.nvidia.com).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12fc6202", + "metadata": { + "nemo_colab_inject": true + }, + "outputs": [], + "source": [ + "%%capture\n", + "!pip install -U data-designer \"python-docx>=1.1.0,<2\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1eb4c54a", + "metadata": { + "nemo_colab_inject": true + }, + "outputs": [], + "source": [ + "import getpass\n", + "import os\n", + "\n", + "from google.colab import userdata\n", + "\n", + "try:\n", + " os.environ[\"NVIDIA_API_KEY\"] = userdata.get(\"NVIDIA_API_KEY\")\n", + "except userdata.SecretNotFoundError:\n", + " os.environ[\"NVIDIA_API_KEY\"] = getpass.getpass(\"Enter your NVIDIA API key: \")" + ] + }, + { + "cell_type": "markdown", + "id": "85eb8b50", + "metadata": {}, + "source": [ + "### ๐Ÿ“ฆ Import Data Designer\n", + "\n", + "- `data_designer.config` provides the configuration API.\n", + "- `DataDesigner` is the main interface for generation.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5737f406", + "metadata": {}, + "outputs": [], + "source": [ + "import data_designer.config as dd\n", + "from data_designer.interface import DataDesigner" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5780cfec", + "metadata": {}, + "outputs": [], + "source": [ + "data_designer = DataDesigner()" + ] + }, + { + "cell_type": "markdown", + "id": "c7fc592d", + "metadata": {}, + "source": [ + "### ๐ŸŽ›๏ธ Define model configurations\n", + "\n", + "- Writing a whole document in one structured call is a long generation, so `max_tokens` matters more here\n", + " than in most tutorials.\n", + "\n", + "- If the model runs out of tokens mid-JSON you do not get a short document, you get a parse failure on the\n", + " column. That is the good outcome: it fails loudly instead of silently truncating.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "95d79303", + "metadata": {}, + "outputs": [], + "source": [ + "MODEL_ALIAS = \"doc-writer\"\n", + "\n", + "model_configs = [\n", + " dd.ModelConfig(\n", + " alias=MODEL_ALIAS,\n", + " model=\"nvidia/nemotron-3-super-120b-a12b\",\n", + " provider=\"nvidia\",\n", + " inference_parameters=dd.ChatCompletionInferenceParams(\n", + " temperature=0.9,\n", + " top_p=0.95,\n", + " max_tokens=8192,\n", + " ),\n", + " )\n", + "]" + ] + }, + { + "cell_type": "markdown", + "id": "55d4090a", + "metadata": {}, + "source": [ + "## ๐Ÿงฑ Model the document, not the prose\n", + "\n", + "The tempting approach is to ask for \"a policy document\", get back a wall of markdown, and then write a\n", + "parser that hunts for `##` and pipe tables and turns them into Word styles.\n", + "\n", + "That parser is where the project dies. Models are inconsistent about markdown in exactly the ways that\n", + "break naive parsers, and every failure mode you fix creates a new regex.\n", + "\n", + "So we never generate prose-with-structure in the first place. We generate the **structure**, with prose\n", + "inside it.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c923c8e2", + "metadata": {}, + "outputs": [], + "source": [ + "from pydantic import BaseModel, Field\n", + "\n", + "\n", + "class DocTable(BaseModel):\n", + " \"\"\"A simple rectangular table.\"\"\"\n", + "\n", + " caption: str = Field(description=\"Short caption describing what the table contains.\")\n", + " columns: list[str] = Field(description=\"Column headers, 2 to 4 of them.\")\n", + " rows: list[list[str]] = Field(description=\"Table rows. Each row has one cell per column header.\")\n", + "\n", + "\n", + "class DocSection(BaseModel):\n", + " \"\"\"One numbered section of the document.\"\"\"\n", + "\n", + " heading: str = Field(description=\"Section heading, title case, no numbering prefix.\")\n", + " paragraphs: list[str] = Field(\n", + " description=\"One to three body paragraphs of prose. No markdown, no bullet characters.\"\n", + " )\n", + " bullets: list[str] = Field(\n", + " default_factory=list,\n", + " description=(\n", + " \"Optional bulleted requirements or steps for this section. Use an empty list when the \"\n", + " \"section reads better as prose only.\"\n", + " ),\n", + " )\n", + "\n", + "\n", + "class WordDocument(BaseModel):\n", + " \"\"\"A complete business document, structured for rendering.\"\"\"\n", + "\n", + " title: str = Field(description=\"Document title.\")\n", + " subtitle: str = Field(description=\"One-line subtitle, e.g. the scope or the owning function.\")\n", + " summary: str = Field(description=\"A single paragraph executive summary, 40-80 words.\")\n", + " sections: list[DocSection] = Field(description=\"Four to six sections that make up the body.\")\n", + " key_data: DocTable = Field(\n", + " description=(\n", + " \"A table carrying the document's structured facts โ€” thresholds, review cadences, \"\n", + " \"roles and responsibilities, retention windows, or similar.\"\n", + " )\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "8738f442", + "metadata": {}, + "source": [ + "This one model does two jobs. It is the `output_format` of an `LLMStructuredColumnConfig`, and it is the\n", + "input contract of the renderer we write further down.\n", + "\n", + "Because both ends share one definition, *\"the LLM produced something the renderer can't handle\"* stops\n", + "being a category of bug you defend against with heuristics. It becomes a Pydantic validation error on the\n", + "column, which Data Designer already knows how to retry.\n", + "\n", + "> ๐Ÿ’ก **Your field descriptions are prompt text**\n", + ">\n", + "> Data Designer serializes the JSON Schema into the prompt inside `` tags and asks for a\n", + "> fenced `json` block. Every `Field(description=...)` you write is shipped to the model verbatim.\n", + ">\n", + "> `description=\"Column headers, 2 to 4 of them.\"` is not documentation. It is the instruction that stops\n", + "> you getting nine-column tables. Write those descriptions like prompts, because that is what they are.\n" + ] + }, + { + "cell_type": "markdown", + "id": "21f7bc5d", + "metadata": {}, + "source": [ + "## ๐ŸŽฒ Design the corpus with samplers\n", + "\n", + "Every document will be joined to a row recording the controls that produced it. *\"Give me the Restricted\n", + "Finance documents\"* becomes a dataframe filter instead of an annotation project โ€” the labels are free\n", + "because they came first.\n", + "\n", + "Note `doc_type` is a **subcategory** of `department`, so a Finance document is a \"Revenue Recognition\n", + "Policy\" and never an \"Adverse Event Reporting Procedure\".\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1a18d68c", + "metadata": {}, + "outputs": [], + "source": [ + "DOC_TYPES_BY_DEPARTMENT = {\n", + " \"Information Security\": [\"Access Control Standard\", \"Incident Response Runbook\"],\n", + " \"Human Resources\": [\"Remote Work Policy\", \"Travel and Expense Policy\"],\n", + " \"Finance\": [\"Revenue Recognition Policy\", \"Month-End Close Runbook\"],\n", + " \"Procurement\": [\"Vendor Onboarding Procedure\", \"Contract Renewal Policy\"],\n", + "}\n", + "\n", + "config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"doc_id\",\n", + " sampler_type=dd.SamplerType.UUID,\n", + " params=dd.UUIDSamplerParams(prefix=\"POL-\", short_form=True, uppercase=True),\n", + " )\n", + ")\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"company\",\n", + " sampler_type=dd.SamplerType.CATEGORY,\n", + " params=dd.CategorySamplerParams(\n", + " values=[\"Northwind Diagnostics\", \"Cobalt Ridge Financial\", \"Halden Biopharma\"]\n", + " ),\n", + " )\n", + ")\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"department\",\n", + " sampler_type=dd.SamplerType.CATEGORY,\n", + " params=dd.CategorySamplerParams(values=list(DOC_TYPES_BY_DEPARTMENT)),\n", + " )\n", + ")\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"doc_type\",\n", + " sampler_type=dd.SamplerType.SUBCATEGORY,\n", + " params=dd.SubcategorySamplerParams(category=\"department\", values=DOC_TYPES_BY_DEPARTMENT),\n", + " )\n", + ")\n", + "\n", + "# Skewed on purpose: a realistic corpus is mostly Internal with a thin tail of\n", + "# Restricted documents, and that tail is usually the interesting test slice.\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"classification\",\n", + " sampler_type=dd.SamplerType.CATEGORY,\n", + " params=dd.CategorySamplerParams(\n", + " values=[\"Public\", \"Internal\", \"Confidential\", \"Restricted\"],\n", + " weights=[1, 6, 3, 1],\n", + " ),\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "8816a155", + "metadata": {}, + "source": [ + "## โœ๏ธ Generate the document\n", + "\n", + "One LLM call per document. Everything above was samplers, which are free.\n", + "\n", + "The `{% if %}` block is what makes `classification` more than a label sitting next to the document โ€” it\n", + "becomes a claim about the body that an extraction pipeline can be scored against.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "becf0183", + "metadata": {}, + "outputs": [], + "source": [ + "config_builder.add_column(\n", + " dd.LLMStructuredColumnConfig(\n", + " name=\"document\",\n", + " model_alias=MODEL_ALIAS,\n", + " output_format=WordDocument,\n", + " prompt=(\n", + " \"Write an internal {{ doc_type }} for {{ company }}, owned by the {{ department }} \"\n", + " \"department. The document ID is {{ doc_id }}.\\n\\n\"\n", + " \"Write in the flat, procedural register of a real corporate policy document. No marketing \"\n", + " \"language, no first person. Sections must be specific to a {{ doc_type }} โ€” scope, roles, the \"\n", + " \"actual procedure, exceptions, enforcement โ€” not filler like 'Introduction'.\\n\"\n", + " \"{% if classification in ['Confidential', 'Restricted'] %}\"\n", + " \"This document is {{ classification }}. State the handling restrictions explicitly in the \"\n", + " \"scope section.\\n\"\n", + " \"{% endif %}\"\n", + " \"The key_data table must carry concrete, checkable facts โ€” thresholds, timeframes, \"\n", + " \"role-to-responsibility mappings โ€” not prose chopped into cells.\\n\"\n", + " \"Body text is plain prose. No markdown, no '**', no bullet characters inside paragraphs.\"\n", + " ),\n", + " )\n", + ")\n", + "\n", + "data_designer.validate(config_builder)" + ] + }, + { + "cell_type": "markdown", + "id": "22bbd513", + "metadata": {}, + "source": [ + "### ๐Ÿ” Iteration is key โ€“ preview the dataset!\n", + "\n", + "Check that the sections are specific to the document type, that the table holds facts rather than chopped\n", + "up prose, and that the classification shows up in the scope section when it should.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ce7541ff", + "metadata": {}, + "outputs": [], + "source": [ + "preview = data_designer.preview(config_builder, num_records=2)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e4c532ad", + "metadata": {}, + "outputs": [], + "source": [ + "preview.display_sample_record()" + ] + }, + { + "cell_type": "markdown", + "id": "01d053ff", + "metadata": {}, + "source": [ + "## ๐Ÿ–จ๏ธ Render to `.docx`\n", + "\n", + "Now `python-docx` turns a `WordDocument` into a real Word file.\n", + "\n", + "Note what this function does **not** do: it does not parse anything. It walks a validated Pydantic object,\n", + "because all the ambiguity was removed one step earlier by the schema.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bc566e5b", + "metadata": {}, + "outputs": [], + "source": [ + "import re\n", + "from pathlib import Path\n", + "\n", + "from docx import Document\n", + "from docx.enum.text import WD_ALIGN_PARAGRAPH\n", + "from docx.shared import Pt\n", + "\n", + "UNSAFE_FILENAME_CHARS = re.compile(r\"[^A-Za-z0-9._-]+\")\n", + "\n", + "\n", + "def safe_filename(name: str) -> str:\n", + " \"\"\"Turn an arbitrary string into something safe to write to disk.\"\"\"\n", + " stem = UNSAFE_FILENAME_CHARS.sub(\"-\", name.strip()).strip(\"-._\")\n", + " return (stem[:120] or \"document\") + \".docx\"\n", + "\n", + "\n", + "def normalize_rows(table: DocTable) -> list[list[str]]:\n", + " \"\"\"Pad or truncate every row to the header width.\n", + "\n", + " Structured outputs constrain the *shape* of the JSON, not the arithmetic inside it. A model told to\n", + " produce a three-column table will occasionally hand back a row with two cells โ€” nothing in the JSON\n", + " Schema forbids it. Retrying costs a whole document generation; padding costs four lines.\n", + " \"\"\"\n", + " width = len(table.columns)\n", + " normalized = []\n", + " for row in table.rows:\n", + " cells = [str(cell) for cell in row][:width]\n", + " cells.extend([\"\"] * (width - len(cells)))\n", + " normalized.append(cells)\n", + " return normalized\n", + "\n", + "\n", + "def add_table(doc: Document, headers: list[str], rows: list[list[str]]) -> None:\n", + " \"\"\"Append a table with a bold header row.\"\"\"\n", + " table = doc.add_table(rows=1, cols=max(len(headers), 1))\n", + " table.style = \"Table Grid\"\n", + " for index, header in enumerate(headers):\n", + " table.rows[0].cells[index].text = str(header)\n", + " for paragraph in table.rows[0].cells[index].paragraphs:\n", + " for run in paragraph.runs:\n", + " run.bold = True\n", + " for row in rows:\n", + " cells = table.add_row().cells\n", + " for index, value in enumerate(row):\n", + " cells[index].text = value\n", + " doc.add_paragraph()\n", + "\n", + "\n", + "def render_document(\n", + " document: WordDocument,\n", + " output_path: str | Path,\n", + " metadata: dict[str, str] | None = None,\n", + " footer_text: str | None = None,\n", + " core_properties: dict[str, str] | None = None,\n", + ") -> Path:\n", + " \"\"\"Render a WordDocument to a .docx file.\"\"\"\n", + " output_path = Path(output_path)\n", + " output_path.parent.mkdir(parents=True, exist_ok=True)\n", + " doc = Document()\n", + "\n", + " doc.add_heading(document.title, level=0)\n", + " subtitle = doc.add_paragraph(document.subtitle)\n", + " for run in subtitle.runs:\n", + " run.italic = True\n", + " run.font.size = Pt(12)\n", + "\n", + " if metadata:\n", + " add_table(doc, [\"Field\", \"Value\"], [[key, value] for key, value in metadata.items()])\n", + "\n", + " doc.add_heading(\"Summary\", level=1)\n", + " doc.add_paragraph(document.summary)\n", + "\n", + " for index, section in enumerate(document.sections, start=1):\n", + " doc.add_heading(f\"{index}. {section.heading}\", level=1)\n", + " for paragraph in section.paragraphs:\n", + " doc.add_paragraph(paragraph)\n", + " for bullet in section.bullets:\n", + " doc.add_paragraph(bullet, style=\"List Bullet\")\n", + "\n", + " doc.add_heading(document.key_data.caption, level=2)\n", + " add_table(doc, document.key_data.columns, normalize_rows(document.key_data))\n", + "\n", + " if footer_text:\n", + " footer = doc.sections[0].footer.paragraphs[0]\n", + " footer.text = footer_text\n", + " footer.alignment = WD_ALIGN_PARAGRAPH.CENTER\n", + "\n", + " # Word core properties travel with the file, and plenty of enterprise\n", + " # tooling reads them. Filling them from columns costs nothing.\n", + " for key, value in (core_properties or {}).items():\n", + " if hasattr(doc.core_properties, key) and value is not None:\n", + " setattr(doc.core_properties, key, str(value))\n", + "\n", + " doc.save(str(output_path))\n", + " return output_path" + ] + }, + { + "cell_type": "markdown", + "id": "ad3d3ad0", + "metadata": {}, + "source": [ + "Structured columns come back as JSON, so we parse them straight into the model that produced them.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4b541746", + "metadata": {}, + "outputs": [], + "source": [ + "def render_row(row, output_dir: str | Path) -> Path:\n", + " \"\"\"Render one dataset row to a .docx file, carrying its metadata across.\"\"\"\n", + " value = row[\"document\"]\n", + " document = WordDocument.model_validate_json(value) if isinstance(value, str) else WordDocument.model_validate(value)\n", + " return render_document(\n", + " document,\n", + " Path(output_dir) / safe_filename(f\"{row['doc_id']}-{row['doc_type']}\"),\n", + " metadata={\n", + " \"Document ID\": row[\"doc_id\"],\n", + " \"Department\": row[\"department\"],\n", + " \"Classification\": row[\"classification\"],\n", + " },\n", + " footer_text=f\"{row['company']} ยท {row['classification']} ยท {row['doc_id']}\",\n", + " core_properties={\"author\": row[\"company\"], \"category\": row[\"doc_type\"]},\n", + " )\n", + "\n", + "\n", + "preview_path = render_row(preview.dataset.iloc[0], \"word-documents-preview\")\n", + "print(f\"๐Ÿ“„ {preview_path} ({preview_path.stat().st_size:,} bytes)\")" + ] + }, + { + "cell_type": "markdown", + "id": "db342664", + "metadata": {}, + "source": [ + "### ๐Ÿ‘€ Read it back\n", + "\n", + "Open the file in Word โ€” but also read it back programmatically, because that is how downstream pipelines\n", + "will see it. The headings, tables, footer, and core properties are all separately addressable.\n", + "\n", + "Those last two matter more than they look. Putting the classification label in the footer is realistic, and\n", + "it is precisely the case where a naive text extractor loses the label entirely. That is a test worth having.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fcaf6508", + "metadata": {}, + "outputs": [], + "source": [ + "rendered = Document(str(preview_path))\n", + "\n", + "for paragraph in rendered.paragraphs:\n", + " if paragraph.style.name.startswith((\"Title\", \"Heading\")):\n", + " print(f\"[{paragraph.style.name}] {paragraph.text}\")\n", + "\n", + "print(f\"\\nkey data {[cell.text for cell in rendered.tables[-1].rows[0].cells]}\")\n", + "print(f\"footer {rendered.sections[0].footer.paragraphs[0].text}\")\n", + "print(f\"core author {rendered.core_properties.author}\")" + ] + }, + { + "cell_type": "markdown", + "id": "99923008", + "metadata": {}, + "source": [ + "### ๐Ÿ†™ Scale up!\n", + "\n", + "Happy with the preview? Generate the corpus and render all of it.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "00bb452f", + "metadata": {}, + "outputs": [], + "source": [ + "results = data_designer.create(config_builder, num_records=3, dataset_name=\"tutorial-7-word-documents\")\n", + "\n", + "dataset = results.load_dataset()\n", + "dataset[\"docx_path\"] = [str(render_row(row, \"word-documents\")) for _, row in dataset.iterrows()]\n", + "\n", + "dataset[[\"doc_id\", \"company\", \"department\", \"doc_type\", \"classification\", \"docx_path\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "a3c64cb7", + "metadata": {}, + "source": [ + "### ๐Ÿ“Š The corpus is labelled by construction\n", + "\n", + "Every `.docx` has a row, every row has a `docx_path`, and both carry the sampler controls that produced\n", + "them. No annotation pass required โ€” the thin tail you actually wanted to test against is a filter away.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "406ef703", + "metadata": {}, + "outputs": [], + "source": [ + "dataset[dataset[\"classification\"].isin([\"Confidential\", \"Restricted\"])][\n", + " [\"doc_id\", \"department\", \"doc_type\", \"classification\"]\n", + "]" + ] + }, + { + "cell_type": "markdown", + "id": "be442f23", + "metadata": {}, + "source": [ + "## ๐Ÿ”Œ Outgrowing the notebook\n", + "\n", + "The loop above is fine for a notebook. It is not fine for 50,000 documents, because it only starts after\n", + "the entire dataset has finished generating and it lives outside the Data Designer config.\n", + "\n", + "The production shape is a [processor plugin](https://docs.nvidia.com/nemo/datadesigner/concepts/processors),\n", + "and you do not have to write one โ€” this renderer is packaged as\n", + "[`data-designer-docx`](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins/data-designer-docx)\n", + "in the NeMo Data Designer Plugins repository. Once installed, rendering happens *inside* the pipeline:\n", + "\n", + "```python\n", + "from data_designer_docx.config import DocxProcessorConfig\n", + "\n", + "config_builder.add_processor(\n", + " DocxProcessorConfig(\n", + " name=\"word-documents\",\n", + " document_column=\"document\",\n", + " filename_template=\"{{ doc_id }}-{{ doc_type }}.docx\",\n", + " metadata_columns={\"Document ID\": \"{{ doc_id }}\", \"Classification\": \"{{ classification }}\"},\n", + " footer_template=\"{{ company }} ยท {{ classification }} ยท {{ doc_id }}\",\n", + " )\n", + ")\n", + "```\n", + "\n", + "Files then stream out as each batch completes rather than after everything finishes, the path lands back in\n", + "the dataset automatically, and the rendering rules travel with the config instead of living in a cell\n", + "someone has to remember to run.\n" + ] + }, + { + "cell_type": "markdown", + "id": "dc78106d", + "metadata": {}, + "source": [ + "## โญ๏ธ Next Steps\n", + "\n", + "- [Processors](https://docs.nvidia.com/nemo/datadesigner/concepts/processors) โ€” what runs at which stage\n", + "\n", + "- [Plugins overview](https://docs.nvidia.com/nemo/datadesigner/plugins/overview) โ€” writing your own\n", + "\n", + "- The pattern generalizes: swap `python-docx` for `python-pptx` and you generate slide decks; swap it for a\n", + " PDF renderer and, combined with [image columns](https://docs.nvidia.com/nemo/datadesigner/tutorials/generating-images),\n", + " you get the scanned-document corpora that VLM document-understanding work runs on. The structure stays,\n", + " only the renderer changes.\n" + ] + } + ], + "metadata": { + "jupytext": { + "text_representation": { + "extension": ".py", + "format_name": "percent", + "format_version": "1.3", + "jupytext_version": "1.18.1" + } + }, + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 35f580ef6b87cd63a7b1050bf70359fc08288f81 Mon Sep 17 00:00:00 2001 From: mvansegbroeck Date: Fri, 21 Aug 2026 12:45:11 -0700 Subject: [PATCH 2/2] docs: use the docx processor plugin in the notebook Replaces the self-contained renderer with the data-designer-docx plugin, so the notebook demonstrates the production shape rather than an inline loop over the finished dataframe. The notebook now installs the plugin, verifies Data Designer discovered it through its entry point, and adds a single DocxProcessorConfig processor. The WordDocument schema is imported from the plugin instead of being redefined, since the schema is the contract between the LLM and the renderer and the two halves belong in the same package. Rendering happens inside the pipeline: files stream out per batch, and the document path is written back into the dataset automatically. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: mvansegbroeck --- .../8-generating-word-documents.ipynb | 597 +++++------------- 1 file changed, 155 insertions(+), 442 deletions(-) diff --git a/docs/colab_notebooks/8-generating-word-documents.ipynb b/docs/colab_notebooks/8-generating-word-documents.ipynb index 1f3c9c4cc..c22ff0ad6 100644 --- a/docs/colab_notebooks/8-generating-word-documents.ipynb +++ b/docs/colab_notebooks/8-generating-word-documents.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "22bade12", + "id": "b8f6ce69", "metadata": { "nemo_colab_inject": true }, @@ -12,62 +12,53 @@ }, { "cell_type": "markdown", - "id": "a55d1f46", + "id": "0a432241", "metadata": {}, "source": [ - "# ๐Ÿ“„ Data Designer Tutorial: Generating Word Documents\n", + "# ๐Ÿ“„ Word Documents with Data Designer\n", "\n", - "#### ๐Ÿ“š What you'll learn\n", + "Data Designer generates rows. Your document pipeline wants `.docx` files.\n", "\n", - "Your document pipeline probably does not ingest parquet. It ingests `.docx`.\n", + "The `data-designer-docx` plugin closes that gap. You describe the dataset as usual, add one processor,\n", + "and every row comes out as a real Word document โ€” headings, tables, footers, the lot.\n", "\n", - "This notebook builds a corpus of synthetic corporate policy documents and writes each row out as a real\n", - "Word file โ€” the kind of dataset an enterprise RAG index, a document classifier, or a DLP scanner needs to\n", - "be developed against, and the kind you are never allowed to copy out of a customer's SharePoint.\n", - "\n", - "- ๐Ÿงฑ **Structure, not prose**: use a Pydantic model as the LLM's `output_format` so you never parse markdown\n", - "- ๐Ÿ’ฌ **Field descriptions are prompt text**: why `Field(description=...)` is the fastest quality lever you have\n", - "- ๐ŸŽฒ **Labelled by construction**: sampler controls become dataset columns, so no annotation pass is needed\n", - "- ๐Ÿ–จ๏ธ **Rendering**: turn each row into a `.docx` with headings, tables, footers, and Word core properties\n", - "\n", - "> **Prerequisites**: This notebook uses [build.nvidia.com](https://build.nvidia.com/models) and\n", - "> [`python-docx`](https://python-docx.readthedocs.io/). The setup cells below install the dependencies\n", - "> and pick up your API key.\n", - "\n", - "This notebook builds on the [structured outputs tutorial](https://docs.nvidia.com/nemo/datadesigner/tutorials/structured-outputs-jinja-expressions-and-conditional-generation).\n", - "If this is your first time using Data Designer, start with the\n", - "[first notebook](https://docs.nvidia.com/nemo/datadesigner/tutorials/the-basics) in this series.\n" + "Install it, then seven short steps. Let's go.\n" ] }, { "cell_type": "markdown", - "id": "1491d940", + "id": "68b80543", "metadata": { "nemo_colab_inject": true }, "source": [ "### โšก Colab Setup\n", "\n", - "Run the cells below to install the dependencies and set up the API key. If you don't have an API key, you can generate one from [build.nvidia.com](https://build.nvidia.com).\n" + "Run the cells below to install the dependencies and set up the API key. If you don't have an API key, you can generate one from [build.nvidia.com](https://build.nvidia.com).\n", + "\n", + "This notebook uses the [`data-designer-docx`](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins/data-designer-docx) plugin, which Data Designer discovers automatically once it is installed.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "12fc6202", + "id": "13a45cd5", "metadata": { "nemo_colab_inject": true }, "outputs": [], "source": [ "%%capture\n", - "!pip install -U data-designer \"python-docx>=1.1.0,<2\"" + "# data-designer-docx is not on PyPI yet; install it from the plugins repository.\n", + "# Once it is released this becomes: !pip install -U data-designer data-designer-docx\n", + "!pip install -U data-designer \\\n", + " \"data-designer-docx @ git+https://github.com/NVIDIA-NeMo/DataDesignerPlugins.git#subdirectory=plugins/data-designer-docx\"" ] }, { "cell_type": "code", "execution_count": null, - "id": "1eb4c54a", + "id": "60ff3499", "metadata": { "nemo_colab_inject": true }, @@ -86,190 +77,117 @@ }, { "cell_type": "markdown", - "id": "85eb8b50", + "id": "314e4f40", "metadata": {}, "source": [ - "### ๐Ÿ“ฆ Import Data Designer\n", + "## Step 0 ยท About the plugin\n", "\n", - "- `data_designer.config` provides the configuration API.\n", - "- `DataDesigner` is the main interface for generation.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5737f406", - "metadata": {}, - "outputs": [], - "source": [ - "import data_designer.config as dd\n", - "from data_designer.interface import DataDesigner" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5780cfec", - "metadata": {}, - "outputs": [], - "source": [ - "data_designer = DataDesigner()" + "The `.docx` writer is not part of Data Designer. It is a separate plugin,\n", + "[`data-designer-docx`](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins/data-designer-docx),\n", + "published in the NeMo Data Designer Plugins repository โ€” the setup cell above already installed it.\n", + "\n", + "Two things to know about plugin installs:\n", + "\n", + "- **Nothing else is needed.** No registration call, no config entry. The package declares a\n", + " `data_designer.plugins` entry point and Data Designer discovers it automatically.\n", + "\n", + "- **Install before the kernel imports Data Designer.** Entry points are read from installed package\n", + " metadata at import time, so a plugin installed into an already-running kernel is not seen until you\n", + " restart the runtime.\n", + "\n", + "Running locally instead of in Colab? Same install, any package manager:\n", + "\n", + "```bash\n", + "pip install data-designer-docx\n", + "```\n" ] }, { "cell_type": "markdown", - "id": "c7fc592d", + "id": "d788df2c", "metadata": {}, "source": [ - "### ๐ŸŽ›๏ธ Define model configurations\n", - "\n", - "- Writing a whole document in one structured call is a long generation, so `max_tokens` matters more here\n", - " than in most tutorials.\n", + "## Step 1 ยท Check the plugin is there\n", "\n", - "- If the model runs out of tokens mid-JSON you do not get a short document, you get a parse failure on the\n", - " column. That is the good outcome: it fails loudly instead of silently truncating.\n" + "Data Designer finds plugins through installed package metadata, so this is really a check that the install\n", + "worked. If the list comes back empty, re-run the setup cell and restart the runtime.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "95d79303", + "id": "1e0fabb8", "metadata": {}, "outputs": [], "source": [ - "MODEL_ALIAS = \"doc-writer\"\n", + "from data_designer.plugins.plugin import PluginType\n", + "from data_designer.plugins.registry import PluginRegistry\n", "\n", - "model_configs = [\n", - " dd.ModelConfig(\n", - " alias=MODEL_ALIAS,\n", - " model=\"nvidia/nemotron-3-super-120b-a12b\",\n", - " provider=\"nvidia\",\n", - " inference_parameters=dd.ChatCompletionInferenceParams(\n", - " temperature=0.9,\n", - " top_p=0.95,\n", - " max_tokens=8192,\n", - " ),\n", - " )\n", - "]" + "print(\"processor plugins:\", PluginRegistry().get_plugin_names(PluginType.PROCESSOR))" ] }, { "cell_type": "markdown", - "id": "55d4090a", + "id": "a25d9f71", "metadata": {}, "source": [ - "## ๐Ÿงฑ Model the document, not the prose\n", - "\n", - "The tempting approach is to ask for \"a policy document\", get back a wall of markdown, and then write a\n", - "parser that hunts for `##` and pipe tables and turns them into Word styles.\n", + "## Step 2 ยท Imports and your API key\n", "\n", - "That parser is where the project dies. Models are inconsistent about markdown in exactly the ways that\n", - "break naive parsers, and every failure mode you fix creates a new regex.\n", + "`WordDocument` comes from the plugin. It describes the *shape* of a document โ€” title, summary, sections,\n", + "a table โ€” and we will hand it straight to the LLM in a moment.\n", "\n", - "So we never generate prose-with-structure in the first place. We generate the **structure**, with prose\n", - "inside it.\n" + "Note where it comes from: the schema ships **with the plugin**, because it is the contract between the\n", + "LLM and the renderer, and the two halves belong in the same package.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "c923c8e2", + "id": "3fa1a329", "metadata": {}, "outputs": [], "source": [ - "from pydantic import BaseModel, Field\n", - "\n", - "\n", - "class DocTable(BaseModel):\n", - " \"\"\"A simple rectangular table.\"\"\"\n", - "\n", - " caption: str = Field(description=\"Short caption describing what the table contains.\")\n", - " columns: list[str] = Field(description=\"Column headers, 2 to 4 of them.\")\n", - " rows: list[list[str]] = Field(description=\"Table rows. Each row has one cell per column header.\")\n", - "\n", - "\n", - "class DocSection(BaseModel):\n", - " \"\"\"One numbered section of the document.\"\"\"\n", - "\n", - " heading: str = Field(description=\"Section heading, title case, no numbering prefix.\")\n", - " paragraphs: list[str] = Field(\n", - " description=\"One to three body paragraphs of prose. No markdown, no bullet characters.\"\n", - " )\n", - " bullets: list[str] = Field(\n", - " default_factory=list,\n", - " description=(\n", - " \"Optional bulleted requirements or steps for this section. Use an empty list when the \"\n", - " \"section reads better as prose only.\"\n", - " ),\n", - " )\n", - "\n", + "import data_designer.config as dd\n", + "from data_designer.interface import DataDesigner\n", "\n", - "class WordDocument(BaseModel):\n", - " \"\"\"A complete business document, structured for rendering.\"\"\"\n", + "from data_designer_docx.config import DocxProcessorConfig\n", + "from data_designer_docx.schema import WordDocument\n", "\n", - " title: str = Field(description=\"Document title.\")\n", - " subtitle: str = Field(description=\"One-line subtitle, e.g. the scope or the owning function.\")\n", - " summary: str = Field(description=\"A single paragraph executive summary, 40-80 words.\")\n", - " sections: list[DocSection] = Field(description=\"Four to six sections that make up the body.\")\n", - " key_data: DocTable = Field(\n", - " description=(\n", - " \"A table carrying the document's structured facts โ€” thresholds, review cadences, \"\n", - " \"roles and responsibilities, retention windows, or similar.\"\n", - " )\n", - " )" + "data_designer = DataDesigner(artifact_path=\"./artifacts\")" ] }, { "cell_type": "markdown", - "id": "8738f442", + "id": "cd9b0103", "metadata": {}, "source": [ - "This one model does two jobs. It is the `output_format` of an `LLMStructuredColumnConfig`, and it is the\n", - "input contract of the renderer we write further down.\n", - "\n", - "Because both ends share one definition, *\"the LLM produced something the renderer can't handle\"* stops\n", - "being a category of bug you defend against with heuristics. It becomes a Pydantic validation error on the\n", - "column, which Data Designer already knows how to retry.\n", - "\n", - "> ๐Ÿ’ก **Your field descriptions are prompt text**\n", - ">\n", - "> Data Designer serializes the JSON Schema into the prompt inside `` tags and asks for a\n", - "> fenced `json` block. Every `Field(description=...)` you write is shipped to the model verbatim.\n", - ">\n", - "> `description=\"Column headers, 2 to 4 of them.\"` is not documentation. It is the instruction that stops\n", - "> you getting nine-column tables. Write those descriptions like prompts, because that is what they are.\n" - ] - }, - { - "cell_type": "markdown", - "id": "21f7bc5d", - "metadata": {}, - "source": [ - "## ๐ŸŽฒ Design the corpus with samplers\n", - "\n", - "Every document will be joined to a row recording the controls that produced it. *\"Give me the Restricted\n", - "Finance documents\"* becomes a dataframe filter instead of an annotation project โ€” the labels are free\n", - "because they came first.\n", + "## Step 3 ยท Say what documents you want\n", "\n", - "Note `doc_type` is a **subcategory** of `department`, so a Finance document is a \"Revenue Recognition\n", - "Policy\" and never an \"Adverse Event Reporting Procedure\".\n" + "Three samplers. They give every document an ID, a company, and a type โ€” and because those values live in\n", + "the dataset, your finished corpus is labelled without you annotating anything.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "1a18d68c", + "id": "a34bd475", "metadata": {}, "outputs": [], "source": [ - "DOC_TYPES_BY_DEPARTMENT = {\n", - " \"Information Security\": [\"Access Control Standard\", \"Incident Response Runbook\"],\n", - " \"Human Resources\": [\"Remote Work Policy\", \"Travel and Expense Policy\"],\n", - " \"Finance\": [\"Revenue Recognition Policy\", \"Month-End Close Runbook\"],\n", - " \"Procurement\": [\"Vendor Onboarding Procedure\", \"Contract Renewal Policy\"],\n", - "}\n", - "\n", - "config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)\n", + "MODEL_ALIAS = \"doc-writer\"\n", + "MODEL_ID = \"nvidia/nemotron-3-super-120b-a12b\"\n", + "\n", + "config_builder = dd.DataDesignerConfigBuilder(\n", + " model_configs=[\n", + " dd.ModelConfig(\n", + " alias=MODEL_ALIAS,\n", + " model=MODEL_ID,\n", + " provider=\"nvidia\",\n", + " # A whole document in one call is a long generation. Give it room.\n", + " inference_parameters=dd.ChatCompletionInferenceParams(temperature=0.9, max_tokens=8192),\n", + " )\n", + " ]\n", + ")\n", "\n", "config_builder.add_column(\n", " dd.SamplerColumnConfig(\n", @@ -291,29 +209,15 @@ "\n", "config_builder.add_column(\n", " dd.SamplerColumnConfig(\n", - " name=\"department\",\n", - " sampler_type=dd.SamplerType.CATEGORY,\n", - " params=dd.CategorySamplerParams(values=list(DOC_TYPES_BY_DEPARTMENT)),\n", - " )\n", - ")\n", - "\n", - "config_builder.add_column(\n", - " dd.SamplerColumnConfig(\n", " name=\"doc_type\",\n", - " sampler_type=dd.SamplerType.SUBCATEGORY,\n", - " params=dd.SubcategorySamplerParams(category=\"department\", values=DOC_TYPES_BY_DEPARTMENT),\n", - " )\n", - ")\n", - "\n", - "# Skewed on purpose: a realistic corpus is mostly Internal with a thin tail of\n", - "# Restricted documents, and that tail is usually the interesting test slice.\n", - "config_builder.add_column(\n", - " dd.SamplerColumnConfig(\n", - " name=\"classification\",\n", " sampler_type=dd.SamplerType.CATEGORY,\n", " params=dd.CategorySamplerParams(\n", - " values=[\"Public\", \"Internal\", \"Confidential\", \"Restricted\"],\n", - " weights=[1, 6, 3, 1],\n", + " values=[\n", + " \"Remote Work Policy\",\n", + " \"Incident Response Runbook\",\n", + " \"Vendor Onboarding Procedure\",\n", + " \"Travel and Expense Policy\",\n", + " ]\n", " ),\n", " )\n", ")" @@ -321,21 +225,21 @@ }, { "cell_type": "markdown", - "id": "8816a155", + "id": "65672fa0", "metadata": {}, "source": [ - "## โœ๏ธ Generate the document\n", + "## Step 4 ยท Write the document\n", "\n", - "One LLM call per document. Everything above was samplers, which are free.\n", + "One LLM column does the writing. The important bit is `output_format=WordDocument`: instead of asking for\n", + "prose and parsing it afterwards, we ask for the document's *structure* and let the model fill it in.\n", "\n", - "The `{% if %}` block is what makes `classification` more than a label sitting next to the document โ€” it\n", - "becomes a claim about the body that an extraction pipeline can be scored against.\n" + "That is why no `.docx` parsing code appears anywhere in this notebook.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "becf0183", + "id": "14db56bf", "metadata": {}, "outputs": [], "source": [ @@ -345,338 +249,147 @@ " model_alias=MODEL_ALIAS,\n", " output_format=WordDocument,\n", " prompt=(\n", - " \"Write an internal {{ doc_type }} for {{ company }}, owned by the {{ department }} \"\n", - " \"department. The document ID is {{ doc_id }}.\\n\\n\"\n", - " \"Write in the flat, procedural register of a real corporate policy document. No marketing \"\n", - " \"language, no first person. Sections must be specific to a {{ doc_type }} โ€” scope, roles, the \"\n", - " \"actual procedure, exceptions, enforcement โ€” not filler like 'Introduction'.\\n\"\n", - " \"{% if classification in ['Confidential', 'Restricted'] %}\"\n", - " \"This document is {{ classification }}. State the handling restrictions explicitly in the \"\n", - " \"scope section.\\n\"\n", - " \"{% endif %}\"\n", - " \"The key_data table must carry concrete, checkable facts โ€” thresholds, timeframes, \"\n", - " \"role-to-responsibility mappings โ€” not prose chopped into cells.\\n\"\n", - " \"Body text is plain prose. No markdown, no '**', no bullet characters inside paragraphs.\"\n", + " \"Write an internal {{ doc_type }} for {{ company }} (document ID {{ doc_id }}).\\n\\n\"\n", + " \"Write like a real corporate policy: flat, procedural, no marketing language. \"\n", + " \"Sections should be specific to a {{ doc_type }} โ€” scope, roles, the actual procedure, \"\n", + " \"exceptions โ€” not filler like 'Introduction'. \"\n", + " \"The key_data table should hold concrete facts: thresholds, timeframes, who does what.\"\n", " ),\n", " )\n", - ")\n", - "\n", - "data_designer.validate(config_builder)" - ] - }, - { - "cell_type": "markdown", - "id": "22bbd513", - "metadata": {}, - "source": [ - "### ๐Ÿ” Iteration is key โ€“ preview the dataset!\n", - "\n", - "Check that the sections are specific to the document type, that the table holds facts rather than chopped\n", - "up prose, and that the classification shows up in the scope section when it should.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ce7541ff", - "metadata": {}, - "outputs": [], - "source": [ - "preview = data_designer.preview(config_builder, num_records=2)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e4c532ad", - "metadata": {}, - "outputs": [], - "source": [ - "preview.display_sample_record()" + ")" ] }, { "cell_type": "markdown", - "id": "01d053ff", + "id": "669ebb4f", "metadata": {}, "source": [ - "## ๐Ÿ–จ๏ธ Render to `.docx`\n", + "## Step 5 ยท Turn each row into a `.docx`\n", "\n", - "Now `python-docx` turns a `WordDocument` into a real Word file.\n", + "Here it is. One processor, and the pipeline now writes Word files.\n", "\n", - "Note what this function does **not** do: it does not parse anything. It walks a validated Pydantic object,\n", - "because all the ambiguity was removed one step earlier by the schema.\n" + "The templates are rendered per row, so anything in the dataset can go into the filename, the front-matter\n", + "table, or the footer.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "bc566e5b", + "id": "a9c22674", "metadata": {}, "outputs": [], "source": [ - "import re\n", - "from pathlib import Path\n", - "\n", - "from docx import Document\n", - "from docx.enum.text import WD_ALIGN_PARAGRAPH\n", - "from docx.shared import Pt\n", - "\n", - "UNSAFE_FILENAME_CHARS = re.compile(r\"[^A-Za-z0-9._-]+\")\n", - "\n", - "\n", - "def safe_filename(name: str) -> str:\n", - " \"\"\"Turn an arbitrary string into something safe to write to disk.\"\"\"\n", - " stem = UNSAFE_FILENAME_CHARS.sub(\"-\", name.strip()).strip(\"-._\")\n", - " return (stem[:120] or \"document\") + \".docx\"\n", - "\n", - "\n", - "def normalize_rows(table: DocTable) -> list[list[str]]:\n", - " \"\"\"Pad or truncate every row to the header width.\n", - "\n", - " Structured outputs constrain the *shape* of the JSON, not the arithmetic inside it. A model told to\n", - " produce a three-column table will occasionally hand back a row with two cells โ€” nothing in the JSON\n", - " Schema forbids it. Retrying costs a whole document generation; padding costs four lines.\n", - " \"\"\"\n", - " width = len(table.columns)\n", - " normalized = []\n", - " for row in table.rows:\n", - " cells = [str(cell) for cell in row][:width]\n", - " cells.extend([\"\"] * (width - len(cells)))\n", - " normalized.append(cells)\n", - " return normalized\n", - "\n", - "\n", - "def add_table(doc: Document, headers: list[str], rows: list[list[str]]) -> None:\n", - " \"\"\"Append a table with a bold header row.\"\"\"\n", - " table = doc.add_table(rows=1, cols=max(len(headers), 1))\n", - " table.style = \"Table Grid\"\n", - " for index, header in enumerate(headers):\n", - " table.rows[0].cells[index].text = str(header)\n", - " for paragraph in table.rows[0].cells[index].paragraphs:\n", - " for run in paragraph.runs:\n", - " run.bold = True\n", - " for row in rows:\n", - " cells = table.add_row().cells\n", - " for index, value in enumerate(row):\n", - " cells[index].text = value\n", - " doc.add_paragraph()\n", - "\n", - "\n", - "def render_document(\n", - " document: WordDocument,\n", - " output_path: str | Path,\n", - " metadata: dict[str, str] | None = None,\n", - " footer_text: str | None = None,\n", - " core_properties: dict[str, str] | None = None,\n", - ") -> Path:\n", - " \"\"\"Render a WordDocument to a .docx file.\"\"\"\n", - " output_path = Path(output_path)\n", - " output_path.parent.mkdir(parents=True, exist_ok=True)\n", - " doc = Document()\n", - "\n", - " doc.add_heading(document.title, level=0)\n", - " subtitle = doc.add_paragraph(document.subtitle)\n", - " for run in subtitle.runs:\n", - " run.italic = True\n", - " run.font.size = Pt(12)\n", - "\n", - " if metadata:\n", - " add_table(doc, [\"Field\", \"Value\"], [[key, value] for key, value in metadata.items()])\n", - "\n", - " doc.add_heading(\"Summary\", level=1)\n", - " doc.add_paragraph(document.summary)\n", - "\n", - " for index, section in enumerate(document.sections, start=1):\n", - " doc.add_heading(f\"{index}. {section.heading}\", level=1)\n", - " for paragraph in section.paragraphs:\n", - " doc.add_paragraph(paragraph)\n", - " for bullet in section.bullets:\n", - " doc.add_paragraph(bullet, style=\"List Bullet\")\n", - "\n", - " doc.add_heading(document.key_data.caption, level=2)\n", - " add_table(doc, document.key_data.columns, normalize_rows(document.key_data))\n", - "\n", - " if footer_text:\n", - " footer = doc.sections[0].footer.paragraphs[0]\n", - " footer.text = footer_text\n", - " footer.alignment = WD_ALIGN_PARAGRAPH.CENTER\n", - "\n", - " # Word core properties travel with the file, and plenty of enterprise\n", - " # tooling reads them. Filling them from columns costs nothing.\n", - " for key, value in (core_properties or {}).items():\n", - " if hasattr(doc.core_properties, key) and value is not None:\n", - " setattr(doc.core_properties, key, str(value))\n", - "\n", - " doc.save(str(output_path))\n", - " return output_path" - ] - }, - { - "cell_type": "markdown", - "id": "ad3d3ad0", - "metadata": {}, - "source": [ - "Structured columns come back as JSON, so we parse them straight into the model that produced them.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "4b541746", - "metadata": {}, - "outputs": [], - "source": [ - "def render_row(row, output_dir: str | Path) -> Path:\n", - " \"\"\"Render one dataset row to a .docx file, carrying its metadata across.\"\"\"\n", - " value = row[\"document\"]\n", - " document = WordDocument.model_validate_json(value) if isinstance(value, str) else WordDocument.model_validate(value)\n", - " return render_document(\n", - " document,\n", - " Path(output_dir) / safe_filename(f\"{row['doc_id']}-{row['doc_type']}\"),\n", - " metadata={\n", - " \"Document ID\": row[\"doc_id\"],\n", - " \"Department\": row[\"department\"],\n", - " \"Classification\": row[\"classification\"],\n", - " },\n", - " footer_text=f\"{row['company']} ยท {row['classification']} ยท {row['doc_id']}\",\n", - " core_properties={\"author\": row[\"company\"], \"category\": row[\"doc_type\"]},\n", + "config_builder.add_processor(\n", + " DocxProcessorConfig(\n", + " name=\"word-documents\",\n", + " document_column=\"document\",\n", + " filename_template=\"{{ doc_id }}-{{ doc_type }}.docx\",\n", + " metadata_columns={\"Document ID\": \"{{ doc_id }}\", \"Company\": \"{{ company }}\"},\n", + " footer_template=\"{{ company }} ยท {{ doc_id }}\",\n", " )\n", + ")\n", "\n", - "\n", - "preview_path = render_row(preview.dataset.iloc[0], \"word-documents-preview\")\n", - "print(f\"๐Ÿ“„ {preview_path} ({preview_path.stat().st_size:,} bytes)\")" + "data_designer.validate(config_builder)" ] }, { "cell_type": "markdown", - "id": "db342664", + "id": "f4fd809e", "metadata": {}, "source": [ - "### ๐Ÿ‘€ Read it back\n", + "## Step 6 ยท Preview one, then make ten\n", "\n", - "Open the file in Word โ€” but also read it back programmatically, because that is how downstream pipelines\n", - "will see it. The headings, tables, footer, and core properties are all separately addressable.\n", + "Always preview first. It is one document and one API call, and it tells you whether the prompt is landing\n", + "before you pay for a hundred.\n", "\n", - "Those last two matter more than they look. Putting the classification label in the footer is realistic, and\n", - "it is precisely the case where a naive text extractor loses the label entirely. That is a test worth having.\n" + "Notice `docx_path` in the output โ€” the processor writes it back into the dataset, so rows and files stay\n", + "joined.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "fcaf6508", + "id": "3bb1ec42", "metadata": {}, "outputs": [], "source": [ - "rendered = Document(str(preview_path))\n", + "preview = data_designer.preview(config_builder, num_records=1)\n", "\n", - "for paragraph in rendered.paragraphs:\n", - " if paragraph.style.name.startswith((\"Title\", \"Heading\")):\n", - " print(f\"[{paragraph.style.name}] {paragraph.text}\")\n", - "\n", - "print(f\"\\nkey data {[cell.text for cell in rendered.tables[-1].rows[0].cells]}\")\n", - "print(f\"footer {rendered.sections[0].footer.paragraphs[0].text}\")\n", - "print(f\"core author {rendered.core_properties.author}\")" - ] - }, - { - "cell_type": "markdown", - "id": "99923008", - "metadata": {}, - "source": [ - "### ๐Ÿ†™ Scale up!\n", - "\n", - "Happy with the preview? Generate the corpus and render all of it.\n" + "preview.dataset[[\"doc_id\", \"company\", \"doc_type\", \"docx_path\"]]" ] }, { "cell_type": "code", "execution_count": null, - "id": "00bb452f", + "id": "b84da9dc", "metadata": {}, "outputs": [], "source": [ - "results = data_designer.create(config_builder, num_records=3, dataset_name=\"tutorial-7-word-documents\")\n", + "results = data_designer.create(config_builder, num_records=10, dataset_name=\"policy-documents-plugin\")\n", "\n", "dataset = results.load_dataset()\n", - "dataset[\"docx_path\"] = [str(render_row(row, \"word-documents\")) for _, row in dataset.iterrows()]\n", - "\n", - "dataset[[\"doc_id\", \"company\", \"department\", \"doc_type\", \"classification\", \"docx_path\"]]" + "dataset[[\"doc_id\", \"company\", \"doc_type\", \"docx_path\"]]" ] }, { "cell_type": "markdown", - "id": "a3c64cb7", + "id": "c0c4a19a", "metadata": {}, "source": [ - "### ๐Ÿ“Š The corpus is labelled by construction\n", + "## Step 7 ยท Open one\n", "\n", - "Every `.docx` has a row, every row has a `docx_path`, and both carry the sampler controls that produced\n", - "them. No annotation pass required โ€” the thin tail you actually wanted to test against is a filter away.\n" + "`docx_path` is relative to the dataset folder, so the whole thing stays portable. Double-click the file to\n", + "open it in Word โ€” or read it back here, the way a downstream pipeline would.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "406ef703", + "id": "0cf4d398", "metadata": {}, "outputs": [], "source": [ - "dataset[dataset[\"classification\"].isin([\"Confidential\", \"Restricted\"])][\n", - " [\"doc_id\", \"department\", \"doc_type\", \"classification\"]\n", - "]" + "from docx import Document\n", + "\n", + "document_path = results.artifact_storage.base_dataset_path / dataset[\"docx_path\"].iloc[0]\n", + "rendered = Document(str(document_path))\n", + "\n", + "print(f\"{document_path.name}\\n\")\n", + "for paragraph in rendered.paragraphs:\n", + " if paragraph.style.name.startswith((\"Title\", \"Heading\")):\n", + " print(f\" {paragraph.text}\")\n", + "\n", + "print(f\"\\nfooter: {rendered.sections[0].footer.paragraphs[0].text}\")\n", + "print(f\"folder: {document_path.parent}\")" ] }, { "cell_type": "markdown", - "id": "be442f23", + "id": "04b8dd6e", "metadata": {}, "source": [ - "## ๐Ÿ”Œ Outgrowing the notebook\n", + "## That's it ๐ŸŽ‰\n", "\n", - "The loop above is fine for a notebook. It is not fine for 50,000 documents, because it only starts after\n", - "the entire dataset has finished generating and it lives outside the Data Designer config.\n", + "Ten real Word documents, and a dataset that knows which row produced which file.\n", "\n", - "The production shape is a [processor plugin](https://docs.nvidia.com/nemo/datadesigner/concepts/processors),\n", - "and you do not have to write one โ€” this renderer is packaged as\n", - "[`data-designer-docx`](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins/data-designer-docx)\n", - "in the NeMo Data Designer Plugins repository. Once installed, rendering happens *inside* the pipeline:\n", - "\n", - "```python\n", - "from data_designer_docx.config import DocxProcessorConfig\n", + "Things you can change from here, all in that one `DocxProcessorConfig`:\n", "\n", - "config_builder.add_processor(\n", - " DocxProcessorConfig(\n", - " name=\"word-documents\",\n", - " document_column=\"document\",\n", - " filename_template=\"{{ doc_id }}-{{ doc_type }}.docx\",\n", - " metadata_columns={\"Document ID\": \"{{ doc_id }}\", \"Classification\": \"{{ classification }}\"},\n", - " footer_template=\"{{ company }} ยท {{ classification }} ยท {{ doc_id }}\",\n", - " )\n", - ")\n", - "```\n", + "| Want to... | Use |\n", + "| --- | --- |\n", + "| Apply your corporate styles | `template_path=\"brand/corporate-template.docx\"` |\n", + "| Fill in Word's author/category fields | `core_property_columns={\"author\": \"{{ owner }}\"}` |\n", + "| Change where files land | `output_subdir=\"documents\"` |\n", + "| Drop the `1.` `2.` numbering | `number_sections=False` |\n", "\n", - "Files then stream out as each batch completes rather than after everything finishes, the path lands back in\n", - "the dataset automatically, and the rendering rules travel with the config instead of living in a cell\n", - "someone has to remember to run.\n" - ] - }, - { - "cell_type": "markdown", - "id": "dc78106d", - "metadata": {}, - "source": [ - "## โญ๏ธ Next Steps\n", + "And when you want more control over the documents themselves, edit `WordDocument` in\n", + "`src/data_designer_docx/schema.py`. The field descriptions in that file are sent to the model as part of\n", + "the prompt, so they are the fastest lever you have.\n", "\n", - "- [Processors](https://docs.nvidia.com/nemo/datadesigner/concepts/processors) โ€” what runs at which stage\n", + "**Next:**\n", "\n", "- [Plugins overview](https://docs.nvidia.com/nemo/datadesigner/plugins/overview) โ€” writing your own\n", "\n", - "- The pattern generalizes: swap `python-docx` for `python-pptx` and you generate slide decks; swap it for a\n", - " PDF renderer and, combined with [image columns](https://docs.nvidia.com/nemo/datadesigner/tutorials/generating-images),\n", - " you get the scanned-document corpora that VLM document-understanding work runs on. The structure stays,\n", - " only the renderer changes.\n" + "- [Processors](https://docs.nvidia.com/nemo/datadesigner/concepts/processors) โ€” the built-in ones\n" ] } ], @@ -690,9 +403,9 @@ } }, "kernelspec": { - "display_name": ".venv", + "display_name": "Word Docs (data-designer)", "language": "python", - "name": "python3" + "name": "word-docs" } }, "nbformat": 4,