diff --git a/docs/colab_notebooks/8-generating-word-documents.ipynb b/docs/colab_notebooks/8-generating-word-documents.ipynb new file mode 100644 index 000000000..c22ff0ad6 --- /dev/null +++ b/docs/colab_notebooks/8-generating-word-documents.ipynb @@ -0,0 +1,413 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b8f6ce69", + "metadata": { + "nemo_colab_inject": true + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "id": "0a432241", + "metadata": {}, + "source": [ + "# 📄 Word Documents with Data Designer\n", + "\n", + "Data Designer generates rows. Your document pipeline wants `.docx` files.\n", + "\n", + "The `data-designer-docx` plugin closes that gap. You describe the dataset as usual, add one processor,\n", + "and every row comes out as a real Word document — headings, tables, footers, the lot.\n", + "\n", + "Install it, then seven short steps. Let's go.\n" + ] + }, + { + "cell_type": "markdown", + "id": "68b80543", + "metadata": { + "nemo_colab_inject": true + }, + "source": [ + "### ⚡ Colab Setup\n", + "\n", + "Run the cells below to install the dependencies and set up the API key. If you don't have an API key, you can generate one from [build.nvidia.com](https://build.nvidia.com).\n", + "\n", + "This notebook uses the [`data-designer-docx`](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins/data-designer-docx) plugin, which Data Designer discovers automatically once it is installed.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13a45cd5", + "metadata": { + "nemo_colab_inject": true + }, + "outputs": [], + "source": [ + "%%capture\n", + "# data-designer-docx is not on PyPI yet; install it from the plugins repository.\n", + "# Once it is released this becomes: !pip install -U data-designer data-designer-docx\n", + "!pip install -U data-designer \\\n", + " \"data-designer-docx @ git+https://github.com/NVIDIA-NeMo/DataDesignerPlugins.git#subdirectory=plugins/data-designer-docx\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "60ff3499", + "metadata": { + "nemo_colab_inject": true + }, + "outputs": [], + "source": [ + "import getpass\n", + "import os\n", + "\n", + "from google.colab import userdata\n", + "\n", + "try:\n", + " os.environ[\"NVIDIA_API_KEY\"] = userdata.get(\"NVIDIA_API_KEY\")\n", + "except userdata.SecretNotFoundError:\n", + " os.environ[\"NVIDIA_API_KEY\"] = getpass.getpass(\"Enter your NVIDIA API key: \")" + ] + }, + { + "cell_type": "markdown", + "id": "314e4f40", + "metadata": {}, + "source": [ + "## Step 0 · About the plugin\n", + "\n", + "The `.docx` writer is not part of Data Designer. It is a separate plugin,\n", + "[`data-designer-docx`](https://github.com/NVIDIA-NeMo/DataDesignerPlugins/tree/main/plugins/data-designer-docx),\n", + "published in the NeMo Data Designer Plugins repository — the setup cell above already installed it.\n", + "\n", + "Two things to know about plugin installs:\n", + "\n", + "- **Nothing else is needed.** No registration call, no config entry. The package declares a\n", + " `data_designer.plugins` entry point and Data Designer discovers it automatically.\n", + "\n", + "- **Install before the kernel imports Data Designer.** Entry points are read from installed package\n", + " metadata at import time, so a plugin installed into an already-running kernel is not seen until you\n", + " restart the runtime.\n", + "\n", + "Running locally instead of in Colab? Same install, any package manager:\n", + "\n", + "```bash\n", + "pip install data-designer-docx\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "d788df2c", + "metadata": {}, + "source": [ + "## Step 1 · Check the plugin is there\n", + "\n", + "Data Designer finds plugins through installed package metadata, so this is really a check that the install\n", + "worked. If the list comes back empty, re-run the setup cell and restart the runtime.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1e0fabb8", + "metadata": {}, + "outputs": [], + "source": [ + "from data_designer.plugins.plugin import PluginType\n", + "from data_designer.plugins.registry import PluginRegistry\n", + "\n", + "print(\"processor plugins:\", PluginRegistry().get_plugin_names(PluginType.PROCESSOR))" + ] + }, + { + "cell_type": "markdown", + "id": "a25d9f71", + "metadata": {}, + "source": [ + "## Step 2 · Imports and your API key\n", + "\n", + "`WordDocument` comes from the plugin. It describes the *shape* of a document — title, summary, sections,\n", + "a table — and we will hand it straight to the LLM in a moment.\n", + "\n", + "Note where it comes from: the schema ships **with the plugin**, because it is the contract between the\n", + "LLM and the renderer, and the two halves belong in the same package.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3fa1a329", + "metadata": {}, + "outputs": [], + "source": [ + "import data_designer.config as dd\n", + "from data_designer.interface import DataDesigner\n", + "\n", + "from data_designer_docx.config import DocxProcessorConfig\n", + "from data_designer_docx.schema import WordDocument\n", + "\n", + "data_designer = DataDesigner(artifact_path=\"./artifacts\")" + ] + }, + { + "cell_type": "markdown", + "id": "cd9b0103", + "metadata": {}, + "source": [ + "## Step 3 · Say what documents you want\n", + "\n", + "Three samplers. They give every document an ID, a company, and a type — and because those values live in\n", + "the dataset, your finished corpus is labelled without you annotating anything.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a34bd475", + "metadata": {}, + "outputs": [], + "source": [ + "MODEL_ALIAS = \"doc-writer\"\n", + "MODEL_ID = \"nvidia/nemotron-3-super-120b-a12b\"\n", + "\n", + "config_builder = dd.DataDesignerConfigBuilder(\n", + " model_configs=[\n", + " dd.ModelConfig(\n", + " alias=MODEL_ALIAS,\n", + " model=MODEL_ID,\n", + " provider=\"nvidia\",\n", + " # A whole document in one call is a long generation. Give it room.\n", + " inference_parameters=dd.ChatCompletionInferenceParams(temperature=0.9, max_tokens=8192),\n", + " )\n", + " ]\n", + ")\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"doc_id\",\n", + " sampler_type=dd.SamplerType.UUID,\n", + " params=dd.UUIDSamplerParams(prefix=\"POL-\", short_form=True, uppercase=True),\n", + " )\n", + ")\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"company\",\n", + " sampler_type=dd.SamplerType.CATEGORY,\n", + " params=dd.CategorySamplerParams(\n", + " values=[\"Northwind Diagnostics\", \"Cobalt Ridge Financial\", \"Halden Biopharma\"]\n", + " ),\n", + " )\n", + ")\n", + "\n", + "config_builder.add_column(\n", + " dd.SamplerColumnConfig(\n", + " name=\"doc_type\",\n", + " sampler_type=dd.SamplerType.CATEGORY,\n", + " params=dd.CategorySamplerParams(\n", + " values=[\n", + " \"Remote Work Policy\",\n", + " \"Incident Response Runbook\",\n", + " \"Vendor Onboarding Procedure\",\n", + " \"Travel and Expense Policy\",\n", + " ]\n", + " ),\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "65672fa0", + "metadata": {}, + "source": [ + "## Step 4 · Write the document\n", + "\n", + "One LLM column does the writing. The important bit is `output_format=WordDocument`: instead of asking for\n", + "prose and parsing it afterwards, we ask for the document's *structure* and let the model fill it in.\n", + "\n", + "That is why no `.docx` parsing code appears anywhere in this notebook.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14db56bf", + "metadata": {}, + "outputs": [], + "source": [ + "config_builder.add_column(\n", + " dd.LLMStructuredColumnConfig(\n", + " name=\"document\",\n", + " model_alias=MODEL_ALIAS,\n", + " output_format=WordDocument,\n", + " prompt=(\n", + " \"Write an internal {{ doc_type }} for {{ company }} (document ID {{ doc_id }}).\\n\\n\"\n", + " \"Write like a real corporate policy: flat, procedural, no marketing language. \"\n", + " \"Sections should be specific to a {{ doc_type }} — scope, roles, the actual procedure, \"\n", + " \"exceptions — not filler like 'Introduction'. \"\n", + " \"The key_data table should hold concrete facts: thresholds, timeframes, who does what.\"\n", + " ),\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "669ebb4f", + "metadata": {}, + "source": [ + "## Step 5 · Turn each row into a `.docx`\n", + "\n", + "Here it is. One processor, and the pipeline now writes Word files.\n", + "\n", + "The templates are rendered per row, so anything in the dataset can go into the filename, the front-matter\n", + "table, or the footer.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a9c22674", + "metadata": {}, + "outputs": [], + "source": [ + "config_builder.add_processor(\n", + " DocxProcessorConfig(\n", + " name=\"word-documents\",\n", + " document_column=\"document\",\n", + " filename_template=\"{{ doc_id }}-{{ doc_type }}.docx\",\n", + " metadata_columns={\"Document ID\": \"{{ doc_id }}\", \"Company\": \"{{ company }}\"},\n", + " footer_template=\"{{ company }} · {{ doc_id }}\",\n", + " )\n", + ")\n", + "\n", + "data_designer.validate(config_builder)" + ] + }, + { + "cell_type": "markdown", + "id": "f4fd809e", + "metadata": {}, + "source": [ + "## Step 6 · Preview one, then make ten\n", + "\n", + "Always preview first. It is one document and one API call, and it tells you whether the prompt is landing\n", + "before you pay for a hundred.\n", + "\n", + "Notice `docx_path` in the output — the processor writes it back into the dataset, so rows and files stay\n", + "joined.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3bb1ec42", + "metadata": {}, + "outputs": [], + "source": [ + "preview = data_designer.preview(config_builder, num_records=1)\n", + "\n", + "preview.dataset[[\"doc_id\", \"company\", \"doc_type\", \"docx_path\"]]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b84da9dc", + "metadata": {}, + "outputs": [], + "source": [ + "results = data_designer.create(config_builder, num_records=10, dataset_name=\"policy-documents-plugin\")\n", + "\n", + "dataset = results.load_dataset()\n", + "dataset[[\"doc_id\", \"company\", \"doc_type\", \"docx_path\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "c0c4a19a", + "metadata": {}, + "source": [ + "## Step 7 · Open one\n", + "\n", + "`docx_path` is relative to the dataset folder, so the whole thing stays portable. Double-click the file to\n", + "open it in Word — or read it back here, the way a downstream pipeline would.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0cf4d398", + "metadata": {}, + "outputs": [], + "source": [ + "from docx import Document\n", + "\n", + "document_path = results.artifact_storage.base_dataset_path / dataset[\"docx_path\"].iloc[0]\n", + "rendered = Document(str(document_path))\n", + "\n", + "print(f\"{document_path.name}\\n\")\n", + "for paragraph in rendered.paragraphs:\n", + " if paragraph.style.name.startswith((\"Title\", \"Heading\")):\n", + " print(f\" {paragraph.text}\")\n", + "\n", + "print(f\"\\nfooter: {rendered.sections[0].footer.paragraphs[0].text}\")\n", + "print(f\"folder: {document_path.parent}\")" + ] + }, + { + "cell_type": "markdown", + "id": "04b8dd6e", + "metadata": {}, + "source": [ + "## That's it 🎉\n", + "\n", + "Ten real Word documents, and a dataset that knows which row produced which file.\n", + "\n", + "Things you can change from here, all in that one `DocxProcessorConfig`:\n", + "\n", + "| Want to... | Use |\n", + "| --- | --- |\n", + "| Apply your corporate styles | `template_path=\"brand/corporate-template.docx\"` |\n", + "| Fill in Word's author/category fields | `core_property_columns={\"author\": \"{{ owner }}\"}` |\n", + "| Change where files land | `output_subdir=\"documents\"` |\n", + "| Drop the `1.` `2.` numbering | `number_sections=False` |\n", + "\n", + "And when you want more control over the documents themselves, edit `WordDocument` in\n", + "`src/data_designer_docx/schema.py`. The field descriptions in that file are sent to the model as part of\n", + "the prompt, so they are the fastest lever you have.\n", + "\n", + "**Next:**\n", + "\n", + "- [Plugins overview](https://docs.nvidia.com/nemo/datadesigner/plugins/overview) — writing your own\n", + "\n", + "- [Processors](https://docs.nvidia.com/nemo/datadesigner/concepts/processors) — the built-in ones\n" + ] + } + ], + "metadata": { + "jupytext": { + "text_representation": { + "extension": ".py", + "format_name": "percent", + "format_version": "1.3", + "jupytext_version": "1.18.1" + } + }, + "kernelspec": { + "display_name": "Word Docs (data-designer)", + "language": "python", + "name": "word-docs" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}