feat(medcat-tutorials): Add bulk inference tutorial

.github/workflows/medcat-v2-tutorials_main.yml

@@ -43,10 +43,14 @@ jobs:
           pytest --collect-only --nbmake ./notebooks/introductory/basic/1*.ipynb
           pytest --collect-only --nbmake ./notebooks/introductory/basic/2*.ipynb
           pytest --collect-only --nbmake ./notebooks/introductory/basic/3*.ipynb
+          pytest --collect-only --nbmake ./notebooks/introductory/basic/4*.ipynb
+          pytest --collect-only --nbmake ./notebooks/introductory/basic/5*.ipynb
           pytest --collect-only --nbmake ./notebooks/introductory/meta/*.ipynb
           pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/basic/1*.ipynb
           pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/basic/2*.ipynb
           pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/basic/3*.ipynb
+          pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/basic/4*.ipynb
+          pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/basic/5*.ipynb
           pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/meta/*.ipynb
           pytest --nbmake -n=auto --nbmake-kernel=smoketests --nbmake-timeout=1800 ./notebooks/introductory/custom/*.ipynb
 

medcat-v2-tutorials/mkdocs.yml

@@ -40,6 +40,7 @@ nav:
       - 2. Unsupervised training on model: introductory/basic/2._Unsupervised_training_on_model.ipynb
       - 3. Supervised training on model: introductory/basic/3._Supervised_training_on_model.ipynb
       - 4. Evaluating performance on dataset: introductory/basic/4._Evaluating_performance_on_dataset.ipynb
+      - 5. Running models on large datasets: introductory/basic/5._Running_model_on_large_datasets.ipynb
     - MetaCAT:
       - README: introductory/meta/README.md
       - 1. Add a MetaCat to a Model: introductory/meta/1._Add_a_MetaCat_to_a_Model.ipynb

medcat-v2-tutorials/notebooks/introductory/basic/5._Running_model_on_large_datasets.ipynb

@@ -0,0 +1,323 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "3179d03d",
+   "metadata": {},
+   "source": [
+    "# Running model on large datasets\n",
+    "\n",
+    "When running the model on large datasets there are multiple aspects to consider in terms of speeding up the process:\n",
+    "- *The concept filter to use*. Use cases often only care about a small subset of concepts a model is trained on. By pre-applying a narrow filter you can make the model's inference run a lot faster because it then does not need to spend time on trying to disambiguate spans that you do not care about.\n",
+    "- *The available RAM on the device*. If there's ample RAM available on the device, multiprocessing might be the right thing to focus on. However, if RAM is limited, processing single threaded might be preferred. That's because multi-processing works on different processes, each of which will effectively need to load the model to perform the inference.\n",
+    "- *The amount of data in question*. If the amount of data in question is larger than what would fit in memory, you may want to use multi-processing. The multiprocessing method is designed for large amounts of data: it processes the incoming data in batches and returns an iterable that needs to be yielded over in order for the method to do its work. This has explicitly been designed in a way that avoids loading all the data into memory at once. It does batching even if only one process is used. The method also provides options to save the output on disk if that's preferred at this stage. PS: You can also do batching of data yourself and use regular inference."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "77aba011",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import time\n",
+    "from medcat.cat import CAT\n",
+    "from medcat.utils.config_utils import temp_changed_config"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2aaf5d2e",
+   "metadata": {},
+   "source": [
+    "## Data preparations\n",
+    "\n",
+    "NOTE: In this tutorial we do not have a very large amount of data, so we will artificially inflate it by repeating the same data over and over again. You would normally not do that (somewhat obviously)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "eb80cad4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "DATA_PATH = os.path.join(\"in_data\", \"dummy_bulk_data.lines\")\n",
+    "INFLATION_COEF = 100\n",
+    "with open(DATA_PATH) as f:\n",
+    "    raw_data = list(map(str.strip, f.readlines()))\n",
+    "\n",
+    "# this will create a generator for the data\n",
+    "# that repeats INFLATION_COEF (100) times\n",
+    "# NOTE: it is best to have each piece of data\n",
+    "#       along with its identifier so you can stitch things\n",
+    "#       together later, but it's not strictly necessary\n",
+    "def data_iter():\n",
+    "    for dupe_num in range(INFLATION_COEF):\n",
+    "        for line_num, line in enumerate(raw_data):\n",
+    "            yield (f\"{line_num:05d}_d{dupe_num:04d}\", line)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b06603ee",
+   "metadata": {},
+   "source": [
+    "## Loading model pack\n",
+    "\n",
+    "We will download the MedMentions model here.\n",
+    "The models we have created earlier in the tutorials are a little too simplistic for the purposes relevant here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "9aa02d20",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--2026-07-28 11:55:30--  https://cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com/medcat-example-models/medmen_wstatus_2021_oct.zip\n",
+      "Resolving cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com (cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com)... 52.95.142.18, 3.5.246.104, 3.5.244.8, ...\n",
+      "Connecting to cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com (cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com)|52.95.142.18|:443... connected.\n",
+      "HTTP request sent, awaiting response... 200 OK\n",
+      "Length: 561947681 (536M) [application/zip]\n",
+      "Saving to: ‘models/medmen_wstatus_2021_oct.zip’\n",
+      "\n",
+      "medmen_wstatus_2021 100%[===================>] 535.92M  71.3MB/s    in 7.8s    \n",
+      "\n",
+      "2026-07-28 11:55:39 (68.7 MB/s) - ‘models/medmen_wstatus_2021_oct.zip’ saved [561947681/561947681]\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Doing legacy conversion on CAT (at 'models/medmen_wstatus_2021_oct'). Set the environment variable MEDCAT_AVOID_LECACY_CONVERSION to `True` to avoid this.\n",
+      "Missing class medcat.config.weighted_average, replacing with LegacyClassNotFound.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Conditions False False IN {'T019', 'T067', 'T028', 'T062', 'T101', 'T092', 'T109', 'T014', 'T097', 'T120', 'T203', 'T102', 'unk', 'T100', 'T201', 'T096', 'T041', 'T204', 'T060', 'T048', 'T077', 'T031', 'T033', 'T070', 'T045', 'T122', 'T024', 'T080', 'T069', 'T034', 'T013', 'T191', 'T039', 'T168', 'T049', 'T059', 'T192', 'T056', 'T054', 'T050', 'T197', 'T044', 'T052', 'T025', 'T065', 'T114', 'T087', 'T194', 'T002', 'T011', 'T125', 'T091', 'T030', 'T185', 'T017', 'T190', 'T082', 'T094', 'T042', 'T116', 'T090', 'T051', 'T123', 'T022', 'T032', 'T086', 'T047', 'T008', 'T010', 'T043', 'T055', 'T021', 'T064', 'T129', 'T075', 'T171', 'T046', 'T071', 'T001', 'T057', 'T200', 'T085', 'T103', 'T169', 'T130', 'T007', 'T020', 'T068', 'T058', 'T074', 'T099', 'T015', 'T121', 'T005', 'T063', 'T016', 'T038', 'T079', 'T012', 'T061', 'T196', 'T098', 'T126', 'T127', 'T073', 'T072', 'T053', 'T078', 'T095', 'T081', 'T093', 'T184', 'T029', 'T083', 'T195', 'T026', 'T131', 'T167', 'T004', 'T040', 'T023', 'T104', 'T037', 'T170', 'T018', 'T089', 'T066'}\n",
+      "Got TypeID2name for 127 TypeIDs out of 127\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Trying to set 'cdb_source_name' for 'General' but no such attribute\n",
+      "Trying to set 'weighted_average_function' for 'Linking' but no such attribute\n",
+      "Optional path 'version.description' not found in old config. Ignoring\n",
+      "Optional path 'version.id' not found in old config. Ignoring\n",
+      "Optional path 'version.ontology' not found in old config. Ignoring\n",
+      "/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-v2-tutorials/.venv312/lib/python3.12/site-packages/spacy/util.py:969: UserWarning: [W095] Model 'en_core_web_md' (3.1.0) was trained with spaCy v3.1.0 and may not be 100% compatible with the current version (3.8.11). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate\n",
+      "  warnings.warn(warn_msg)\n",
+      "/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-v2-tutorials/.venv312/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "\n",
+    "# Download the MedMentions modelpack\n",
+    "! wget -N https://cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com/medcat-example-models/medmen_wstatus_2021_oct.zip -P models/\n",
+    "model_path = os.path.join(\"models\", \"medmen_wstatus_2021_oct.zip\")\n",
+    "\n",
+    "cat = CAT.load_model_pack(model_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b07225e8",
+   "metadata": {},
+   "source": [
+    "## Demonstrating power of filtering in sequential inference\n",
+    "\n",
+    "Runtime speed will differ from model to model and your model will (almost certainly) be bigger than the one we're using here.\n",
+    "\n",
+    "However, just to demonstrate the power of filters, we will run the same data with and without them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "cc1199d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def time_entire_run():\n",
+    "    start_time = time.perf_counter()\n",
+    "    ent_counter = 0\n",
+    "    for cntr, (tid, text) in enumerate(data_iter()):\n",
+    "        # NOTE: we're currently not doing anything with the output\n",
+    "        #       but in a real use case you would want to\n",
+    "        ents = cat.get_entities(text)[\"entities\"]\n",
+    "        ent_counter += len(ents)\n",
+    "    end_time = time.perf_counter()\n",
+    "    print(\"Took\", end_time - start_time, \"seconds for\", cntr + 1, \"texts\")\n",
+    "    print(\"We found\", ent_counter, \"entities\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "6c89db5d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Took 32.59635237487964 seconds for 6700 texts\n",
+      "We found 17200 entities\n"
+     ]
+    }
+   ],
+   "source": [
+    "# without any filtering\n",
+    "time_entire_run()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "358e861f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Took 31.438669624971226 seconds for 6700 texts\n",
+      "We found 200 entities\n"
+     ]
+    }
+   ],
+   "source": [
+    "# NOTE: You would likely need to come up with your own filter.\n",
+    "#       This model is UMLS based, but in Snomed the concept\n",
+    "#       IDs are fully numeric, but need to be sent to medcat\n",
+    "#       as strings\n",
+    "CUI_FILTER = {\n",
+    "    \"C0039231\", # tachycardia\n",
+    "    \"C0428977\", # bradycardia\n",
+    "    \"C0018799\", # heart disease\n",
+    "}\n",
+    "# NOTE: we're running a temporarily changed filter here so that\n",
+    "#       we don't have to change the model state, but you would\n",
+    "#       probably simply make the change permanent in the model:\n",
+    "# cat.config.components.linking.filters.cuis = CUI_FILTER\n",
+    "with temp_changed_config(cat.config.components.linking.filters, 'cuis', CUI_FILTER):\n",
+    "    time_entire_run()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d497fc74",
+   "metadata": {},
+   "source": [
+    "NOTE: In the real world, with bigger models, the time difference is likely to be a fair bit bigger. That's because the small model we're using here has less training and fewer ambiguous concepts, so it doesn't need to do as much work on the disamibguation."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4f36bfd",
+   "metadata": {},
+   "source": [
+    "## Showing multiprocessing\n",
+    "\n",
+    "We can also run the same data through the multiprocessing pipeline.\n",
+    "The API reference of this method is available (here)[https://docs.cogstack.org/projects/nlp/en/latest/reference/medcat/cat/#medcat.cat.CAT.get_entities_multi_texts].\n",
+    "However, we'll give a quick overview of some of the key parameters of this (`get_entities_multi_texts`) method:\n",
+    "- `texts` - the input texts. This can be a list or a generator. Each item can be either a single string (the text) or a tuple of two strings (the text index along with the text). The latter helps with mapping the annotations to the documents if/when required.\n",
+    "- `n_process` - the number of processes to use. Each parallel process will consume more memory / RAM. But running the text in parallel should help speed up the process when running through a lot of data. The default is 1 (single-threaded, but still batched).\n",
+    "- `batch_size` - the size of a batch. That is, the number of texts / documents to batch at a time. For large datasets it's important not to load all the data into memory at once. And you can control that by changing this value. This defaults to -1 and in such a case character-based batching is used instead.\n",
+    "- `batch_size_chars` - the size of a batch in number of characters. In different use cases, the documents are different lengths. But the memory footprint is normally dictated by the number of characters. So this option allows one to specify the number of characters rather than the number of documents for batching. This is usually the more sensible soluton. This defaults to 1 million. Set to -1 to disable.\n",
+    "- `save_dir_path` - the folder to write the output to. If you wish the output to be written on disk instead of staying in memory, set this to the path you wish to use. The outputs (usually multiple) will be saved in `part_<num>.pickle` format in this folder.\n",
+    "\n",
+    "NB! This method returns an iterator. That means that no work will be done unless the output is iterated over.\n",
+    "This is a deliberate design approach to avoid loading all the input data (along with the annotated output) in memory at once."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "14243e5b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def time_run_w_mp():\n",
+    "    start_time = time.perf_counter()\n",
+    "    ent_counter = 0\n",
+    "    for text_cntr, (text_id, ents) in enumerate(cat.get_entities_multi_texts(data_iter(), n_process=2)):\n",
+    "        ent_counter += len(ents[\"entities\"])\n",
+    "    ent_counter += len(ents)\n",
+    "    end_time = time.perf_counter()\n",
+    "    print(\"Took\", end_time - start_time, \"seconds for\", text_cntr + 1, \"texts\")\n",
+    "    print(\"We found\", ent_counter, \"entities\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "450caff2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Took 51.71125883399509 seconds for 6700 texts\n",
+      "We found 17202 entities\n",
+      "Took 32.595435000024736 seconds for 6700 texts\n",
+      "We found 200 entities\n"
+     ]
+    }
+   ],
+   "source": [
+    "# regular\n",
+    "time_run_w_mp()\n",
+    "# with filter\n",
+    "with temp_changed_config(cat.config.components.linking.filters, 'cuis', CUI_FILTER):\n",
+    "    time_entire_run()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "682a98af",
+   "metadata": {},
+   "source": [
+    "NOTE: In order to actually reap the benefit (in terms of time) from multi-processing, you will need to run through a lot more data than we did in this example. In this example, a lot of the time was probably spent setting up the orchestration for these separate processes. When running an example workflow with 1000 MIMIC-IV documents, the multiprocessing (with `n_process=2`) yielded about a 10% overall speedup."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv312 (3.12.10)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}