Topic Merging

docs/SensTopic.md

@@ -162,6 +162,31 @@ model.print_topics()
 | 4 | tennis, competing, federer, wimbledon, iaaf, olympic, tournament, athlete, rugby, olympics |
 | 5 | gdp, stock, economy, earnings, investments, investment, invest, exports, finance, economies |
 
+## Batch fitting
+
+SensTopic models can be fit in mini-batches. This is done by fitting separate models on all batches, and continuously merging models of the new batches into the current model.
+!!! info
+    To find more information on model merging and its variants, please look at [Model Merging](topic_merging.md).
+
+```python
+from itertools import batched
+
+BATCH_SIZE = 2000
+corpus: Iterable[str] = [...]
+batches = batched(corpus, BATCH_SIZE)
+
+model = SensTopic(
+    sparsity=5.0,
+    random_state=42,
+)
+for batch in batches:
+    batch = list(batch)
+    batch_doc_topic = model.partial_fit_transform(
+        batch, merge_method="asymmetric_mean"
+    )
+model.print_topics()
+```
+
 ## Citation
 
 Please cite Turftopic when using the SensTopic model:

docs/analyzers.md

@@ -1,6 +1,6 @@
-# Topic Analysis with LLMs
+# Topic Analysis
 
-Topic analyzers are large language models, that are capable of interpreting topics' contents and can give human-readable descriptions of topics.
+Topic analyzers are language-model based solutions, that are capable of interpreting topics' contents and can give human-readable descriptions of topics.
 This can be incredibly useful when it would require excessive manual labour to label and understand topics.
 
 <figure>
@@ -10,12 +10,11 @@ This can be incredibly useful when it would require excessive manual labour to l
 
 Analyzers can do the following tasks:
 
-  - **Summarize documents** to make it easier for your topic model to consume.
+  - **Summarize documents** (*optional*) to make it easier for your topic model to consume.
   - **Name topics** topics in a sensible and human-readable way based on top documents and keywords
   - **Describe topics** in a couple of sentences
 
-While previously, smaller language models were not able to meaningfully accomplish this task,
-advances in in the field now allow you to generate highly accurate topic descriptions on your own laptop using the power of small LLMs.
+This can either be achieved using generative, text2text or extractive language models.
 
 !!! warning
 
@@ -45,6 +44,18 @@ There are multiple types of analyzers in Turftopic that you can utilize for thes
         analyzer = LLMAnalyzer(use_summaries=True)
         ```
 
+    === "WikiAnalyzer (experimental)"
+
+        You can use the Wikipedia API to search for candidate topic names and descriptions and retrieve the best one using an encoder-style language model. By default, this uses the `encoder` of a topic model:
+        **NOTE:** The WikiAnalyzer can only name and describe topics and cannot summarize documents.
+
+        ```python
+        from turftopic.analyzers import WikiAnalyzer
+
+        model = SensTopic()
+        analyzer = WikiAnalyzer(model)
+        ```
+
     === "OpenAI API"
 
         You will have to install OpenAI, as it is not installed by default:

docs/distribution_learning.md

@@ -0,0 +1,53 @@
+# Topic Distribution Learning
+
+While in most scenarios you can store an entire document-topic matrix in memory, this is not always the case, especially with extremely large datasets.
+Distribution learners in Turftopic are exactly developed for this reason.
+
+With a distribution learner, you can pass document-topic matrices per batch, and update its parameters, while slowly learning the true distribution of topics in the dataset with uncertainty.
+
+To use distribution learners you should install `conjugate-models`:
+
+```bash
+pip install turftopic[conjugate]
+```
+
+## Example
+
+```python
+import numpy as np
+import pandas as pd
+
+from sklearn.datasets import fetch_20newsgroups
+from turftopic import SensTopic
+from turftopic.distribution_learning import GaussianDistributionLearner
+
+ds = fetch_20newsgroups(remove=("headers", "footers", "quotes"), subset="all")
+corpus = ds.data
+
+batch_size = 2000
+model = SensTopic(random_state=42)
+# Initializing the distribution learner
+distribution_learner = GaussianDistributionLearner()
+# batch fitting over the dataset
+for batch_start in range(0, len(corpus), batch_size):
+    batch_end = batch_start + batch_size
+    # Calculating doc_topic_matrix for current batch
+    batch_doc_topic_matrix = model.partial_fit_transform(
+        corpus[batch_start:batch_end],
+        merge_method="asymmetric_mean",
+    )
+    # Updating the posteriors
+    distribution_learner.update(batch_doc_topic_matrix)
+
+# `pip install plotly` if you want to plot
+distribution_learner.plot_topic_distribution(model.topic_names)
+```
+
+<figure>
+  <iframe src="../images/distribution_learner.html", title="Learned topic distribution", style="height:420px;width:900px;padding:0px;border:none;"></iframe>
+  <figcaption> Topic distribution learned by the GaussianDistributionLearner. </figcaption>
+</figure>
+
+## API Reference
+
+::: turftopic.distribution_learning.GaussianDistributionLearner