Document GitHub Sentiment Dataset Pipeline

[splimon]

Purpose

To create clear and reproducible documentation so future contributors can understand and rerun the GitHub sentiment pipeline without reverse‑engineering prior work. The notebooks should explain the commit/PR comment flow and produce outputs compatible with Kaiaulu.

Process

Create a documentation PR that:

• Adds the .yml files for the GitHub projects used in the dataset
• Adds the Python script used to load sentiment labels into MySQL
• Converts existing MySQL Workbench queries into Jupyter notebooks
• Organizes notebooks in a clear way:
  • Load sentiment CSV into MySQL
  • Explore DB + identify relevant tables
  • Scale/automate across all projects
• Export each notebook to a .py file so reviewers can read the code directly in GitHub (in addition to the .ipynb)
• Does not commit any data files (e.g., CSV exports, database dumps, generated data, etc.), only code and documentation (scripts, notebooks, configs)

The final notebook must output a file that can be inner joined with Kaiaulu output for the same project.

Task List

• Add .yml config project files to the repo
• Add Python loader script for loading sentiment labels into MySQL
• Create Notebook 1: load sentiment CSV into MySQL
• Create Notebook 2: explore DB + locate relevant tables/joins
• Create Notebook 3: scale/automate pipeline across all projects
• Export notebooks to .py for code review
• Ensure final notebook outputs a file that is joinable with Kaiaulu

[splimon]

4/3/26 Capstone Meeting Notes (Sentiment Classifier)

Hi @carlosparadis,

Next Steps

Notebook Documentation

• Use Python library calls to access and run SQL queries automatically in the notebooks (no manual SQL copy/paste)
• Display query results inline so users can validate outputs quickly
• Finalize the GitHub commit + PR comment documentation workflow. Use the notebooks as a guide so a new contributor can follow the notebook steps end-to-end without additional clarification
• After GitHub documentation is complete, apply the same workflow and documentation pattern to the Jira emotion dataset: Implement sentiment workflow with Jira Dataset emotion_jira_dataset#1

[splimon]

4/10/26 Meeting Notes

Hi @carlosparadis ,

Next Steps

Notebook Documentation

• Notebooks 1-2 narrative:
  • Problem: the Gold Standard CSV only has ID, polarity, and text. It can't connect to Kaiaulu as-is
  • Goal: add columns from GHTorrent so the CSV can be INNER JOINed to Kaiaulu output
• Notebook 2 (Exploring GHTorrent Tables) revisions:
  • End with two DataFrames: (1) original Gold Standard header, (2) contextualized dataset header with an explanation of why each new column is needed. Also add final contextualized CSV at the end. This will be the CSV that will be INNER JOINed to the Kaiaulu output and is also the input for Notebook 4.
• Notebook 3 (Scale config files) narrative:
  • Reframe intro: "Now that we have the contextualized CSV, let's generate configs so we can get Kaiaulu data to join it with"
• Notebook 4 (NEW NOTEBOOK):
  • Inputs: contextualized CSV from Notebook 2 + parsed Kaiaulu output
  • Make clear how the Kaiaulu output was produced: Name the downloader/refresher function and parser function used
  • Load both files, INNER JOIN on comment_id, show the result
  • Calling exec in Kaiaulu to do this programmatically is a future implementation

[splimon]

Hi @carlosparadis

4/17/26 Meeting Notes

Next Steps

Notebook Documentation

• Final sentiment + columns CSV code currently converts string polarity values ('positive', 'negative', 'neutral') to integers (0 = neutral, 1 = positive, 2 = negative). Change integers to represent 0 = neutral, 1 = positive, -1 = negative to be compatible with the inputs expected for engagement metrics

Sentiment Dataset Cheatsheet

• On Sentiment Dataset Cheatsheet, may want to emphasize somewhere that columns INNER JOINed to Kaiaulu are matched via comment_id

Jira Emotion Dataset (sailuh/emotion_jira_dataset)

• Verify the database join confirm that comment IDs from Notebook 1 (inter-rater agreement analysis) can be joined to jira_issue_comment via id
  • From there, join to jira_issue_report for project/key/repository context
  • Look up one entry in the browser to confirm the IDs are real
• Identify an Apache project in the database that Kaiaulu already has a config for
• Download that project using Kaiaulu. Use the date range from dataset to limit the download and avoid an IP block from Apache's API
• Test that the output can be inner joined with Kaiaulu the same way as the GitHub dataset
• Document end-to-end flow in a notebook, following similar structure of notebook documentation in sailuh/sentiment_github_dataset

[splimon]

Hi @carlosparadis ,

Here are the meeting notes from today's meeting:

5/1/26 Meeting Notes (Sentiment Dataset)

Next Steps

Data versioning

• Version project CSVs from Github dataset to sentiment_github_dataset in Add Sentiment Github Dataset Documentation Notebooks #3
• Version at least 1 outputted CSV from Jira dataset to jira_emotion_dataset in Add Jira Emotion Dataset Documentation Notebooks emotion_jira_dataset#2

Sentiment Notebook (divide work with Adam)

Goal: Train and predict on GitHub data using the sentiment_classifier repo

• step 1: Clone sentiment_classifier repo
• step 2: Run conda with env.yml to install dependencies
• step 3: Update tools.yml to point pysenti to sentiment_classifier/exec/train_or_predict.py (ensure you're in the conda environment)
• step 4: Run sentiment_analysis.Rmd
  • Run only Github-related chunks
  • Clean up the notebook: keep the GitHub sections, delete Mailing List and Jira chunks
  • Only need to run pysenti_train and pysenti_predict
    • TRAIN the GitHub labeled dataset
    • PREDICT the Kaiaulu PR comments (downloaded from download_github_pull_request_comments.Rmd)
• If able to train and predict on contextualized Github dataset - success! (hand output to Jared to build a network graph)
• Bonus: Call Adam's function with the predicted table (predicting Kaiaulu's own comments and PRs)
  • Insert into Adam's engagement sentiment metric in sentiment_analysis.Rmd and verify it produces output

[splimon]

Hi @carlosparadis ,

An update on my progress this week:

I versioned the CSVs from the GitHub dataset to this PR: #3. The CSVs include polarity and text columns + Kaiaulu columns for each project, separated into /PR inline comments and /commit comments folders.

I cleaned up sentiment_analysis.Rmd to remove the Jira and Mbox chunks and keep only the GitHub-related chunks. I got the notebook running end-to-end and used pysenti_train to train on the GitHub labeled dataset CSVs using the BERT model, then used pysenti_predict to predict sentiment on Kaiaulu's own PR comments. This is a sample of what the CSV with the predictions looks like:

I sent those predictions to Jared to build a network graph.

One thing I discovered when going through the full pipeline: the polarity convention in the CSVs is 0=neutral, 1=positive, 2=negative because the model internally uses slot indices 0, 1, and 2, so a value like -1 isn't a valid slot and would crash the model.

I ran adam's engagement function, but had to change the column names from the CSVs to match his parameter names so:
CSV name -> engagement function parameter name:
created_at -> reply_datetimetz
comment_user_login -> reply_from

This is what I see after running his engagement function on PR inline comments I downloaded from Kaiaulu:

For the sentiment_analysis.Rmd changes needed to support the full pipeline, would you like me to commit those to sailuh/kaiaulu#347?

[splimon]

To run the sentiment_analysis.Rmd notebook addressed in Kaiaulu issue #346, we need these files:

For training, we will use a CSV of commit comments with labeled polarity data extracted from the sentiment_github_dataset:
akka_sentiment_commit_comments_joined.csv

For predicting, we will use a CSV of unlabeled PR comments extracted using Kaiaulu's download_github_pull_request_comments.Rmd:
kaiaulu_pr_inline_comments.csv

[carlosparadis]

This comment is missing my recommendation on how to proceed:

Use the combined model across all projects I added as a review on your dataset (add a code block somewhere in your 4 notebooks that exports it so others know where it came from).

Then for the train you can fread its url.

However, for the predict you should not rely on any csv. The notebook already loads Kaiaulu comments presumed to have been downloaded by the downloader. Use them to construct within the notebook the predict table of comments, make your column changes in the notebook and call pysenti predict.

If you do not do this way, the user can't rely on your notebook to predict their data, since the code will not prepare the comment for them to predict on them with the trained model.

[carlosparadis]

Just some clarification on this:

Updating the github_dataset to have the 3 additional datasets, and subsequently linking it to the notebook via fread() will enable the following for users:

1. User can choose any one dataset by simply changing the fread URL as you just did for akka. They can do then per project temporal expansion or horizontal according to your cheatsheet.
2. If you add the combined .csv and the code block on your notebook you also enable the user to quickly:
   1. Choose all pr comments in a project (if they want to test if the model performs better by using all comments regardless of projects).
   2. Choose all commit comments .csv across project for the same reason above.
   3. Choose everything.
3. What this means is that you don't take for granted any one project or any way of combining the data is best: The user can use any of the experimental setups to train various machine learning models, and see what they think is better, all with the change of a single URL (which could be put on a csv and looped at by them as they so prefer).

With the intent of having this dataset being the de-facto way to use the sentiment notebook in Kaiaulu, this will set a good structure for Fall'26 group to look at this and create the same organization for a future emotion Notebook. At the same time, the group experimenting with different model families will greatly benefit from this change on your sentiment notebook to do their work.

The request to use the Notebook already prepared github serves the same purpose: Someone using the sentiment notebook can just adjust the config to a new project, run the code, to prepare the data they want the sentiment labels for in a run click, and the setup changing the url from the previous step.

Combined you just created the simplest way possible anyone could get "sentiment labels" out of a Notebook across two repos.

[splimon]

@carlosparadis

I added an additional notebook in the sentiment_github_dataset repo (5_combine_all_projects.ipynb) to address the different ways a user can train data (per project, all commit comments, all PR comments, or all comments combined) and committed the changes to PR #3.

To run the sentiment_analysis.Rmd notebook addressed in Kaiaulu issue #346, we need these files:

For training, depending on their goal, a user has 4 options to train data:

1. Per project (e.g. akka_sentiment_commit_comments_joined.csv)
2. All commit comments combined: combined_commit_comments.csv
3. All PR comments combined: combined_pr_inline_comments.csv
4. All comments from all projects (commit + PR) combined:
   combined_all_comments.csv