Distributed Parameter Server

An event-driven distributed training system written in C. A central parameter server coordinates multiple workers over TCP while each worker trains on a local shard of a dataset. The current model is binary logistic regression, implemented with the GNU Scientific Library (GSL).

This project is a compact demonstration of systems programming and distributed machine-learning fundamentals: socket protocols, partial I/O, worker pools, timeouts, signal handling, and synchronous gradient aggregation.

Showcase

                 TCP workers
        +-----------------------------+
        |                             |
   +---------+   +---------+   +---------+
   | worker  |   | worker  |   | worker  |
   | shard A |   | shard B |   | shard C |
   +----+----+   +----+----+   +----+----+
        |             |             |
        +-------------+-------------+
                      |
              +-------v--------+
              | parameter      |
              | server         |
              | poll() + BSP   |
              +-------+--------+
                      |
                 weights.txt

The server:

• Accepts and validates worker handshakes.
• Broadcasts the current parameter vector at the start of each round.
• Aggregates worker gradients and local losses synchronously.
• Applies an averaged gradient-descent update.
• Prunes workers that miss a round deadline.
• Handles partial socket writes, disconnects, SIGINT, and SIGTERM gracefully.

The worker:

• Loads one CSV shard into GSL matrices and vectors.
• Computes logits, sigmoid probabilities, cross-entropy loss, and gradients.
• Sends one gradient update per training round.
• Receives the final weights and termination reason from the server.

Features

• Event-driven server loop using poll().
• Length-prefixed TCP messages with explicit protocol types.
• Up to 255 active workers per server.
• Configurable worker threshold, learning rate, loss threshold, epoch limit, round timeout, verbosity, and port.
• Reusable CSV sharding utility with label-column selection, deterministic shuffling, and configurable output directories.
• Final model parameters written to weights.txt.

Requirements

• C compiler and GNU Make
• GSL development headers and libraries
• Python 3.10+ and the docopt package for split_shards.py
• pkg-config is recommended for automatic GSL discovery

On macOS:

brew install gcc gsl pkg-config

On Debian or Ubuntu:

sudo apt install build-essential libgsl-dev pkg-config python3 python3-pip
python3 -m pip install docopt

If GSL is installed in a non-standard location, pass its flags explicitly:

make GSL_CFLAGS="-I/path/to/gsl/include" \
     GSL_LIBS="-L/path/to/gsl/lib -lgsl -lgslcblas -lm"

Build

make

This produces two binaries:

• server: coordinates training rounds and owns the global parameter vector.
• worker: connects to the server and trains on one local shard.

The default TCP port is 61234. Override it at build time for both binaries:

make clean
make PORT=62000

Run

First split a normal CSV file into worker-compatible shards. The last column is used as the label by default:

python3 split_shards.py data.csv 4 --output-dir shards

The server command is:

./server <num_features> <max_epochs> <learning_rate> \
         <loss_threshold> <min_workers> <round_timeout_sec> \
         [-v <0|1|2>]

For example, for four workers and three features:

./server 3 100 0.01 0.001 4 30 -v 1

Start one worker per shard in separate terminals:

./worker 127.0.0.1 shards/01_of_04.csv -v 1
./worker 127.0.0.1 shards/02_of_04.csv -v 1
./worker 127.0.0.1 shards/03_of_04.csv -v 1
./worker 127.0.0.1 shards/04_of_04.csv -v 1

Training begins when min_workers workers have completed the handshake. The server writes the final vector, including the bias term, to weights.txt.

Data Format

Workers expect each shard in this format:

<num_samples>, <num_features + 1>
feature_1,feature_2,...,label
value,value,...,target

The first parameter is the bias. The remaining parameters correspond to the feature columns. split_shards.py converts ordinary CSV files into this format and places the label column last.

Useful examples:

python3 split_shards.py data.csv 8 --label-col target
python3 split_shards.py data.csv 4 --no-shuffle --seed 42
python3 split_shards.py raw.csv 2 --no-header --label-col 0

Repository Layout

src/
  server.c, server_utils.c, worker_pool.c   server and worker lifecycle
  worker.c, worker_utils.c, ml.c            worker training and math
  net_utils.c, protocol.h                   TCP framing and wire protocol
  error.h, logging.h                        shared diagnostics
split_shards.py                             dataset preparation utility
Makefile                                    build configuration

Scope and Limitations

This is an educational systems project rather than a production training service. It currently uses one parameter server, logistic regression, CSV data loaded into memory, and a fixed binary protocol. TCP connections are not authenticated or encrypted, and floating-point payloads are transmitted in the host representation. Production use would require stronger transport security, protocol versioning, model abstraction, persistent state, and more comprehensive integration testing.