Description

Event-driven static HTTP server sustaining 10,000 concurrent connections under load testing with realistic web workloads, built in C++, raw Unix TCP sockets and non-blocking I/O with poll(). Incremental HTTP 1.1 parsing directly from the RFC. Provided nginx-style configuration supporting virtual servers.

Key features:

• Single process
• Non-blocking
• Event-driven
• I/O multiplexed
• High concurrency
• High throughput
• Error handling
• Partial write management
• Full HTTP request lifecycle
• Incremental HTTP request parser
• Incremental HTTP response streaming
• Configuration with subset of nginx syntax
• Virtual servers
• Routing rules
• Directory autoindexing
• CGI support

Modules

Module	Description
Config	Parses the configuration file for initialization of the server(s)
Network	Handles server creation, management, clients, data transmission
HTTP	Parses transmitted data into HTTP
Handler	Performs operations requested via HTTP

Measurements

$ wrk -t12 -c10000 -d60s http://127.0.0.1:8081/canvas/5_rock_paper_scissors.html
Running 1m test @ http://127.0.0.1:8081/canvas/5_rock_paper_scissors.html
12 threads and 10000 connections
Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.94ms   52.38ms   1.99s    99.51%
    Req/Sec     1.25k     0.92k    7.63k    69.61%
752456 requests in 1.00m, 0.87GB read
Socket errors: connect 0, read 752472, write 0, timeout 876
Requests/sec:  12525.81
Transfer/sec:     14.78MB

Configuration

Example

events {} # Always empty, just to make configs interchangable with nginx

http {
	server {
        client_max_body_size 512;
		listen 8080;
        server_name localhost myserver;
        root /var/www/example.com;
        index example.html;
        error_page 404 /errors/404.html;
        error_page 403 /errors/403.html;

        location /contact {
            client_max_body_size 2048;
            root /var/www/example.com/contact;
            index contact.html;
            error_page 500 /errors/500_contact.html;
            autoindex off;
            allow_methods GET POST; # equivalent to `limit_except` in nginx
            allow_upload on;
        }
        
        location /upload {
            upload_store /path/to/permanent/uploads;
        }
        
        location /redirect {
            return /contact;
        }
	}

	server {
		server_name commonname orange;
		listen 8081;
		root /var/www/mywebsite.com;

		location / {
			index main.html;
		}

		location /cgi-bin {
			allow_methods GET POST;

			cgi .py /usr/bin/python3;
			cgi .sh /usr/bin/sh;
		}
	}

	server {
        # This is a virtual server since it listens to the same port as the above server.
        # First server names are also clashing.
        # This is being handled and second name is used.
		server_name commonname banana;
		listen 8081;
		root /var/www/anotherwebsite.com;
		index start.html;

		location /filestash {
			autoindex on;
		}
	}
	
	server {
		server_name guestbook;
		listen 8082;
		root /var/www/goodoldwebsite.com;

		location /cgi-bin {
			autoindex on;
			allow_methods GET POST DELETE;
			cgi .pl /usr/bin/perl;
			cgi .sh /usr/bin/sh;
		}
	}
}

Blocks

Configuration has 4 blocks:

1. events {}: Exists for compatibility with nginx configurations, it is always empty.
2. http {}: There must be only one http block containing the configuration. It may only have server blocks inside it.
3. server {}: Defines a virtual server.
4. location {}: Defines a specific location/directory for a server.

Directives

Following are the configuration directives, their scopes, and brief explanations. If a directive is set in both server and location blocks, location value will be considered for requests to that location.

Directive	server block	location block	Notes
listen	✅	❌	Specify port to listen.
client_max_body_size	✅	✅	Set maximum request body size.
root	✅	✅	Root directory for the server or location.
autoindex	❌	✅	Enables directory listings.
index	✅	✅	Sets default file to serve in a directory.
alias	❌	✅	Swaps location.route to alias and doesn’t use root but rather alias for resolvedPath.
error_page	✅	✅	Specify custom error pages.
server_name	✅	❌	Host header sent by the client will target this value.
allowMethods	❌	✅	Allows HTTP methods like GET, POST etc.
return	❌	✅	Redirect to another location.
cgi	❌	✅	Process this request with a CGI program if it targets a suitable file. Specify the file extension and interpreter like Python, PHP, or Perl.
allow_upload	❌	✅	Allow or disallow upload.
upload_store	❌	✅	Location where uploads will be stored.

Further Notes

• cgi Directive

  Takes 2 arguments: file extension (including dot), and path to executable of interpreter. If client sends a request targeted at a file with one of these extensions, it will be executed in a separate process using the provided executable.

   cgi .py /usr/bin/python3;
   cgi .php /usr/bin/php;
  

Limitations & Improvement Potentials

Here are the list of improvement that are not implemented due to time cost and diminishing returns for learning:

1. O(n) poll()

   Backbone of the project is the poll() functionon, providing I/O multiplexing and concurrency. However, it is O(n) instead of O(1) epoll() of Linux or kqueue() of macOS. Project was made on macOS but since this is mainly an educational project, we chose to move forward with the portable option despite the performance tradeoff.

2. Syscall heavy hot path

   Hot path of the server code was not implemented with the minimum amount of system calls in mind.

3. No zero-copy strategy

   Hot path of the server code was not implemented with a zero copy strategy in mind.

4. Filesystem calls in request path

   Hot path of the server code was not implemented with the minimum amount of filesystem calls in mind.

5. String-heavy response pipeline

6. User mode - kernel mode switching on sending

   Due to project requirements, we are forced to use send() instead of sendfile(), being forced to unnecessarily switch between user and kernel mode for each send operation.

7. No HTTP keep-alive

   HTTP keep-alive is not implemented, causing every request to form a new TCP connection with a full TCP handshake and teardown overhead per request.