WebGPU Video Processor

A high-performance, GPU-accelerated video processing library for web applications. Built with WebGPU and WebGL2 fallback support.

Overview

WebGPU Video Processor is an open-source library that enables efficient video processing directly in the browser using modern GPU acceleration. It provides a simple API for common video processing tasks while leveraging the power of WebGPU (with WebGL2 fallback) for optimal performance.

Key Features

🚀 GPU-accelerated video processing
🎨 Easy-to-use API for common video effects
🔄 Automatic WebGL2 fallback for broader browser support
🎯 High-performance frame processing
🛠️ Extensible architecture for custom effects
📦 Zero dependencies

Current Status

Core GPU operations implemented
WebGL2 fallback support
Comprehensive test coverage
Cross-browser compatibility
GPU testing utilities

Use Cases

Logo overlay on videos
Caption burning
Background image compositing
Layout transformations
Color space conversions
Frame rate handling
Real-time video effects

Technical Details

Architecture

The library is built with a layered architecture:

Core Layer: Handles GPU initialization and resource management
Processing Layer: Implements video processing algorithms
Effects Layer: Provides pre-built effects and transformations
API Layer: Exposes a simple, developer-friendly interface

Technology Stack

Primary: WebGPU (Modern GPU API)
Fallback: WebGL2
Language: TypeScript
Build Tool: Vite

Performance Considerations

Minimizes CPU/GPU memory copies
Leverages GPU parallelization
Efficient resource management
Automatic format conversion optimization

Current Implementation

interface VideoProcessorOptions {
  width: number;
  height: number;
  debug?: boolean;
}

class VideoProcessor {
  constructor(options: VideoProcessorOptions);
  async checkGPUSupport(): Promise<boolean>;
  async destroy(): Promise<void>;
}

Project Structure

webgpu-video-processor/
├── src/
│   ├── core/           # Core GPU operations
│   ├── utils/          # Utility functions
│   └── types/          # TypeScript definitions
├── tests/
│   ├── unit/          # Jest tests
│   └── e2e/           # Playwright tests
├── .github/           # GitHub Actions
└── docs/             # Documentation

Features

🚀 GPU-accelerated video processing using WebGPU
🔄 Automatic WebGL2 fallback for broader browser support
🎨 Real-time video effects and filters
📦 Zero dependencies
🔍 Comprehensive test coverage
📱 Cross-browser compatibility

Installation

npm install webgpu-video-processor

Quick Start

import { VideoProcessor } from 'webgpu-video-processor';

// Initialize the video processor
const processor = new VideoProcessor({
  width: 1920,
  height: 1080,
  useWebGPU: true // Automatically falls back to WebGL2 if WebGPU is not available
});

// Process a video frame
const outputTexture = await processor.processFrame(inputTexture);

// Apply effects
await processor.applyEffect('blur', { radius: 5 });
await processor.applyEffect('colorAdjust', { brightness: 1.2 });

Browser Support

Chrome/Chromium 113+: WebGPU support (primary)
Edge 113+: WebGPU support
Firefox: WebGL2 fallback
Safari 16.4+: WebGL2 fallback

Development

Prerequisites

Node.js 20 or higher
npm 9 or higher
Modern web browser with WebGPU support

Setup

Clone the repository:

git clone https://github.com/yourusername/webgpu-video-processor.git
cd webgpu-video-processor

Install dependencies:

npm install

Start development server:

npm run dev

Testing Infrastructure

Unit Tests (Jest)

npm test              # Run all unit tests
npm run test:watch    # Run tests in watch mode
npm run test:coverage # Run tests with coverage report

End-to-End Tests (Playwright)

npm run test:e2e      # Run all E2E tests
npm run test:e2e:ui   # Run E2E tests with UI
npm run test:e2e:debug # Run E2E tests in debug mode

Building

npm run build

The build output will be available in the dist directory.

API Documentation

VideoProcessor

The main class for video processing operations.

Constructor Options

interface VideoProcessorOptions {
  width: number;
  height: number;
  useWebGPU?: boolean;  // Default: true
  fallbackToWebGL2?: boolean;  // Default: true
  debug?: boolean;  // Default: false
}

Methods

processFrame(inputTexture: GPUTexture): Promise<GPUTexture>
applyEffect(effectName: string, options: EffectOptions): Promise<void>
destroy(): void

Available Effects

blur: Gaussian blur effect
colorAdjust: Color adjustment (brightness, contrast, saturation)
edgeDetection: Edge detection filter
sharpen: Image sharpening
denoise: Noise reduction

Detailed API Reference

Core Methods

class VideoProcessor {
  constructor(options: VideoProcessorOptions);
  
  // Core methods
  async init(): Promise<void>;
  async destroy(): Promise<void>;
  async checkGPUSupport(): Promise<boolean>;
  
  // Frame processing
  async processFrame(input: HTMLVideoElement | HTMLCanvasElement): Promise<GPUTexture>;
  async renderToCanvas(texture: GPUTexture, canvas: HTMLCanvasElement): Promise<void>;
  
  // Effect management
  async applyEffect(effectName: string, options: EffectOptions): Promise<void>;
  registerEffect(name: string, effect: CustomEffect): void;
  
  // Resource management
  createTexture(descriptor: GPUTextureDescriptor): GPUTexture;
  destroyTexture(texture: GPUTexture): void;
}

Effect Options Interface

interface EffectOptions {
  // Common effect parameters
  intensity?: number;  // Range: 0.0 to 1.0
  
  // Color adjustment parameters
  brightness?: number;  // Range: -1.0 to 1.0
  contrast?: number;   // Range: 0.0 to 2.0
  saturation?: number; // Range: 0.0 to 2.0
  
  // Filter parameters
  filterType?: 'grayscale' | 'sepia' | 'invert';
  
  // Custom effect parameters
  [key: string]: any;
}

Custom Effect Interface

interface CustomEffect {
  shader: string;  // WGSL or GLSL shader code
  uniforms?: {
    [key: string]: {
      type: 'float' | 'vec2' | 'vec3' | 'vec4' | 'mat4';
      default?: number | number[];
    }
  };
  vertexLayout?: GPUVertexBufferLayout[];
}

Method Details

`constructor(options: VideoProcessorOptions)`

Creates a new instance of the VideoProcessor with specified dimensions and options.

`async init(): Promise<void>`

Initializes the GPU context and sets up necessary resources. Must be called before any processing.

`async destroy(): Promise<void>`

Cleans up all GPU resources and destroys the processor instance.

`async checkGPUSupport(): Promise<boolean>`

Checks if the current environment supports WebGPU or WebGL2.

`async processFrame(input: HTMLVideoElement | HTMLCanvasElement): Promise<GPUTexture>`

Processes a single frame from the input source and returns a GPU texture.

`async renderToCanvas(texture: GPUTexture, canvas: HTMLCanvasElement): Promise<void>`

Renders a processed texture to a canvas element.

`async applyEffect(effectName: string, options: EffectOptions): Promise<void>`

Applies a pre-built or custom effect to the current frame.

`registerEffect(name: string, effect: CustomEffect): void`

Registers a custom effect for use with the processor.

Usage Examples

Color Adjustment

await processor.applyEffect('colorAdjust', {
  brightness: 0.5,   // Range: -1.0 to 1.0
  contrast: 1.2,     // Range: 0.0 to 2.0
  saturation: 1.5    // Range: 0.0 to 2.0
});

Filter Effects

await processor.applyEffect('filter', {
  filterType: 'grayscale',
  intensity: 0.8
});

Custom Effects

processor.registerEffect('customEffect', {
  shader: `
    @vertex
    fn vertexMain(@location(0) position: vec2f) -> @builtin(position) vec4f {
      return vec4f(position, 0.0, 1.0);
    }
    
    @fragment
    fn fragmentMain(@location(0) uv: vec2f) -> @location(0) vec4f {
      return vec4f(uv.x, uv.y, 0.0, 1.0);
    }
  `,
  uniforms: {
    intensity: { type: 'float', default: 1.0 }
  }
});

await processor.applyEffect('customEffect', {
  intensity: 0.5
});

Examples

Basic Video Processing

import { VideoProcessor } from 'webgpu-video-processor';

const video = document.querySelector('video');
const canvas = document.querySelector('canvas');
const processor = new VideoProcessor({
  width: video.videoWidth,
  height: video.videoHeight
});

// Process video frames
video.addEventListener('play', async () => {
  while (!video.paused) {
    const frame = await processor.processFrame(video);
    processor.renderToCanvas(frame, canvas);
    await new Promise(resolve => requestAnimationFrame(resolve));
  }
});

Applying Effects

// Apply multiple effects
await processor.applyEffect('blur', { radius: 3 });
await processor.applyEffect('colorAdjust', {
  brightness: 1.1,
  contrast: 1.2,
  saturation: 1.1
});
await processor.applyEffect('sharpen', { amount: 0.5 });

Custom Effects

// Create a custom effect
processor.registerEffect('customEffect', {
  shader: `
    @vertex
    fn vertexMain(@location(0) position: vec2f) -> @builtin(position) vec4f {
      return vec4f(position, 0.0, 1.0);
    }
    
    @fragment
    fn fragmentMain(@location(0) uv: vec2f) -> @location(0) vec4f {
      return vec4f(uv.x, uv.y, 0.0, 1.0);
    }
  `,
  uniforms: {
    intensity: { type: 'float', default: 1.0 }
  }
});

Roadmap

Phase 1 - Foundation (Current)

Project setup
Basic GPU operations
Testing infrastructure
CI/CD pipeline

Phase 2 - Core Features (In Progress)

Video frame processing
Basic effects pipeline
Performance optimization
WebGL2 fallback improvements

Phase 3 - Advanced Features (Planned)

Custom shader support
Advanced video effects
Real-time processing
Performance monitoring

Troubleshooting

Common Issues

WebGPU Not Available
- Ensure you're using a supported browser
- Check if hardware acceleration is enabled
- Try using WebGL2 fallback
Performance Issues
- Monitor GPU memory usage
- Check for unnecessary texture copies
- Optimize shader complexity

Contributing

Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

WebGPU Working Group
WebGL Working Group
Contributors and maintainers

Support

For support, please open an issue in the GitHub repository.

Features

Core Shader Modules

Video Frame Shader
- Basic video frame rendering
- Efficient texture sampling
- Support for both WebGPU and WebGL2
Color Adjustment Shader
- Brightness control
- Contrast adjustment
- Saturation manipulation
- Real-time parameter updates
Filter Effects Shader
- Grayscale filter
- Sepia tone effect
- Color inversion
- Adjustable filter intensity

Installation

npm install webgpu-video-processor

Usage

Basic Video Frame Rendering

import { videoFrameShader } from 'webgpu-video-processor';

// WebGPU usage
const pipeline = await device.createRenderPipelineAsync({
  layout: 'auto',
  vertex: {
    module: device.createShaderModule({
      code: videoFrameShader.webgpu.vertex
    }),
    entryPoint: 'main'
  },
  fragment: {
    module: device.createShaderModule({
      code: videoFrameShader.webgpu.fragment
    }),
    entryPoint: 'main'
  }
});

// WebGL2 usage
const gl = canvas.getContext('webgl2');
const program = createShaderProgram(gl, {
  vertex: videoFrameShader.webgl2.vertex,
  fragment: videoFrameShader.webgl2.fragment
});

Color Adjustments

import { colorAdjustShader } from 'webgpu-video-processor';

// Set color adjustment parameters
const colorParams = {
  brightness: 0.5,  // Range: -1.0 to 1.0
  contrast: 1.2,    // Range: 0.0 to 2.0
  saturation: 1.5   // Range: 0.0 to 2.0
};

Filter Effects

import { filterEffectsShader } from 'webgpu-video-processor';

// Apply filter effect
const filterParams = {
  filterType: 1,     // 1: grayscale, 2: sepia, 3: invert
  intensity: 0.8     // Range: 0.0 to 1.0
};

Development

Setup

git clone https://github.com/yourusername/webgpu-video-processor.git
cd webgpu-video-processor
npm install

Running Tests

npm test                 # Run all tests
npm run test:watch      # Run tests in watch mode

Browser Support

WebGPU: Chrome 113+, Edge 113+, Firefox Nightly
WebGL2: All modern browsers

Contributing

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add some amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Changelog

See CHANGELOG.md for a list of changes and version history.

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
.github		.github
src		src
tests		tests
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.npmignore		.npmignore
.prettierrc		.prettierrc
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
jest.config.ts		jest.config.ts
jest.setup.ts		jest.setup.ts
package-lock.json		package-lock.json
package.json		package.json
playwright.config.ts		playwright.config.ts
tsconfig.json		tsconfig.json
tsconfig.node.json		tsconfig.node.json
vite.config.ts		vite.config.ts

License

unfoundbox/webgpu-video-processor

Folders and files

Latest commit

History

Repository files navigation

WebGPU Video Processor

Overview

Key Features

Current Status

Use Cases

Technical Details

Architecture

Technology Stack

Performance Considerations

Current Implementation

Project Structure

Features

Installation

Quick Start

Browser Support

Development

Prerequisites

Setup

Testing Infrastructure

Unit Tests (Jest)

End-to-End Tests (Playwright)

Building

API Documentation

VideoProcessor

Constructor Options

Methods

Available Effects

Detailed API Reference

Core Methods

Effect Options Interface

Custom Effect Interface

Method Details

constructor(options: VideoProcessorOptions)

async init(): Promise<void>

async destroy(): Promise<void>

async checkGPUSupport(): Promise<boolean>

async processFrame(input: HTMLVideoElement | HTMLCanvasElement): Promise<GPUTexture>

async renderToCanvas(texture: GPUTexture, canvas: HTMLCanvasElement): Promise<void>

async applyEffect(effectName: string, options: EffectOptions): Promise<void>

registerEffect(name: string, effect: CustomEffect): void

Usage Examples

Examples

Basic Video Processing

Applying Effects

Custom Effects

Roadmap

Phase 1 - Foundation (Current)

Phase 2 - Core Features (In Progress)

Phase 3 - Advanced Features (Planned)

Troubleshooting

Common Issues

Contributing

License

Acknowledgments

Support

Features

Core Shader Modules

Installation

Usage

Basic Video Frame Rendering

Color Adjustments

Filter Effects

Development

Setup

Running Tests

Browser Support

Contributing

License

Changelog

About

Resources

License

Code of conduct

Security policy

`constructor(options: VideoProcessorOptions)`

`async init(): Promise<void>`

`async destroy(): Promise<void>`

`async checkGPUSupport(): Promise<boolean>`

`async processFrame(input: HTMLVideoElement | HTMLCanvasElement): Promise<GPUTexture>`

`async renderToCanvas(texture: GPUTexture, canvas: HTMLCanvasElement): Promise<void>`

`async applyEffect(effectName: string, options: EffectOptions): Promise<void>`

`registerEffect(name: string, effect: CustomEffect): void`

Packages