High-performance SIMD/AVX2-powered HTML and XML data-mining engine for the JVM.
FastWebScrape is the data-extraction substrate of the FastJava web stack. It provides highly-optimized native algorithms to strip formatting blocks, find hyperlinks, extract structured tags, and parse JSON-LD schemas in microseconds—bypassing the latency, memory allocations, and heap pressure of traditional heavy DOM parsers.
import fastwebscrape.FastWebScrape;
import java.nio.charset.StandardCharsets;
import java.util.List;
public class Demo {
public static void main(String[] args) {
// 1. Open high-speed native AVX2 scraper
FastWebScrape scraper = FastWebScrape.open();
byte[] htmlData = ("<html><body><h1>SIMD Architecture</h1>" +
"<p>Vector processing on modern CPUs.</p>" +
"<a href=\"https://en.wikipedia.org/wiki/AVX-512\">AVX-512 Details</a>" +
"</body></html>").getBytes(StandardCharsets.UTF_8);
// 2. Zero-allocation clean plain text extraction for LLMs
String cleanText = scraper.extractReadableText(htmlData);
System.out.println("Clean Text:\n" + cleanText);
// 3. Ultra-fast hyperlink harvesting
List<String> links = scraper.extractLinks(htmlData);
System.out.printf("Harvested %,d hyperlinks in microseconds.\n", links.size());
}
}- Key Features
- Performance
- API Quick Reference
- Installation
- Technical Examples & Hero Demos
- Platform Support
- Modular Ecosystem
- License
- ⚡ SIMD/AVX2 Acceleration: Loads 32-byte chunks into CPU vector registers to skip tags and whitespace instantly.
- 🔍 Zero-Copy Region Locking: Employs
GetPrimitiveArrayCriticalJNI regions to lock the GC and parse Java arrays directly on the native C++ heap. - 🤖 LLM & RAG Optimized: Strips
<script>,<style>, and comments while inserting block layout newlines to form clean readable text. - ⚙️ Dynamic Runtime CPU Detection: Auto-detects AVX2 using
__cpuidat startup with seamless scalar fallback routines for non-AVX2 hardware.
Measured on Intel/AMD x64 Hardware with AVX2 instruction support.
| Operation | Input Size | Java (Regex / Standard) | FastWebScrape Native (0.1.0) | Speedup |
|---|---|---|---|---|
| Text Strip | 5 MB Page | ~210 ms | ~5 ms | 42x |
| Link Scan | 5 MB Page | ~45 ms | ~2 ms | 22x |
| JSON-LD Pull | 5 MB Page | ~38 ms | ~1 ms | 38x |
Note
Speedups scale directly with document size due to AVX2 vector unrolling and zero-heap instantiation during native parsing.
| Method | Description | Target Path |
|---|---|---|
extractReadableText(...) |
Cleans document markup and reformats block spacing for LLMs. | Reference → |
extractLinks(...) |
Scans for anchor elements and aggregates hyper-links natively. | Reference → |
extractByTag(...) |
Finds all elements matching target name and extracts inner content. | Reference → |
extractJsonLD(...) |
Isolates all linked JSON-LD metadata schemas concurrently. | Reference → |
Tip
Use FastWebScrape.open() to obtain the thread-safe native implementation class.
Add the JitPack repository and the dependencies to your pom.xml:
<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
</repositories>
<dependencies>
<!-- FastWebScrape Library -->
<dependency>
<groupId>com.github.andrestubbe</groupId>
<artifactId>FastWebScrape</artifactId>
<version>0.1.3</version>
</dependency>
<!-- FastCore (Required Native Loader) -->
<dependency>
<groupId>com.github.andrestubbe</groupId>
<artifactId>FastCore</artifactId>
<version>0.1.0</version>
</dependency>
<!-- FastRegex -->
<dependency>
<groupId>com.github.andrestubbe</groupId>
<artifactId>FastRegex</artifactId>
<version>0.1.1</version>
</dependency>
</dependencies>repositories {
maven { url 'https://jitpack.io' }
}
dependencies {
implementation 'com.github.andrestubbe:FastWebScrape:0.1.3'
implementation 'com.github.andrestubbe:FastCore:0.1.0'
implementation 'com.github.andrestubbe:FastRegex:0.1.1'
}Download the latest JARs directly to add them to your classpath:
- 📦 FastWebScrape-0.1.3.jar (The Core Library)
- ⚙️ fastcore-0.1.0.jar (The Mandatory Native Loader)
Important
All JARs must be in your classpath for the native JNI calls to function correctly.
Explore the complete source configurations and benchmarks:
- ⚡ Interactive Live Stream Demo: Demo.java (
.\run-demo.bat) — Multi-article live concurrent ingestion and zero-allocation AVX2 text extraction stream. - 📈 Multi-Tier Comparison: Benchmark.java (
.\run-compare.bat) — Races FastWebScrape against standard JDK RegEx across 3 tiers (CleanText, Links, Tags). - 🚀 OpenJDK JMH Benchmark: FastWebScrapeJmhBenchmark.java (
.\run-benchmark.bat) — Formal JMH microbenchmarks measuring ops/ms throughput. - 🧪 Test Suite: FastWebScrapeTest.java — Comprehensive JUnit 5 validation.
Run the hero demo locally from the command line:
.\run-demo.bat| Platform | Status |
|---|---|
| Windows 10/11 (x64) | ✅ Fully Supported (WinHTTP + AVX2 Native) |
| Linux | 🚧 Planned |
| macOS | 🚧 Planned |
Combine FastWebScrape with other accelerators for maximum efficiency:
- FastWebSpider — Native WinHTTP crawler.
- FastCore — Native loading substrate.
- FastBytes — Hardware-aligned byte arrays.
- FastJSON — SIMD-powered JSON parser.
MIT License — See LICENSE file for details.
- FastCore — Native Library Loader for Java
- FastWebScrape — High-performance RawInput engine
- FastTheme — Advanced UI styling engine
Part of the FastJava Ecosystem — Making the JVM faster.