diff --git a/README-NuGet.md b/README-NuGet.md index 6d9fb9f5..39c03ff8 100644 --- a/README-NuGet.md +++ b/README-NuGet.md @@ -1,101 +1,132 @@ +## MiniExcel +[![NuGet Version](https://img.shields.io/nuget/v/MiniExcel.svg)](https://www.nuget.org/packages/MiniExcel)  +[![NuGet Downloads](https://img.shields.io/nuget/dt/MiniExcel.svg)](https://www.nuget.org/packages/MiniExcel)  +[![GitHub Stars](https://img.shields.io/github/stars/mini-software/MiniExcel?logo=github)](https://github.com/mini-software/MiniExcel)  +[![Gitee Stars](https://gitee.com/dotnetchina/MiniExcel/badge/star.svg)](https://gitee.com/dotnetchina/MiniExcel)  +[![.NET Version](https://img.shields.io/badge/.NET-%3E%3D%204.5-red.svg)](https://www.nuget.org/packages/MiniExcel)  +[![DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/mini-software/MiniExcel) This project is part of the [.NET Foundation](https://dotnetfoundation.org/projects/project-detail/miniexcel) and operates under their code of conduct. --- -### Introduction +MiniExcel is a simple and efficient Excel processing tool for .NET, specifically designed to minimize memory usage. -MiniExcel is simple and efficient to avoid OOM's .NET processing Excel tool. +At present, most popular frameworks need to load all the data from an Excel document into memory to facilitate operations, but this may cause memory consumption problems. MiniExcel's approach is different: the data is processed row by row in a streaming manner, reducing the original consumption from potentially hundreds of megabytes to just a few megabytes, effectively preventing out-of-memory(OOM) issues. -At present, most popular frameworks need to load all the data into the memory to facilitate operation, but it will cause memory consumption problems. MiniExcel tries to use algorithm from a stream to reduce the original 1000 MB occupation to a few MB to avoid OOM(out of memory). +```mermaid +flowchart LR + A1(["Excel analysis
process"]) --> A2{{"Unzipping
XLSX file"}} --> A3{{"Parsing
OpenXML"}} --> A4{{"Model
conversion"}} --> A5(["Output"]) -![image](https://user-images.githubusercontent.com/12729184/113086657-ab8bd000-9214-11eb-9563-c970ac1ee35e.png) + B1(["Other Excel
Frameworks"]) --> B2{{"Memory"}} --> B3{{"Memory"}} --> B4{{"Workbooks &
Worksheets"}} --> B5(["All rows at
the same time"]) + C1(["MiniExcel"]) --> C2{{"Stream"}} --> C3{{"Stream"}} --> C4{{"POCO or dynamic"}} --> C5(["Deferred execution
row by row"]) -### Features + classDef analysis fill:#D0E8FF,stroke:#1E88E5,color:#0D47A1,font-weight:bold; + classDef others fill:#FCE4EC,stroke:#EC407A,color:#880E4F,font-weight:bold; + classDef miniexcel fill:#E8F5E9,stroke:#388E3C,color:#1B5E20,font-weight:bold; -- Low memory consumption, avoid OOM (out of memory) and full GC -- Supports real time operation of each row of data -- Supports LINQ deferred execution, it can do low-consumption, fast paging and other complex queries -- Lightweight, without Microsoft Office installed, no COM+, DLL size is less than 400KB -- Easy API style to read/write/fill excel + class A1,A2,A3,A4,A5 analysis; + class B1,B2,B3,B4,B5 others; + class C1,C2,C3,C4,C5 miniexcel; +``` -### Get Started +### Features -- [Import/Query Excel](#getstart1) +- Minimizes memory consumption, preventing out-of-memory (OOM) errors and avoiding full garbage collections +- Enables real-time, row-level data operations for better performance on large datasets +- Supports LINQ with deferred execution, allowing for fast, memory-efficient paging and complex queries +- Lightweight, without the need for Microsoft Office or COM+ components, and a size under 800KB +- Simple and intuitive API to import, export, and template Excel worksheets -- [Export/Create Excel](#getstart2) +### Quickstart -- [Excel Template](#getstart3) +#### Importing -- [Excel Column Name/Index/Ignore Attribute](#getstart4) +You can query worksheets and map the data either to strongly typed classes or dynamic objects: -- [Examples](#getstart5) +```csharp +public class UserAccount +{ + public Guid ID { get; set; } + public string Name { get; set; } + public DateTime DateOfBirth { get; set; } + public int Age { get; set; } + public bool Vip { get; set; } + public decimal Points { get; set; } +} +var userRows = MiniExcel.Query(path); +// or simply -### Installation +var rows = MiniExcel.Query(path); +``` -You can install the package [from NuGet](https://www.nuget.org/packages/MiniExcel) +#### Exporting -### Release Notes +There are multiple ways to export data to an Excel document: -Please Check [Release Notes](docs) +```csharp +// From strongly typed objects -### TODO +var values = new[] +{ + new { Name = "MiniExcel", Value = 1 }, + new { Name = "Github", Value = 2 } +}; +MiniExcel.SaveAs(yourPath, values); -Please Check [TODO](https://github.com/mini-software/MiniExcel/projects/1?fullscreen=true) -### Performance +// From anonymous objects -The code for the benchmarks can be found in [MiniExcel.Benchmarks](https://github.com/mini-software/MiniExcel/tree/master/benchmarks/MiniExcel.Benchmarks). -To run all the benchmarks use: +public class TestType +{ + public string Name { get; set; } + public int Value { get; set; } +} -```bash -dotnet run -project .\benchmarks\MiniExcel.Benchmarks -c Release -f net9.0 -filter * --join -``` +TestType[] values = +[ + new TestType { Name = "MiniExcel", Value = 1 }, + new TestType { Name = "Github", Value = 2 } +]; +MiniExcel.SaveAs(yourPath, values); -Hardware and settings used are the following: -``` -BenchmarkDotNet v0.15.0, Linux Ubuntu 24.04.2 LTS (Noble Numbat) -AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores -.NET SDK 9.0.300 - [Host] : .NET 9.0.5 (9.0.525.21509), X64 RyuJIT AVX2 - ShortRun : .NET 9.0.5 (9.0.525.21509), X64 RyuJIT AVX2 -``` -#### Import/Query Excel +//From a IEnumerable> -The file used to test performance is [**Test1,000,000x10.xlsx**](https://github.com/mini-software/MiniExcel/tree/master/benchmarks/MiniExcel.Benchmarks/Test1%2C000%2C000x10.xlsx), a 32MB document containing 1,000,000 rows * 10 columns whose cells are filled with the string "HelloWorld". +List>() dicts = +[ + new Dictionary { { "Name", "MiniExcel" }, { "Value", 1 } }, + new Dictionary { { "Name", "Github" }, { "Value", 2 } } +]; +MiniExcel.SaveAs(yourPath, dicts); -| Method | Mean | StdDev | Error | Gen0 | Gen1 | Gen2 | Allocated | -|--------------------------------------|-----------------:|---------------:|-----------------:|------------:|------------:|----------:|--------------:| -| 'MiniExcel QueryFirst' | 63.70 μs | 0.337 μs | 6.144 μs | 2.9297 | 2.7669 | - | 49.67 KB | -| 'ExcelDataReader QueryFirst' | 5,010,679.51 μs | 53,245.186 μs | 971,390.400 μs | 105000.0000 | 333.3333 | - | 1717272.56 KB | -| 'MiniExcel Query' | 9,172,286.91 μs | 12,805.326 μs | 233,616.824 μs | 448500.0000 | 4666.6667 | - | 7327883.36 KB | -| 'ExcelDataReader Query' | 10,609,617.09 μs | 29,055.953 μs | 530,088.745 μs | 275666.6667 | 68666.6667 | - | 4504691.87 KB | -| 'Epplus QueryFirst' | 13,770,656.24 μs | 45,909.809 μs | 837,565.827 μs | 174333.3333 | 88833.3333 | 4333.3333 | 3700587.76 KB | -| 'Epplus Query' | 19,257,306.83 μs | 63,117.956 μs | 1,151,506.486 μs | 452333.3333 | 90500.0000 | 5333.3333 | 8223933.16 KB | -| 'ClosedXml Query' | 31,070,263.83 μs | 342,973.671 μs | 6,257,116.502 μs | 401666.6667 | 104166.6667 | 3333.3333 | 6822559.68 KB | -| 'ClosedXml QueryFirst' | 31,141,877.48 μs | 21,006.538 μs | 383,237.459 μs | 402166.6667 | 104833.3333 | 3833.3333 | 6738357.8 KB | -| 'OpenXmlSDK QueryFirst' | 31,750,686.63 μs | 263,328.569 μs | 4,804,093.357 μs | 374666.6667 | 374500.0000 | 3166.6667 | 6069266.96 KB | -| 'OpenXmlSDK Query' | 32,919,119.46 μs | 411,395.682 μs | 7,505,388.691 μs | 374666.6667 | 374500.0000 | 3166.6667 | 6078467.83 KB | +// Directly from a IDataReader -#### Export/Create Excel +using var connection = yourConnectionProvider.GetConnection(); +connection.Open(); -Logic: create a total of 10,000,000 "HelloWorld" cells Excel document +using var cmd = connection.CreateCommand(); +cmd.CommandText = """ + SELECT 'MiniExcel' AS "Name", 1 AS "Value" + UNION ALL + SELECT 'Github', 2 + """; -| Method | Mean | StdDev | Error | Gen0 | Gen1 | Gen2 | Allocated | -|----------------------------------------------|---------:|---------:|---------:|------------:|------------:|----------:|----------:| -| 'MiniExcel Create Xlsx' | 4.427 s | 0.0056 s | 0.1023 s | 251666.6667 | 1833.3333 | 1666.6667 | 3.92 GB | -| 'OpenXmlSdk Create Xlsx by DOM mode' | 22.729 s | 0.1226 s | 2.2374 s | 307000.0000 | 306833.3333 | 3833.3333 | 6.22 GB | -| 'ClosedXml Create Xlsx' | 22.851 s | 0.0190 s | 0.3473 s | 195500.0000 | 54500.0000 | 4166.6667 | 4.48 GB | -| 'Epplus Create Xlsx' | 23.027 s | 0.0088 s | 0.1596 s | 89000.0000 | 17500.0000 | 6000.0000 | 2.51 GB | +using var reader = cmd.ExecuteReader(); +MiniExcel.SaveAs(yourPath, reader); -Warning: these results may be outdated. You can find the benchmarks for the latest release [here](https://github.com/mini-software/MiniExcel/tree/master/benchmarks/results). +// From a DataTable -### Documents +var table = new DataTable(); +table.Columns.Add("Name", typeof(string)); +table.Columns.Add("Value", typeof(int)); +table.Rows.Add("MiniExcel", 1); +table.Rows.Add("Github", 2); -https://github.com/mini-software/MiniExcel +MiniExcel.SaveAs(path, table); +``` diff --git a/src/MiniExcel/Csv/CsvWriter.cs b/src/MiniExcel/Csv/CsvWriter.cs index cc67e1cc..59336ce4 100644 --- a/src/MiniExcel/Csv/CsvWriter.cs +++ b/src/MiniExcel/Csv/CsvWriter.cs @@ -104,12 +104,12 @@ private async Task WriteValuesAsync(StreamWriter writer, object values, str cancellationToken.ThrowIfCancellationRequested(); IMiniExcelWriteAdapter writeAdapter = null; -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 IAsyncMiniExcelWriteAdapter asyncWriteAdapter = null; #endif try { -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 if (!MiniExcelWriteAdapterFactory.TryGetAsyncWriteAdapter(values, _configuration, out asyncWriteAdapter)) { writeAdapter = MiniExcelWriteAdapterFactory.GetWriteAdapter(values, _configuration); @@ -156,7 +156,7 @@ private async Task WriteValuesAsync(StreamWriter writer, object values, str rowsWritten++; } } -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 else { await foreach (var row in asyncWriteAdapter.GetRowsAsync(props, cancellationToken)) @@ -182,7 +182,7 @@ private async Task WriteValuesAsync(StreamWriter writer, object values, str } finally { -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 if (asyncWriteAdapter is IAsyncDisposable asyncDisposable) { await asyncDisposable.DisposeAsync().ConfigureAwait(false); diff --git a/src/MiniExcel/MiniExcelLibs.csproj b/src/MiniExcel/MiniExcelLibs.csproj index 6961225b..6ce9db3d 100644 --- a/src/MiniExcel/MiniExcelLibs.csproj +++ b/src/MiniExcel/MiniExcelLibs.csproj @@ -1,7 +1,7 @@  - net45;netstandard2.0;net8.0;net9.0;net10.0 - 1.44.1 + net45;net461;netstandard2.0;net8.0;net9.0;net10.0 + 1.45.0 14 enable MiniExcelLibs @@ -13,11 +13,7 @@ MiniExcel MiniExcel excel;xlsx;csv;micro-helper;mini;openxml;helper; - Fast, Low-Memory, Easy Excel .NET helper to import/export/template spreadsheet -Github : https://github.com/mini-software/MiniExcel -Gitee : https://gitee.com/dotnetchina/MiniExcel -Issues : https://github.com/mini-software/MiniExcel/issues -Todo : https://github.com/mini-software/MiniExcel/projects/1?fullscreen=true + Lightweight, fast and simple .NET processing tool for importing, exporting and templating spreadsheets. Wei Lin, Michele Bastione, PING-HSIU SHIH, Amos(izanhzh), eynarhaji, Mini-Software team MiniExcel Wei Lin, 2021 onwards @@ -43,13 +39,15 @@ Todo : https://github.com/mini-software/MiniExcel/projects/1?fullscreen=true - - - - + + + + + - - + + + diff --git a/src/MiniExcel/OpenXml/ExcelOpenXmlSheetWriter.Async.cs b/src/MiniExcel/OpenXml/ExcelOpenXmlSheetWriter.Async.cs index 2a22a243..fb389738 100644 --- a/src/MiniExcel/OpenXml/ExcelOpenXmlSheetWriter.Async.cs +++ b/src/MiniExcel/OpenXml/ExcelOpenXmlSheetWriter.Async.cs @@ -171,12 +171,12 @@ private async Task WriteValuesAsync(MiniExcelAsyncStreamWriter writer, obje cancellationToken.ThrowIfCancellationRequested(); IMiniExcelWriteAdapter writeAdapter = null; -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 IAsyncMiniExcelWriteAdapter asyncWriteAdapter = null; #endif try { -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 if (!MiniExcelWriteAdapterFactory.TryGetAsyncWriteAdapter(values, _configuration, out asyncWriteAdapter)) { writeAdapter = MiniExcelWriteAdapterFactory.GetWriteAdapter(values, _configuration); @@ -261,7 +261,7 @@ await WriteCellAsync(writer, currentRowIndex, cellValue.CellIndex, cellValue.Val await writer.WriteAsync(WorksheetXml.EndRow); } } -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 else { await foreach (var row in asyncWriteAdapter.GetRowsAsync(props, cancellationToken)) @@ -309,7 +309,7 @@ await OverWriteColumnWidthPlaceholdersAsync(writer, columnWidthsPlaceholderPosit } finally { -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 if (asyncWriteAdapter is IAsyncDisposable asyncDisposable) { await asyncDisposable.DisposeAsync().ConfigureAwait(false); diff --git a/src/MiniExcel/Utils/ImageHelper.cs b/src/MiniExcel/Utils/ImageHelper.cs index 820b553b..19f9f156 100644 --- a/src/MiniExcel/Utils/ImageHelper.cs +++ b/src/MiniExcel/Utils/ImageHelper.cs @@ -2,84 +2,57 @@ internal static class ImageHelper { - public enum ImageFormat - { - bmp, - jpg, - gif, - tiff, - png, - unknown - } - -#if NET45||NETSTANDARD2_0 - public static ImageFormat GetImageFormat(byte[] bytes) - { - var bmp = new byte[] { (byte)'B', (byte)'M' }; // BMP - var gif = new byte[] { (byte)'G', (byte)'I', (byte)'F' }; // GIF - var png = new byte[] { 137, 80, 78, 71 }; // PNG - var tiff = new byte[] { 73, 73, 42 }; // TIFF - var tiff2 = new byte[] { 77, 77, 42 }; // TIFF - var jpeg = new byte[] { 255, 216, 255, 224 }; // jpeg - var jpeg2 = new byte[] { 255, 216, 255, 225 }; // jpeg canon - - if (bytes.StartsWith(bmp)) - return ImageFormat.bmp; - - if (bytes.StartsWith(gif)) - return ImageFormat.gif; - - if (bytes.StartsWith(png)) - return ImageFormat.png; - - if (bytes.StartsWith(tiff)) - return ImageFormat.tiff; - - if (bytes.StartsWith(tiff2)) - return ImageFormat.tiff; - - if (bytes.StartsWith(jpeg)) - return ImageFormat.jpg; - - if (bytes.StartsWith(jpeg2)) - return ImageFormat.jpg; +#if NET + private static ReadOnlySpan Bmp => "BM"u8; // BMP + private static ReadOnlySpan Gif => "GIF"u8; // GIF + private static ReadOnlySpan Png => [137, 80, 78, 71]; // PNG + private static ReadOnlySpan Tiff => "II*"u8; // TIFF + private static ReadOnlySpan Tiff2 => "MM*"u8; // TIFF + private static ReadOnlySpan Jpeg => [255, 216, 255, 224]; // JPEG + private static ReadOnlySpan Jpeg2 => [255, 216, 255, 225]; // JPEG canon +#else + private static readonly byte[] Bmp = [(byte)'B', (byte)'M']; // BMP + private static readonly byte[] Gif = [(byte)'G', (byte)'I', (byte)'F']; // GIF + private static readonly byte[] Png = [137, 80, 78, 71]; // PNG + private static readonly byte[] Tiff = [(byte)'I', (byte)'I', (byte)'*']; // TIFF + private static readonly byte[] Tiff2 = [(byte)'M', (byte)'M', (byte)'*']; // TIFF + private static readonly byte[] Jpeg = [255, 216, 255, 224]; // JPEG + private static readonly byte[] Jpeg2 = [255, 216, 255, 225]; // JPEG canon +#endif - return ImageFormat.unknown; - } - -#elif NET5_0_OR_GREATER - public static ImageFormat GetImageFormat(ReadOnlySpan bytes) + public static ImageFormat GetImageFormat( +#if NET + ReadOnlySpan bytes +#else + byte[] bytes +#endif + ) { - ReadOnlySpan bmp = stackalloc byte[] { (byte)'B', (byte)'M' }; // BMP - ReadOnlySpan gif = stackalloc byte[] { (byte)'G', (byte)'I', (byte)'F' }; // GIF - ReadOnlySpan png = stackalloc byte[] { 137, 80, 78, 71 }; // PNG - ReadOnlySpan tiff = stackalloc byte[] { 73, 73, 42 }; // TIFF - ReadOnlySpan tiff2 = stackalloc byte[] { 77, 77, 42 }; // TIFF - ReadOnlySpan jpeg = stackalloc byte[] { 255, 216, 255, 224 }; // jpeg - ReadOnlySpan jpeg2 = stackalloc byte[] { 255, 216, 255, 225 }; // jpeg canon - - if (bytes.StartsWith(bmp)) + if (bytes.StartsWith(Bmp)) return ImageFormat.bmp; - if (bytes.StartsWith(gif)) + if (bytes.StartsWith(Gif)) return ImageFormat.gif; - if (bytes.StartsWith(png)) + if (bytes.StartsWith(Png)) return ImageFormat.png; - if (bytes.StartsWith(tiff)) - return ImageFormat.tiff; - - if (bytes.StartsWith(tiff2)) + if (bytes.StartsWith(Tiff) || bytes.StartsWith(Tiff2)) return ImageFormat.tiff; - if (bytes.StartsWith(jpeg)) - return ImageFormat.jpg; - - if (bytes.StartsWith(jpeg2)) + if (bytes.StartsWith(Jpeg) || bytes.StartsWith(Jpeg2)) return ImageFormat.jpg; return ImageFormat.unknown; } -#endif -} \ No newline at end of file + + public enum ImageFormat + { + bmp, + jpg, + gif, + tiff, + png, + unknown + } +} diff --git a/src/MiniExcel/Utils/TypeHelper.cs b/src/MiniExcel/Utils/TypeHelper.cs index 38aa5215..2a28a059 100644 --- a/src/MiniExcel/Utils/TypeHelper.cs +++ b/src/MiniExcel/Utils/TypeHelper.cs @@ -245,7 +245,7 @@ public static bool IsNumericType(Type type, bool isNullableUnderlyingType = fals return newValue; } -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 public static bool IsAsyncEnumerable(this Type type, out Type genericArgument) { var asyncEnumrableInterfaceType = type diff --git a/src/MiniExcel/WriteAdapter/AsyncEnumerableWriteAdapter.cs b/src/MiniExcel/WriteAdapter/AsyncEnumerableWriteAdapter.cs index 1020aa0f..3664dd19 100644 --- a/src/MiniExcel/WriteAdapter/AsyncEnumerableWriteAdapter.cs +++ b/src/MiniExcel/WriteAdapter/AsyncEnumerableWriteAdapter.cs @@ -2,7 +2,7 @@ using System.Collections; using System.Runtime.CompilerServices; -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 namespace MiniExcelLibs.WriteAdapter; internal class AsyncEnumerableWriteAdapter : IAsyncMiniExcelWriteAdapter, IAsyncDisposable diff --git a/src/MiniExcel/WriteAdapter/IAsyncMiniExcelWriteAdapter.cs b/src/MiniExcel/WriteAdapter/IAsyncMiniExcelWriteAdapter.cs index 28267863..94c1fcdc 100644 --- a/src/MiniExcel/WriteAdapter/IAsyncMiniExcelWriteAdapter.cs +++ b/src/MiniExcel/WriteAdapter/IAsyncMiniExcelWriteAdapter.cs @@ -1,6 +1,6 @@ using MiniExcelLibs.Utils; -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 namespace MiniExcelLibs.WriteAdapter { internal interface IAsyncMiniExcelWriteAdapter diff --git a/src/MiniExcel/WriteAdapter/MiniExcelDataReaderWriteAdapter.cs b/src/MiniExcel/WriteAdapter/MiniExcelDataReaderWriteAdapter.cs index 8e8b94f1..6fdbfa3a 100644 --- a/src/MiniExcel/WriteAdapter/MiniExcelDataReaderWriteAdapter.cs +++ b/src/MiniExcel/WriteAdapter/MiniExcelDataReaderWriteAdapter.cs @@ -1,7 +1,7 @@ using MiniExcelLibs.Utils; using System.Runtime.CompilerServices; -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 namespace MiniExcelLibs.WriteAdapter; internal class MiniExcelDataReaderWriteAdapter : IAsyncMiniExcelWriteAdapter diff --git a/src/MiniExcel/WriteAdapter/MiniExcelWriteAdapterFactory.cs b/src/MiniExcel/WriteAdapter/MiniExcelWriteAdapterFactory.cs index 96d3f3ed..32b6fee7 100644 --- a/src/MiniExcel/WriteAdapter/MiniExcelWriteAdapterFactory.cs +++ b/src/MiniExcel/WriteAdapter/MiniExcelWriteAdapterFactory.cs @@ -6,7 +6,7 @@ namespace MiniExcelLibs.WriteAdapter; internal static class MiniExcelWriteAdapterFactory { -#if NETSTANDARD2_0_OR_GREATER || NET +#if !NET45 public static bool TryGetAsyncWriteAdapter(object values, Configuration configuration, out IAsyncMiniExcelWriteAdapter writeAdapter) { writeAdapter = null;