diff --git a/docs/CLI-Usage.md b/docs/CLI-Usage.md new file mode 100644 index 00000000..000e9ef7 --- /dev/null +++ b/docs/CLI-Usage.md @@ -0,0 +1,235 @@ +# CLI Usage + +```sh +pdu [OPTIONS] [FILES]... +``` + +## Arguments + +* `[FILES]...`: List of files and/or directories. + +## Options + + + +### `--json-input` + +Read JSON data from stdin. + + + +### `--json-output` + +Print JSON data instead of an ASCII chart. + + + +### `--bytes-format` + +* _Aliases:_ `-b`. +* _Default:_ `metric`. +* _Choices:_ + - `plain`: Display plain number of bytes without units + - `metric`: Use metric scale, i.e. 1K = 1000B, 1M = 1000K, and so on + - `binary`: Use binary scale, i.e. 1K = 1024B, 1M = 1024K, and so on + +How to display the numbers of bytes. + + + +### `--deduplicate-hardlinks` + +* _Aliases:_ `-H`, `--detect-links`, `--dedupe-links`. + +Detect and subtract the sizes of hardlinks from their parent directory totals. + + + +### `--one-file-system` + +* _Aliases:_ `-x`. + +Skip directories on different filesystems. + + + +### `--top-down` + +Print the tree top-down instead of bottom-up. + + + +### `--align-right` + +Set the root of the bars to the right. + + + +### `--quantity` + +* _Aliases:_ `-q`. +* _Default:_ `block-size`. +* _Choices:_ + - `apparent-size`: Measure apparent sizes + - `block-size`: Measure block sizes (block-count * 512B) + - `block-count`: Count numbers of blocks + +Aspect of the files/directories to be measured. + + + +### `--max-depth` + +* _Aliases:_ `-d`, `--depth`. +* _Default:_ `10`. + +Maximum depth to display the data. Could be either "inf" or a positive integer. + + + +### `--total-width` + +* _Aliases:_ `-w`, `--width`. + +Width of the visualization. + + + +### `--column-width` + +Maximum widths of the tree column and width of the bar column. + + + +### `--min-ratio` + +* _Aliases:_ `-m`. +* _Default:_ `0.01`. + +Minimal size proportion required to appear. + + + +### `--no-sort` + +Do not sort the branches in the tree. + + + +### `--silent-errors` + +* _Aliases:_ `-s`, `--no-errors`. + +Prevent filesystem error messages from appearing in stderr. + + + +### `--progress` + +* _Aliases:_ `-p`. + +Report progress being made at the expense of performance. + + + +### `--threads` + +* _Default:_ `auto`. + +Set the maximum number of threads to spawn. Could be either "auto", "max", or a positive integer. + + + +### `--omit-json-shared-details` + +Do not output `.shared.details` in the JSON output. + + + +### `--omit-json-shared-summary` + +Do not output `.shared.summary` in the JSON output. + + + +### `--help` + +* _Aliases:_ `-h`. + +Print help. + + + +### `--version` + +* _Aliases:_ `-V`. + +Print version. + +## Examples + +### Show disk usage chart of current working directory + +```sh +pdu +``` + +### Show disk usage chart of a single file or directory + +```sh +pdu path/to/file/or/directory +``` + +### Compare disk usages of multiple files and/or directories + +```sh +pdu file.txt dir/ +``` + +### Show chart in apparent sizes instead of block sizes + +```sh +pdu --quantity=apparent-size +``` + +### Detect and subtract the sizes of hardlinks from their parent nodes + +```sh +pdu --deduplicate-hardlinks +``` + +### Show sizes in plain numbers instead of metric units + +```sh +pdu --bytes-format=plain +``` + +### Show sizes in base 2¹⁰ units (binary) instead of base 10³ units (metric) + +```sh +pdu --bytes-format=binary +``` + +### Show disk usage chart of all entries regardless of size + +```sh +pdu --min-ratio=0 +``` + +### Only show disk usage chart of entries whose size is at least 5% of total + +```sh +pdu --min-ratio=0.05 +``` + +### Show disk usage data as JSON instead of chart + +```sh +pdu --min-ratio=0 --max-depth=inf --json-output | jq +``` + +### Visualize existing JSON representation of disk usage data + +```sh +pdu --json-input < disk-usage.json +``` diff --git a/docs/Library-Building-Trees.md b/docs/Library-Building-Trees.md new file mode 100644 index 00000000..86ac54dc --- /dev/null +++ b/docs/Library-Building-Trees.md @@ -0,0 +1,121 @@ +# Building Trees + +A [`DataTree`](Library-DataTree.md) comes from one of three places: `FsTreeBuilder` for the real filesystem, `TreeBuilder` for any other hierarchy, or a +[`Reflection`](Library-DataTree.md#reflection). This page covers the first two. + +## `FsTreeBuilder` + +`fs_tree_builder::FsTreeBuilder` walks a real directory tree. It is a struct of parameters, and the traversal is performed by its `From` implementation, so measurement starts at `.into()`. + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::device::DeviceBoundary; +use parallel_disk_usage::fs_tree_builder::FsTreeBuilder; +use parallel_disk_usage::get_size::GetApparentSize; +use parallel_disk_usage::hardlink::HardlinkIgnorant; +use parallel_disk_usage::os_string_display::OsStringDisplay; +use parallel_disk_usage::reporter::{ErrorOnlyReporter, ErrorReport}; +use parallel_disk_usage::size::Bytes; + +let builder = FsTreeBuilder { +root: "/usr/share".into(), +size_getter: GetApparentSize, +hardlinks_recorder: & HardlinkIgnorant, +reporter: & ErrorOnlyReporter::new(ErrorReport::SILENT), +device_boundary: DeviceBoundary::Cross, +max_depth: 10, +}; + +let data_tree: DataTree = builder.into(); +``` + +### Fields + +| Field | Type | Meaning | +|----------------------|---------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| +| `root` | `PathBuf` | The directory or file at the top of the walk. Becomes the name of the root node. | +| `size_getter` | `impl GetSize + Sync` | Decides what a file's size is. See [Sizes and Formatting](Library-Sizes-And-Formatting.md#getsize). | +| `hardlinks_recorder` | `&impl RecordHardlinks` | Detects and records hardlinks during the walk. Use `&HardlinkIgnorant` to skip detection. See [Hardlinks](Library-Hardlinks.md). | +| `reporter` | `&impl Reporter + Sync` | Receives progress and error events. See [Reporters](Library-Reporters.md). | +| `device_boundary` | `DeviceBoundary` | `Cross` descends into other filesystems, `Stay` does not. | +| `max_depth` | `u64` | Deepest level retained as nodes. | + +`From>` is implemented for `DataTree`, where `Size` comes from the `size_getter`. The name type is always `OsStringDisplay`. Annotate the destination binding explicitly to pin both down. + +### `max_depth` + +`max_depth` limits the depth of the *stored* tree, not the depth of the *walk*. Everything below the cutoff is measured and contributes to its ancestors' totals, but is not kept as a separate node. A +`max_depth` of `1` yields a childless root whose size is the total of the whole tree. Use +`u64::MAX` for no limit. + +Lowering `max_depth` therefore reduces memory usage but not traversal time. + +### `device_boundary` + +`DeviceBoundary::Stay` reproduces `--one-file-system`. It reads the device ID of `root` once, then skips any directory whose device ID differs. If `root` cannot be stated, the builder reports a +`SymlinkMetadata` error and returns a single zero-sized file node. + +`DeviceBoundary::Cross` performs no device check. + +### Errors + +`FsTreeBuilder` never fails as a whole. Every I/O problem becomes an `Event::EncounterError` handed to the reporter, and the affected subtree is recorded with a size of zero and no children. The three failing operations are `SymlinkMetadata`, `ReadDirectory`, and `AccessEntry`. See +[Reporters](Library-Reporters.md#error-reports). + +The walk uses `symlink_metadata`, so symbolic links are measured as links and never followed. + +## `TreeBuilder` + +`tree_builder::TreeBuilder` is the generic engine underneath `FsTreeBuilder`. It knows nothing about the filesystem: supply two closures and it performs the parallel recursion. Use it for a hierarchy that is not a directory tree, such as an archive index, an object store listing, or a test fixture. + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::size::Bytes; +use parallel_disk_usage::tree_builder::{Info, TreeBuilder}; + +let builder = TreeBuilder:: { +path: "root".to_string(), +name: "root".to_string(), +get_info: | path| Info { +size: Bytes::new(path.len() as u64), +children: Vec::new(), +}, +join_path: | prefix, name | format !("{prefix}/{name}"), +max_depth: 10, +}; + +let data_tree: DataTree = builder.into(); +``` + +### Fields + +| Field | Meaning | +|-------------|---------------------------------------------------------------------------------------------| +| `path` | The address of the root in whatever address space you are walking. | +| `name` | The label of the root node. `Path` and `Name` may be different types. | +| `get_info` | `Fn(&Path) -> Info`. Returns the node's own size and the names of its children. | +| `join_path` | `Fn(&Path, &Name) -> Path`. Combines a parent path with a child name. | +| `max_depth` | Same meaning as in `FsTreeBuilder`. | + +### `Info` + +```rust +pub struct Info { + pub size: Size, + pub children: Vec, +} +``` + +`size` is the node's own size, excluding children. `TreeBuilder` recurses into each child and +`DataTree::dir` adds their totals on top. + +### Closure requirements + +Both closures are `Copy + Send + Sync`, since they are cloned into every Rayon task. That rules out closures capturing by mutable reference or owning non-`Sync` state. For shared mutable state, capture a shared reference to something internally synchronized, as `FsTreeBuilder` does with its reporter and hardlink recorder. + +`get_info` has no error channel and must not panic. Report failures out of band and return +`Info { size: default, children: vec![] }`, again following `FsTreeBuilder`. + +## Choosing between the two + +Use `FsTreeBuilder` for real directories. Reach for `TreeBuilder` when the source is not a filesystem, or when you need name and path types that `FsTreeBuilder` fixes for you. diff --git a/docs/Library-DataTree.md b/docs/Library-DataTree.md new file mode 100644 index 00000000..a84df6f9 --- /dev/null +++ b/docs/Library-DataTree.md @@ -0,0 +1,192 @@ +# DataTree + +`data_tree::DataTree` is the measured tree. Every other part of the library produces one, transforms one, or renders one. + +```rust +pub struct DataTree { + name: Name, + size: Size, + children: Vec, +} +``` + +The fields are private, which guarantees the tree's invariant: **a node's `size` is always greater than or equal to the sum of its children's sizes.** The constructors maintain it, and +`Reflection::par_try_into_tree` verifies it when converting untrusted data. + +## Construction + +Trees are normally built by [`FsTreeBuilder` or `TreeBuilder`](Library-Building-Trees.md). The direct constructors serve tests and adapters. + +| Constructor | Description | +|----------------------------------------------------|------------------------------------------------------------------------------------------| +| `DataTree::file(name, size)` | A leaf node of the given size. | +| `DataTree::dir(name, inode_size, children)` | A directory whose total is `inode_size` plus the sum of its children. | +| `DataTree::fixed_size_dir_constructor(inode_size)` | Returns a `Fn(Name, Vec) -> Self` applying the same inode size to every directory. | + +`dir` takes the directory's own size, not its total. The total is computed for you. + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::size::Bytes; + +let dir = DataTree::< & str, Bytes>::fixed_size_dir_constructor(Bytes::new(4096)); + +let tree = dir("root", vec![ + DataTree::file("a.txt", Bytes::new(1024)), + dir("nested", vec![DataTree::file("b.txt", Bytes::new(2048))]), +]); + +assert_eq!(tree.size().inner(), 4096 + 1024 + 4096 + 2048); +``` + +## Inspection + +| Method | Returns | +|--------------|------------------| +| `name()` | `&Name` | +| `name_mut()` | `&mut Name` | +| `size()` | `Size` (by copy) | +| `children()` | `&Vec` | + +Nothing mutates size or children in place. To restructure a tree, use the parallel transformations below or round-trip through a [`Reflection`](#reflection). + +Walking the tree is ordinary recursion over `children()`: + +```rust +fn print_tree( + tree: ¶llel_disk_usage::data_tree::DataTree, + depth: usize, +) { + println!("{:indent$}{} {:?}", "", tree.name(), tree.size(), indent = depth * 2); + for child in tree.children() { + print_tree(child, depth + 1); + } +} +``` + +## Sorting + +Sorting is recursive and parallel. The comparator receives two sibling subtrees. + +| Method | Description | +|----------------------------|--------------------------------| +| `par_sort_by(compare)` | Sorts every level in place. | +| `into_par_sorted(compare)` | The consuming, chainable form. | + +`pdu` sorts descending by size: + +```rust +let tree = tree.into_par_sorted( | left, right| left.size().cmp( & right.size()).reverse()); +``` + +The comparator must be `Copy + Sync`. The sort is unstable, so equal-sized siblings may land in any order; add a tiebreaker on `name()` for deterministic output. + +## Filtering with `par_retain` + +| Method | Description | +|--------------------------------|---------------------------------------------------------------------------| +| `par_retain(predicate)` | Recursively drops every descendant for which `predicate` returns `false`. | +| `into_par_retained(predicate)` | The consuming, chainable form. | + +The predicate is `Fn(&Self, u64) -> bool`, where the second argument is the depth of the node's *parent*, starting at `0` for the root's children. + +Dropping a node drops its whole subtree, and surviving ancestors keep their original sizes. That is deliberate: the chart still shows a directory as large when its contents are too small to list. + +Trim to a fixed depth: + +```rust +let tree = tree.into_par_retained( | _, depth| depth < 3); +``` + +Drop anything under one mebibyte: + +```rust +use parallel_disk_usage::size::Bytes; + +let tree = tree.into_par_retained( | node, _ | node.size() > = Bytes::new(1024 * 1024)); +``` + +### `par_cull_insignificant_data` + +*(feature: `cli`)* Drops every descendant smaller than `min_ratio` of the root's size, implementing +`--min-ratio`. A ratio that is zero, negative, or NaN is a no-op. Without the `cli` feature: + +```rust +let minimal = tree.size().inner() as f32 * min_ratio; +tree.par_retain( | node, _ | node.size().inner() as f32 > = minimal); +``` + +## Deduplicating hardlinks + +A file with several links inside the tree is counted once per link. Correcting that is the job of +`DeduplicateSharedSize::deduplicate`. See [Hardlinks](Library-Hardlinks.md). + +## Reflection + +`data_tree::Reflection`, also exported as `DataTreeReflection`, is a structurally identical mirror of `DataTree` with public fields. + +```rust +pub struct Reflection { + pub name: Name, + pub size: Size, + pub children: Vec, +} +``` + +It is the serializable form of a tree, and it lets tests construct trees literally, including invalid ones. + +### Converting + +| Direction | API | Notes | +|----------------------------|--------------------------------------------|--------------------------------------------------------| +| `DataTree` to `Reflection` | `into_reflection()`, or `Reflection::from` | Always succeeds. | +| `Reflection` to `DataTree` | `par_try_into_tree()` | Validates the size invariant in parallel and may fail. | + +`par_try_into_tree` returns `ConversionError::ExcessiveChildren` when a node is smaller than one of its children. The error carries the `path` from the root as a `VecDeque`, the parent's +`size`, and the offending `child`, and implements `Display`. + +```rust +use parallel_disk_usage::data_tree::reflection::{ConversionError, Reflection}; +use parallel_disk_usage::size::Bytes; + +let reflection = Reflection { +name: "root".to_string(), +size: Bytes::new(100), +children: vec![Reflection { + name: "child".to_string(), + size: Bytes::new(200), + children: Vec::new(), +}], +}; + +match reflection.par_try_into_tree() { +Ok(_) => unreachable ! ("the child is larger than its parent"), +Err(ConversionError::ExcessiveChildren { path, .. }) => { +assert_eq ! (path.len(), 1); +} +} +``` + +`ConversionError` is `#[non_exhaustive]`; add a trailing `_ => {}` arm to keep compiling across versions. + +### Transforming + +| Method | Description | +|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------| +| `par_try_map(transform)` | Applies `Fn(Name, Size) -> Result<(TargetName, TargetSize), Error>` to every node in parallel. Changes the name type, the size unit, or both. | +| `par_convert_names_to_utf8()` | Converts `OsString`-like names to `String`, returning the first offending name as the error. `pdu` runs this before emitting JSON. | + +```rust +let reflection = data_tree +.into_reflection() +.par_convert_names_to_utf8() +.expect("all names are valid UTF-8"); +``` + +### Serialization + +*(feature: `json`)* `Reflection` derives `Serialize` and `Deserialize` with +`rename_all = "kebab-case"`. `DataTree` deliberately does not, so deserialization cannot produce a tree that violates the invariant. Deserialize into a reflection, then validate with +`par_try_into_tree`. + +For the full document format of `pdu --json-output`, see [JSON Data](Library-JSON-Data.md). diff --git a/docs/Library-Feature-Flags.md b/docs/Library-Feature-Flags.md new file mode 100644 index 00000000..62feeaa7 --- /dev/null +++ b/docs/Library-Feature-Flags.md @@ -0,0 +1,50 @@ +# Feature Flags + +| Feature | Default | Enables | +|-------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `cli` | yes | The `pdu` binary, the `app`, `args`, `man_page`, `usage_md`, and `runtime_error` modules, the `clap`, `clap_complete`, and `clap_utilities` re-exports, and the `json` feature. | +| `json` | no | `Serialize` and `Deserialize` implementations, and the `serde` and `serde_json` re-exports. | +| `cli-completions` | no | The `pdu-completions` binary. Implies `cli`. | +| `man-page` | no | The `pdu-man-page` binary. Implies `cli`. | +| `usage-md` | no | The `pdu-usage-md` binary. Implies `cli`. | +| `ai-instructions` | no | The `pdu-ai-instructions` binary. | + +The four binary-producing features generate the project's own documentation artifacts and are not useful to library consumers. + +## Recommended configuration + +Disabling default features removes `clap` and its dependency tree. + +```toml +[dependencies] +parallel-disk-usage = { version = "0.24", default-features = false } +``` + +Add `features = ["json"]` to serialize a tree. + +## Gated behind `cli` + +Most of the measurement and visualization API needs no features at all. The exceptions: + +* `DataTree::par_cull_insignificant_data`, the ratio filter behind `--min-ratio`. Without `cli`, express it with [`par_retain`](Library-DataTree.md#filtering-with-par_retain). +* The `app`, `args`, `man_page`, `usage_md`, and `runtime_error` modules, which implement the command-line program. + +## Gated behind `json` + +* `Serialize` and `Deserialize` for `Reflection`, `Bytes`, `Blocks`, `OsStringDisplay`, + `InodeNumber`, `DeviceNumber`, `HardlinkListReflection`, `SharedLinkSummary`, and `JsonData`. +* The `parallel_disk_usage::serde` and `parallel_disk_usage::serde_json` re-exports, which expose the exact versions the crate was compiled against. + +The `json_data` module itself always exists; only its derives are conditional. + +## Unix-only items + +Independent of any feature flag, these rely on `std::os::unix::fs::MetadataExt`: + +* `get_size::GetBlockSize` and `get_size::GetBlockCount`. +* `hardlink::HardlinkAware` and the `hardlink::aware` module, and therefore all hardlink deduplication. `hardlink::HardlinkIgnorant` exists everywhere. +* `InodeNumber::get` and `DeviceNumber::get`. + +Code that must build on Windows should gate these behind `#[cfg(unix)]` or use the +`hardlink::record::Do` and `hardlink::record::DoNot` aliases described in +[Hardlinks](Library-Hardlinks.md#platform-support). diff --git a/docs/Library-Getting-Started.md b/docs/Library-Getting-Started.md new file mode 100644 index 00000000..6240aa37 --- /dev/null +++ b/docs/Library-Getting-Started.md @@ -0,0 +1,112 @@ +# Getting Started + +## Installation + +The default feature is `cli`, which builds the `pdu` binary and pulls in `clap`. Library consumers rarely want that. + +```toml +[dependencies] +parallel-disk-usage = { version = "0.24", default-features = false } +``` + +Add `json` for serialization: + +```toml +[dependencies] +parallel-disk-usage = { version = "0.24", default-features = false, features = ["json"] } +``` + +See [Feature Flags](Library-Feature-Flags.md) for the full list. + +The crate name is hyphenated; the library is imported with underscores: + +```rust +use parallel_disk_usage::data_tree::DataTree; +``` + +## First measurement + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::device::DeviceBoundary; +use parallel_disk_usage::fs_tree_builder::FsTreeBuilder; +use parallel_disk_usage::get_size::GetApparentSize; +use parallel_disk_usage::hardlink::HardlinkIgnorant; +use parallel_disk_usage::os_string_display::OsStringDisplay; +use parallel_disk_usage::reporter::{ErrorOnlyReporter, ErrorReport}; +use parallel_disk_usage::size::Bytes; + +fn main() { + let builder = FsTreeBuilder { + root: std::env::current_dir().expect("get the current directory"), + size_getter: GetApparentSize, + hardlinks_recorder: &HardlinkIgnorant, + reporter: &ErrorOnlyReporter::new(ErrorReport::SILENT), + device_boundary: DeviceBoundary::Cross, + max_depth: u64::MAX, + }; + + let data_tree: DataTree = builder.into(); + + println!("{} bytes", data_tree.size().inner()); +} +``` + +`FsTreeBuilder` is a struct of parameters; the traversal happens in its `From` implementation, so +`builder.into()` is what triggers the work. The type annotation on `data_tree` is what selects +`Bytes`. `reporter` and `hardlinks_recorder` are borrowed because they are shared across worker threads. + +`max_depth` limits how deep the tree is *stored*, not how deep it is *measured*. Sizes below the cutoff still count toward their ancestors' totals. + +## Adding a chart + +Sort before rendering; the visualizer does not sort. + +```rust +use parallel_disk_usage::bytes_format::BytesFormat; +use parallel_disk_usage::visualizer::{BarAlignment, ColumnWidthDistribution, Direction, Visualizer}; + +let data_tree = data_tree.into_par_sorted( | left, right| left.size().cmp( & right.size()).reverse()); + +let visualizer = Visualizer { +data_tree: & data_tree, +bytes_format: BytesFormat::MetricUnits, +direction: Direction::BottomUp, +bar_alignment: BarAlignment::Left, +column_width_distribution: ColumnWidthDistribution::total(100), +}; + +println!("{visualizer}"); +``` + +## Reporting errors + +`ErrorReport::SILENT` discards every error. `ErrorReport::TEXT` prints each one to stderr through the library's status board: + +```rust +use parallel_disk_usage::reporter::{ErrorOnlyReporter, ErrorReport}; + +let reporter = ErrorOnlyReporter::new(ErrorReport::TEXT); +``` + +Both are `fn(ErrorReport)` values. Supply any `Fn(ErrorReport)` to handle reports yourself: + +```rust +let reporter = ErrorOnlyReporter::new( | report: ErrorReport| { +eprintln ! ( +"[error] {operation} {path:?}: {error}", +operation = report.operation.name(), +path = report.path, +error = report.error, +); +}); +``` + +For live progress counters, use `ProgressAndErrorReporter`. See [Reporters](Library-Reporters.md). + +## Next + +* [Building Trees](Library-Building-Trees.md) for every `FsTreeBuilder` field and the generic + `TreeBuilder`. +* [DataTree](Library-DataTree.md) for inspection, sorting, filtering, and serialization. +* [Recipes](Library-Recipes.md) for end-to-end examples. diff --git a/docs/Library-Hardlinks.md b/docs/Library-Hardlinks.md new file mode 100644 index 00000000..f8bcb479 --- /dev/null +++ b/docs/Library-Hardlinks.md @@ -0,0 +1,156 @@ +# Hardlinks + +A file with several hardlinks occupies its bytes once, but a naive walk counts them once per link. The `hardlink` module detects such files during measurement and subtracts the surplus afterwards. It is the library side of `--deduplicate-hardlinks`. + +## Two phases + +Deduplication is split in two: the first phase runs inside the parallel walk, the second only once the tree is complete. + +1. **Record.** `RecordHardlinks::record_hardlinks` is called by `FsTreeBuilder` for every item it stats. An implementation notes files whose link count exceeds one and the paths where they were seen. +2. **Deduplicate.** `DeduplicateSharedSize::deduplicate` consumes the recorder, walks the finished tree, and reduces the size of every directory holding more than one link to the same inode. + +One type usually implements both traits. + +## The traits + +```rust +pub trait RecordHardlinks { + type Error; + fn record_hardlinks(&self, argument: RecordHardlinksArgument) -> Result<(), Self::Error>; +} + +pub trait DeduplicateSharedSize: Sized { + type Report; + type Error; + fn deduplicate(self, data_tree: &mut DataTree) -> Result; +} +``` + +`RecordHardlinksArgument` carries the `path`, the `stats`, the computed `size`, and the `reporter`, so an implementation can emit `Event::DetectHardlink` as it goes. + +`record_hardlinks` takes `&self` and runs concurrently on every worker thread. `FsTreeBuilder` +discards its returned error, so an implementation that must surface a failure has to route it through the reporter. + +`deduplicate` takes `self` by value, guaranteeing that recording has finished before correction begins. Its `Report` is the accumulated record. + +## Implementations + +### `HardlinkIgnorant` + +Available everywhere. Both trait implementations are no-ops with `Error = Infallible`, and +`Report` is `()`. Hardlinks are treated as ordinary files. This is what `pdu` does without +`--deduplicate-hardlinks`. + +```rust +use parallel_disk_usage::hardlink::HardlinkIgnorant; + +let recorder = HardlinkIgnorant; +``` + +### `HardlinkAware` + +*(Unix only)* Wraps a [`HardlinkList`](#hardlinklist). For every non-directory whose `nlink` +exceeds one, it emits `Event::DetectHardlink` and records the inode number, device number, size, link count, and path. + +```rust +#[cfg(unix)] +use parallel_disk_usage::hardlink::HardlinkAware; +#[cfg(unix)] +use parallel_disk_usage::size::Bytes; + +#[cfg(unix)] +let recorder = HardlinkAware::::new(); +``` + +`deduplicate` returns the accumulated `HardlinkList`, with `Error = Infallible`. Recording can fail with `ReportHardlinksError::AddToRecord` when the same inode is observed with two different sizes or link counts, which means the filesystem changed mid-scan. + +Directories are skipped, since their link counts reflect subdirectory entries rather than shared content. + +## Putting it together + +The recorder is borrowed during measurement and consumed afterwards, so bind it to a variable. + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::device::DeviceBoundary; +use parallel_disk_usage::fs_tree_builder::FsTreeBuilder; +use parallel_disk_usage::get_size::GetApparentSize; +use parallel_disk_usage::hardlink::{DeduplicateSharedSize, HardlinkAware}; +use parallel_disk_usage::os_string_display::OsStringDisplay; +use parallel_disk_usage::reporter::{ErrorOnlyReporter, ErrorReport}; +use parallel_disk_usage::size::Bytes; + +let recorder = HardlinkAware::::new(); + +let mut data_tree: DataTree = FsTreeBuilder { +root: "/var/cache".into(), +size_getter: GetApparentSize, +hardlinks_recorder: & recorder, +reporter: & ErrorOnlyReporter::new(ErrorReport::SILENT), +device_boundary: DeviceBoundary::Cross, +max_depth: u64::MAX, +} +.into(); + +let record = recorder +.deduplicate( & mut data_tree) +.expect("deduplication is infallible"); + +println!("{} shared inodes", record.len()); +``` + +### Ordering constraints + +`deduplicate` matches recorded paths against node names by stripping the root's name as a prefix, so it must run after anything that renames the root and while the tree still contains the nodes holding the links. `pdu` runs build, cull, sort, deduplicate, then renames the synthetic root to +`(total)`. Follow the same order. + +Deduplication only subtracts where at least two links to the same inode sit under a given directory. A link whose sibling lives outside the measured tree stays counted in full, since as far as the scan can tell the space is genuinely attributable to that directory. + +## `HardlinkList` + +`HardlinkList` is the storage behind `HardlinkAware`: a concurrent map from an inode key, being an inode number paired with a device number, to that file's size, total link count, and the paths where it was seen. + +| Method | Description | +|-----------------------|--------------------------------------------------------------------------------------------------| +| `new()` | An empty list. | +| `len()`, `is_empty()` | Number of distinct shared inodes. | +| `iter()` | Iterates the entries. Items expose `ino()`, `dev()`, `size()`, `links()`, and `paths()`. | +| `into_reflection()` | Converts into a comparable and serializable [`HardlinkListReflection`](#reflection-and-summary). | + +The paths of one inode are held in a `LinkPathList`, offering `len()`, `is_empty()`, and +`into_reflection()`. + +## Reflection and summary + +As with `DataTree`, the concurrent structures implement neither `PartialEq` nor the `serde` traits. Convert them first. + +* `HardlinkListReflection` is a `Vec` of `ReflectionEntry`, sorted by inode and device number, every pair unique. Each entry has public `ino`, `dev`, `size`, `links`, and `paths` + fields. It offers `len()`, `is_empty()`, and `iter()`. +* `LinkPathListReflection` is the serializable form of a path list. Converting is O (n), since it turns a `Vec` into a `HashSet`. +* `SharedLinkSummary`, produced by the `SummarizeHardlinks` trait, aggregates a whole list. + +`SharedLinkSummary` is `#[non_exhaustive]` and carries: + +| Field | Meaning | +|-------------------------|------------------------------------------------------------| +| `inodes` | Number of inodes with more than one link. | +| `exclusive_inodes` | How many of those have no links outside the measured tree. | +| `all_links` | Total link count across all shared inodes. | +| `detected_links` | How many of those links were found inside the tree. | +| `exclusive_links` | Links belonging to exclusive inodes. | +| `shared_size` | Total size of all shared inodes. | +| `exclusive_shared_size` | Total size of the exclusive ones. | + +The "all" against "exclusive" distinction tells you whether deleting a directory would free the space, or whether another link elsewhere would keep it alive. + +## Platform support + +Hardlink detection needs `std::os::unix::fs::MetadataExt::nlink`, stable only on Unix. The Windows equivalent requires nightly, so `HardlinkAware` does not exist there. + +The module provides aliases that resolve to the right type: + +* `hardlink::record::Do` and `hardlink::deduplicate::Do` are `HardlinkAware`, Unix only. +* `hardlink::record::DoNot` and `hardlink::deduplicate::DoNot` are `HardlinkIgnorant`, available everywhere. + +Code that must compile on Windows should use `DoNot` unconditionally or gate the aware path behind +`#[cfg(unix)]`. diff --git a/docs/Library-JSON-Data.md b/docs/Library-JSON-Data.md new file mode 100644 index 00000000..854f74d4 --- /dev/null +++ b/docs/Library-JSON-Data.md @@ -0,0 +1,132 @@ +# JSON Data + +*(feature: `json`)* + +The `json_data` module defines the document `pdu --json-output` writes and `pdu --json-input` +reads. Use it to exchange trees with the `pdu` program, or as a stable on-disk format for a measurement. + +The module always exists, but its `Serialize` and `Deserialize` derives require the `json` feature. See [Feature Flags](Library-Feature-Flags.md). + +## Structure + +```text +JsonData +├── schema-version : SchemaVersion +├── pdu : Option +└── body : JsonDataBody (flattened) + └── unit : "bytes" | "blocks" (internally tagged) + └── JsonTree + ├── tree : DataTreeReflection (flattened via Deref) + └── shared : JsonShared + ├── details : Option> + └── summary : Option> +``` + +Field names are kebab-case. `JsonDataBody` is internally tagged on `unit`, so one document can carry either a byte tree or a block tree without ambiguity. + +```json +{ + "schema-version": "2026-04-02", + "pdu": "0.24.0", + "unit": "bytes", + "tree": { + "name": "root", + "size": 4096, + "children": [] + } +} +``` + +## Types + +| Type | Role | +|--------------------|-------------------------------------------------------------------------| +| `JsonData` | The whole document. | +| `JsonDataBody` | `Bytes(JsonTree)` or `Blocks(JsonTree)`. | +| `JsonTree` | The tree plus its hardlink information. Derefs to the inner reflection. | +| `JsonShared` | Optional hardlink `details` and `summary`. | +| `SchemaVersion` | A zero-sized token that validates the schema string on deserialization. | +| `BinaryVersion` | The version of the `pdu` that produced the document. | + +`JsonDataBody` derives `From` and `TryInto`, so `json_tree.into()` builds the body and +`body.try_into()` extracts a tree of a specific unit. + +## Versioning + +`SchemaVersion` serializes to the constant `json_data::SCHEMA_VERSION` and deserializes only from that exact string. Anything else produces an `InvalidSchema` error naming the offending input, so an incompatible document fails at parse time rather than producing a wrong tree. + +`BinaryVersion` is informational and not validated. `BinaryVersion::current()` returns the version of the crate you compiled against, from `json_data::CURRENT_VERSION`. + +## Writing + +Convert the tree to a reflection with `String` names first; JSON cannot represent an arbitrary +`OsString`. + +```rust +use parallel_disk_usage::json_data::{JsonData, JsonShared, JsonTree, SchemaVersion, BinaryVersion}; + +let tree = data_tree +.into_reflection() +.par_convert_names_to_utf8() +.expect("all names are valid UTF-8"); + +let json_data = JsonData { +schema_version: SchemaVersion, +binary_version: Some(BinaryVersion::current()), +body: JsonTree { tree, shared: JsonShared::default () }.into(), +}; + +serde_json::to_writer(std::io::stdout(), & json_data).expect("serialize the tree"); +``` + +`par_convert_names_to_utf8` returns the first non-UTF-8 name as its error. Handle it rather than unwrapping if the scanned filesystem might contain such names. + +`shared` is skipped during serialization when both `details` and `summary` are absent or empty, so a document without hardlink information has no `shared` key. + +## Reading + +```rust +use parallel_disk_usage::json_data::{JsonData, JsonDataBody}; + +let json_data: JsonData = serde_json::from_reader(std::io::stdin()).expect("parse the document"); + +match json_data.body { +JsonDataBody::Bytes(json_tree) => { +let data_tree = json_tree.tree.par_try_into_tree().expect("valid tree"); +// render or inspect +} +JsonDataBody::Blocks(json_tree) => { +let data_tree = json_tree.tree.par_try_into_tree().expect("valid tree"); +} +} +``` + +Deserialization yields a `Reflection`, not a `DataTree`, and the reflection is untrusted. +`par_try_into_tree` verifies that no node is smaller than its children; do not skip it on data you did not produce. See [DataTree](Library-DataTree.md#reflection). + +The two arms have different `Size` types, so code handling both ends up duplicated or generic. This is the monomorphization constraint described in +[Sizes and Formatting](Library-Sizes-And-Formatting.md#choosing-a-unit-at-run-time). + +## Hardlink information + +Populate `JsonShared` from the record that +[`DeduplicateSharedSize::deduplicate`](Library-Hardlinks.md) returned: + +```rust +use parallel_disk_usage::hardlink::hardlink_list::summary::SummarizeHardlinks; +use parallel_disk_usage::json_data::JsonShared; + +let summary = record.iter().summarize_hardlinks(); +let shared = JsonShared { +details: Some(record.into_reflection()), +summary: Some(summary), +}; +``` + +Set either field to `None` to omit it, as `--omit-json-shared-details` and +`--omit-json-shared-summary` do. + +## Using the crate's `serde` + +The crate re-exports the exact `serde` and `serde_json` versions it was built against as +`parallel_disk_usage::serde` and `parallel_disk_usage::serde_json`. Prefer them over your own dependency entries on a version mismatch in the derived traits. diff --git a/docs/Library-Module-Index.md b/docs/Library-Module-Index.md new file mode 100644 index 00000000..778367d6 --- /dev/null +++ b/docs/Library-Module-Index.md @@ -0,0 +1,86 @@ +# Module Index + +A map of the `parallel_disk_usage` crate. Items marked *(cli)*, *(json)*, or *(unix)* are conditional; see [Feature Flags](Library-Feature-Flags.md). + +Generated API documentation lives at +[docs.rs/parallel-disk-usage](https://docs.rs/parallel-disk-usage). This page tells you which module to open. + +## Measurement + +**`fs_tree_builder`** — `FsTreeBuilder`, which walks a real directory tree. See +[Building Trees](Library-Building-Trees.md#fstreebuilder). + +**`tree_builder`** — `TreeBuilder` and `Info`, the generic parallel recursion underneath +`FsTreeBuilder`. See [Building Trees](Library-Building-Trees.md#treebuilder). + +**`get_size`** — `GetSize`, plus `GetApparentSize`, `GetBlockSize` *(unix)*, and +`GetBlockCount` *(unix)*. See +[Sizes and Formatting](Library-Sizes-And-Formatting.md#getsize). + +## The tree + +**`data_tree`** — `DataTree` with its constructors, getters, and the `par_sort_by` and `par_retain` +transformations. `data_tree::reflection` holds `Reflection`, also exported as +`DataTreeReflection`, and `ConversionError`. See [DataTree](Library-DataTree.md). + +**`size`** — the `Size` trait and the units `Bytes` and `Blocks`. See +[Sizes and Formatting](Library-Sizes-And-Formatting.md#the-size-trait). + +**`os_string_display`** — `OsStringDisplay`, the name type `FsTreeBuilder` produces. It wraps an +`OsString`, displays valid UTF-8 as text, and falls back to the `Debug` form otherwise. +`os_string_from` constructs one; `as_os_str` and `inner` read it back. It derefs to its inner value and derives `Serialize` and `Deserialize` *(json)*. + +## Presentation + +**`visualizer`** — `Visualizer` with `Direction`, `BarAlignment`, and `ColumnWidthDistribution`, plus the rendering components `ProportionBar`, `ProportionBarBlock`, `TreeSkeletalComponent`, +`TreeHorizontalSlice`, `ChildPosition`, and `Parenthood`. See [Visualizer](Library-Visualizer.md). + +**`bytes_format`** — `BytesFormat`, `Formatter`, `ParsedValue`, `Output`, and the `scale_base` +constants. See [Sizes and Formatting](Library-Sizes-And-Formatting.md#formatting-bytes). + +**`status_board`** — `StatusBoard` and the `GLOBAL_STATUS_BOARD` static, which arbitrate between transient and permanent messages on stderr. See +[Reporters](Library-Reporters.md#the-status-board). + +## Progress and errors + +**`reporter`** — `Reporter`, `ParallelReporter`, `Event`, the implementations `ErrorOnlyReporter` +and `ProgressAndErrorReporter`, and `ErrorReport`, `error_report::Operation`, and `ProgressReport`. See [Reporters](Library-Reporters.md). + +## Hardlinks and filesystem identity + +**`hardlink`** — `RecordHardlinks` and `DeduplicateSharedSize`, the implementations +`HardlinkIgnorant` and `HardlinkAware` *(unix)*, and the storage types `HardlinkList`, +`LinkPathList`, their reflections, and `SharedLinkSummary`. See [Hardlinks](Library-Hardlinks.md). + +**`inode`** — `InodeNumber`, a newtype over `u64`. `InodeNumber::get(&metadata)` *(unix)* reads one from `Metadata`. + +**`device`** — `DeviceBoundary`, the `Cross` or `Stay` choice passed to `FsTreeBuilder`, and +`DeviceNumber`, whose `get(&metadata)` *(unix)* reads a filesystem's device number. Both newtypes implement `Display` and the hexadecimal and octal formatting traits. + +## Interchange + +**`json_data`** — `JsonData`, `JsonDataBody`, `JsonTree`, `JsonShared`, `SchemaVersion`, and +`BinaryVersion`, plus the `SCHEMA_VERSION` and `CURRENT_VERSION` constants. See +[JSON Data](Library-JSON-Data.md). + +## Command-line program + +These modules implement `pdu`. They are public so the binary can use them, but are not a stable library surface. `app::Sub::run` is the reference example of the full pipeline. + +| Module | Contents | +|-------------------------|--------------------------------------------------------------------| +| `app` *(cli)* | `App::from_env` and `App::run`, and the generic `app::Sub`. | +| `args` *(cli)* | The `clap` argument definitions, including `Depth` and `Fraction`. | +| `runtime_error` *(cli)* | `RuntimeError` and its exit codes. | +| `man_page` *(cli)* | Man page generation. | +| `usage_md` *(cli)* | `USAGE.md` generation. | + +The crate also provides `parallel_disk_usage::main`, the entry point of the `pdu` binary. + +## Re-exports + +| Re-export | Condition | +|-------------------------------------------|-----------| +| `zero_copy_pads` | Always | +| `serde`, `serde_json` | *(json)* | +| `clap`, `clap_complete`, `clap_utilities` | *(cli)* | diff --git a/docs/Library-Overview.md b/docs/Library-Overview.md new file mode 100644 index 00000000..3e108f39 --- /dev/null +++ b/docs/Library-Overview.md @@ -0,0 +1,68 @@ +# Library Overview + +`parallel-disk-usage` ships both the `pdu` command-line program and a reusable Rust library named +`parallel_disk_usage`. This page describes the shape of the library. For the command-line program, see [CLI Usage](CLI-Usage.md). + +The library measures disk usage of a filesystem tree in parallel, stores the result in an in-memory tree, and renders that tree as an ASCII chart or as JSON. + +## The pipeline + +Every use of the library follows the same three stages. Each stage is independent, so you may stop after any of them or substitute your own implementation. + +```text + ┌─────────────────────────────────────────────────────────────────────────┐ + │ 1. Measure │ + │ FsTreeBuilder (real filesystem) │ + │ TreeBuilder (any hierarchical source) │ + └────────────────────────────────┬────────────────────────────────────────┘ + │ DataTree + ┌────────────────────────────────┴────────────────────────────────────────┐ + │ 2. Transform │ + │ par_sort_by, par_retain, par_cull_insignificant_data, │ + │ DeduplicateSharedSize::deduplicate │ + └────────────────────────────────┬────────────────────────────────────────┘ + │ DataTree + ┌────────────────────────────────┴────────────────────────────────────────┐ + │ 3. Render │ + │ Visualizer (ASCII chart) DataTree::into_reflection (JSON) │ + └─────────────────────────────────────────────────────────────────────────┘ +``` + +Progress and errors are delivered during stage 1 by a [`Reporter`](Library-Reporters.md), so measurement never stops to print a message. + +## Central types + +| Type | Module | Role | +|-------------------------------------------------------|-------------------------|------------------------------------------------------------------------------------------| +| [`FsTreeBuilder`](Library-Building-Trees.md) | `fs_tree_builder` | Walks a real directory tree and produces a `DataTree`. | +| [`TreeBuilder`](Library-Building-Trees.md) | `tree_builder` | Generic parallel tree construction from any source. | +| [`DataTree`](Library-DataTree.md) | `data_tree` | The measured tree. | +| [`Reflection`](Library-DataTree.md#reflection) | `data_tree::reflection` | Public-field mirror of `DataTree`, used for serialization and for construction in tests. | +| [`Visualizer`](Library-Visualizer.md) | `visualizer` | Renders a `DataTree` into an ASCII chart. | +| [`Reporter`](Library-Reporters.md) | `reporter` | Receives progress and error events during measurement. | +| [`GetSize`](Library-Sizes-And-Formatting.md#getsize) | `get_size` | Decides what "size" means for a given file. | +| [`Size`](Library-Sizes-And-Formatting.md#the-size-trait) | `size` | Trait implemented by `Bytes` and `Blocks`. | +| [`RecordHardlinks`](Library-Hardlinks.md) | `hardlink` | Detects hardlinks so their size is not counted twice. | +| [`JsonData`](Library-JSON-Data.md) | `json_data` | The schema of `--json-output` and `--json-input`. | + +## Generic parameters + +The same three names appear throughout the source. + +* `Name` is a node's label. `FsTreeBuilder` always produces + [`OsStringDisplay`](Library-Module-Index.md#the-tree), which displays a valid UTF-8 path as text and falls back to the `Debug` form otherwise. +* `Size` is the unit of measurement, either `Bytes` or `Blocks`. +* `Report` is the progress sink, almost always taken by reference so it can be shared across Rayon worker threads. + +## Parallelism + +Measurement and most tree transformations use [Rayon](https://docs.rs/rayon). Parallel methods carry a `par_` prefix. Because their closures are handed to multiple threads, those closures must generally be `Copy + Sync`. + +The library uses the ambient Rayon global pool and creates no pool of its own. To bound concurrency, install your own with +[`rayon::ThreadPoolBuilder`](https://docs.rs/rayon/latest/rayon/struct.ThreadPoolBuilder.html) and run the measurement inside it. + +## Next + +* [Getting Started](Library-Getting-Started.md) for installation and a first program. +* [Feature Flags](Library-Feature-Flags.md) to trim the dependency tree. +* [Recipes](Library-Recipes.md) for task-oriented examples. diff --git a/docs/Library-Recipes.md b/docs/Library-Recipes.md new file mode 100644 index 00000000..1b311e2e --- /dev/null +++ b/docs/Library-Recipes.md @@ -0,0 +1,273 @@ +# Recipes + +Task-oriented examples, each assuming the dependency configuration from +[Getting Started](Library-Getting-Started.md). + +## Total size of a directory + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::device::DeviceBoundary; +use parallel_disk_usage::fs_tree_builder::FsTreeBuilder; +use parallel_disk_usage::get_size::GetApparentSize; +use parallel_disk_usage::hardlink::HardlinkIgnorant; +use parallel_disk_usage::os_string_display::OsStringDisplay; +use parallel_disk_usage::reporter::{ErrorOnlyReporter, ErrorReport}; +use parallel_disk_usage::size::Bytes; +use std::path::Path; + +fn total_size(root: &Path) -> Bytes { + let data_tree: DataTree = FsTreeBuilder { + root: root.to_path_buf(), + size_getter: GetApparentSize, + hardlinks_recorder: &HardlinkIgnorant, + reporter: &ErrorOnlyReporter::new(ErrorReport::SILENT), + device_boundary: DeviceBoundary::Cross, + max_depth: 1, + } + .into(); + + data_tree.size() +} +``` + +`max_depth: 1` keeps only the root node. The traversal still visits every file, so the total is complete. + +## Largest entries directly under a directory + +```rust +use parallel_disk_usage::size::Size; + +let data_tree: DataTree = FsTreeBuilder { +root: root.to_path_buf(), +size_getter: GetApparentSize, +hardlinks_recorder: & HardlinkIgnorant, +reporter: & ErrorOnlyReporter::new(ErrorReport::SILENT), +device_boundary: DeviceBoundary::Cross, +max_depth: 2, +} +.into(); + +let data_tree = data_tree.into_par_sorted( | left, right| left.size().cmp( & right.size()).reverse()); + +for child in data_tree.children().iter().take(10) { +println ! ("{:>10} {}", child.size().display(BytesFormat::MetricUnits), child.name()); +} +``` + +`max_depth: 2` retains the root and its immediate children. + +## A chart sized to the terminal + +```rust +use parallel_disk_usage::bytes_format::BytesFormat; +use parallel_disk_usage::visualizer::{BarAlignment, ColumnWidthDistribution, Direction, Visualizer}; + +let width = terminal_size::terminal_size() +.map_or(80, | (terminal_size::Width(width), _) | usize::from(width)); + +let visualizer = Visualizer { +data_tree: & data_tree, +bytes_format: BytesFormat::MetricUnits, +direction: Direction::BottomUp, +bar_alignment: BarAlignment::Left, +column_width_distribution: ColumnWidthDistribution::total(width), +}; + +print!("{visualizer}"); // the chart already ends with a newline +``` + +## Live progress during a long scan + +```rust +use parallel_disk_usage::reporter::{ + ErrorReport, ParallelReporter, ProgressAndErrorReporter, ProgressReport, +}; +use parallel_disk_usage::status_board::GLOBAL_STATUS_BOARD; +use std::time::Duration; + +let reporter = ProgressAndErrorReporter::::new( +ProgressReport::TEXT, +Duration::from_millis(100), +ErrorReport::TEXT, +); + +let data_tree: DataTree = FsTreeBuilder { +root: root.to_path_buf(), +size_getter: GetApparentSize, +hardlinks_recorder: & HardlinkIgnorant, +reporter: & reporter, +device_boundary: DeviceBoundary::Cross, +max_depth: u64::MAX, +} +.into(); + +if reporter.destroy().is_err() { +eprintln ! ("[warning] the progress reporting thread panicked"); +} + +GLOBAL_STATUS_BOARD.clear_line(0); +``` + +Destroying the reporter before rendering keeps the progress line out of the chart; `clear_line(0)` +erases the last one it printed. + +## Several roots at once + +`pdu` puts every root under a synthetic parent of zero size, so the total is the sum. + +```rust +use parallel_disk_usage::data_tree::DataTree; + +let children: Vec > = roots +.iter() +.map( | root| { +FsTreeBuilder { +root: root.to_path_buf(), +size_getter: GetApparentSize, +hardlinks_recorder: & HardlinkIgnorant, +reporter: &ErrorOnlyReporter::new(ErrorReport::SILENT), +device_boundary: DeviceBoundary::Cross, +max_depth: u64::MAX, +} +.into() +}) +.collect(); + +let total = DataTree::dir( +OsStringDisplay::os_string_from("(total)"), +Bytes::new(0), +children, +); +``` + +If you also deduplicate hardlinks, name the synthetic root with an empty string until after +`deduplicate` has run, then rename it with `name_mut`. Path prefix stripping treats the empty string as a prefix of every path; any other name prevents matches. This is why `pdu` renames late. + +## Filtering out the noise + +Keep entries accounting for at least one percent of the total: + +```rust +let minimal = data_tree.size().inner() as f32 * 0.01; +let data_tree = data_tree.into_par_retained( | node, _ | node.size().inner() as f32 > = minimal); +``` + +Limit display depth without limiting measurement: + +```rust +let data_tree = data_tree.into_par_retained( | _, depth| depth < 3); +``` + +Both leave ancestor sizes untouched, so a directory keeps its true size after its contents are dropped. + +## Apparent size against on-disk size + +On Unix, measuring twice with different size getters shows the space lost to block rounding or saved by sparse files. + +```rust +#[cfg(unix)] +use parallel_disk_usage::get_size::{GetApparentSize, GetBlockSize}; + +#[cfg(unix)] +fn compare(root: &std::path::Path) { + let apparent: DataTree = FsTreeBuilder { + root: root.to_path_buf(), + size_getter: GetApparentSize, + hardlinks_recorder: &HardlinkIgnorant, + reporter: &ErrorOnlyReporter::new(ErrorReport::SILENT), + device_boundary: DeviceBoundary::Cross, + max_depth: 1, + } + .into(); + + let on_disk: DataTree = FsTreeBuilder { + root: root.to_path_buf(), + size_getter: GetBlockSize, + hardlinks_recorder: &HardlinkIgnorant, + reporter: &ErrorOnlyReporter::new(ErrorReport::SILENT), + device_boundary: DeviceBoundary::Cross, + max_depth: 1, + } + .into(); + + println!("apparent {:?}, on disk {:?}", apparent.size(), on_disk.size()); +} +``` + +## Staying on one filesystem + +```rust +use parallel_disk_usage::device::DeviceBoundary; + +let builder = FsTreeBuilder { +root: "/".into(), +device_boundary: DeviceBoundary::Stay, +// ... +}; +``` + +The equivalent of `--one-file-system`, which keeps a scan of `/` out of network mounts and removable media. + +## A tree from something other than a filesystem + +Anything with a parent-child structure and a size works with `TreeBuilder`. This walks an in-memory map. + +```rust +use parallel_disk_usage::data_tree::DataTree; +use parallel_disk_usage::size::Bytes; +use parallel_disk_usage::tree_builder::{Info, TreeBuilder}; +use std::collections::HashMap; + +fn build(entries: &HashMap)>) -> DataTree { + TreeBuilder:: { + path: "/".to_string(), + name: "/".to_string(), + get_info: |path| match entries.get(path) { + Some((size, children)) => Info { + size: Bytes::new(*size), + children: children.clone(), + }, + None => Info { + size: Bytes::new(0), + children: Vec::new(), + }, + }, + join_path: |prefix, name| format!("{prefix}/{name}"), + max_depth: u64::MAX, + } + .into() +} +``` + +Both closures capture `entries` by shared reference, satisfying the `Copy + Sync` bound. See +[Building Trees](Library-Building-Trees.md#treebuilder). + +## Round-tripping through JSON + +*(feature: `json`)* + +```rust +use parallel_disk_usage::json_data::{BinaryVersion, JsonData, JsonDataBody, JsonShared, JsonTree, SchemaVersion}; + +let tree = data_tree +.into_reflection() +.par_convert_names_to_utf8() +.expect("all names are valid UTF-8"); + +let json_data = JsonData { +schema_version: SchemaVersion, +binary_version: Some(BinaryVersion::current()), +body: JsonTree { tree, shared: JsonShared::default () }.into(), +}; + +let text = serde_json::to_string( & json_data).expect("serialize"); +let parsed: JsonData = serde_json::from_str( & text).expect("deserialize"); + +let JsonDataBody::Bytes(json_tree) = parsed.body else { +panic ! ("expected a byte tree"); +}; +let data_tree = json_tree.tree.par_try_into_tree().expect("valid tree"); +``` + +The output is byte-compatible with `pdu --json-output` and can be piped into `pdu --json-input`. See [JSON Data](Library-JSON-Data.md). diff --git a/docs/Library-Reporters.md b/docs/Library-Reporters.md new file mode 100644 index 00000000..d8982e3a --- /dev/null +++ b/docs/Library-Reporters.md @@ -0,0 +1,157 @@ +# Reporters + +Measurement produces two kinds of side information: progress counters and filesystem errors. The +`reporter` module delivers both through one trait, so the traversal never stops to print. + +## The traits + +```rust +pub trait Reporter { + fn report(&self, event: Event); +} + +pub trait ParallelReporter: Reporter { + type DestructionError; + fn destroy(self) -> Result<(), Self::DestructionError>; +} +``` + +`report` takes `&self` because the reporter is shared by every worker thread, so any state it accumulates must be internally synchronized, typically with atomics. + +`ParallelReporter::destroy` shuts down whatever background threads the reporter owns. Call it once the tree is complete and before rendering, so a progress line does not interleave with the chart. + +A blanket implementation forwards `Reporter` through shared references, which is why +`FsTreeBuilder` can hold a `&Report`. + +## Events + +`reporter::Event` is the message type. It is `#[non_exhaustive]`, so match with a catch-all arm. + +| Variant | Emitted when | +|-------------------------------------|--------------------------------------------------------------------------------| +| `ReceiveData(Size)` | An item's size has been measured. Once per file and directory. | +| `EncounterError(ErrorReport)` | A filesystem operation failed. | +| `DetectHardlink(HardlinkDetection)` | A file with more than one link was found. Only with a hardlink-aware recorder. | + +`HardlinkDetection` carries the `path`, the `stats`, the `size`, and `links`, the total number of links to the inode including this one. + +## Error reports + +```rust +pub struct ErrorReport<'a> { + pub operation: Operation, + pub path: &'a Path, + pub error: std::io::Error, +} +``` + +`Operation` names the failing call and offers `name()` for a human-readable form. + +| Variant | `name()` | Cause | +|-------------------|--------------------|--------------------------------------------------------| +| `SymlinkMetadata` | `symlink_metadata` | The entry could not be stated. | +| `ReadDirectory` | `read_dir` | A directory could not be listed. | +| `AccessEntry` | `access entry` | One entry of a listed directory could not be accessed. | + +Two handlers are provided as associated constants of type `fn(ErrorReport)`: + +* `ErrorReport::SILENT` ignores the report. +* `ErrorReport::TEXT` prints `[error] {operation} {path:?}: {error}` to stderr via the + [status board](#the-status-board). + +The report borrows its path, so a handler that keeps the information must copy it out. + +## `ErrorOnlyReporter` + +Ignores progress and forwards only errors. + +```rust +use parallel_disk_usage::reporter::{ErrorOnlyReporter, ErrorReport}; + +let reporter = ErrorOnlyReporter::new(ErrorReport::TEXT); +``` + +`new` accepts any `Fn(ErrorReport)`. Its `destroy` is a no-op that cannot fail, since it owns no threads. This is the right choice whenever you do not need live progress. + +## `ProgressAndErrorReporter` + +Accumulates counters in atomics and spawns a thread that publishes them on a fixed interval. + +```rust +use parallel_disk_usage::reporter::{ErrorReport, ProgressAndErrorReporter, ProgressReport}; +use parallel_disk_usage::size::Bytes; +use std::time::Duration; + +let reporter = ProgressAndErrorReporter::::new( +ProgressReport::TEXT, +Duration::from_millis(100), +ErrorReport::TEXT, +); +``` + +The arguments are the progress handler, the publication interval, and the error handler. The progress handler runs on the reporter's own thread, so a slow handler delays publication but never the measurement. + +`ProgressReport` holds the counters: + +| Field | Meaning | +|----------|------------------------------------------------------| +| `items` | Number of items measured. | +| `total` | Sum of their sizes. | +| `errors` | Number of errors encountered. | +| `linked` | Total number of links across all detected hardlinks. | +| `shared` | Total size of all detected hardlinks. | + +`ProgressReport::TEXT` renders these as the one-line status `pdu` shows, omitting the hardlink and error counts when they are zero. The struct also derives `with_`-prefixed setters, useful in tests. + +Errors are reported before progress, so an error message never appears to describe a state newer than the counters printed after it. + +### Shutting it down + +`destroy` stops the thread and joins it, returning `Result<(), Box>`, the payload of a panic in the reporting thread. `stop_progress_reporter` signals the thread without joining. Dropping the reporter also stops it, but joining is what guarantees no further output. + +```rust +use parallel_disk_usage::reporter::ParallelReporter; + +if reporter.destroy().is_err() { +eprintln ! ("[warning] the progress reporting thread panicked"); +} +``` + +## Writing a custom reporter + +```rust +use parallel_disk_usage::reporter::{Event, Reporter}; +use parallel_disk_usage::size::Bytes; +use std::sync::atomic::{AtomicU64, Ordering::Relaxed}; + +#[derive(Default)] +struct Counter { + items: AtomicU64, + errors: AtomicU64, +} + +impl Reporter for Counter { + fn report(&self, event: Event) { + match event { + Event::ReceiveData(_) => { self.items.fetch_add(1, Relaxed); } + Event::EncounterError(_) => { self.errors.fetch_add(1, Relaxed); } + _ => {} + } + } +} +``` + +`report` runs once per measured item from many threads at once, squarely in the hot path. Keep it allocation-free and lock-free, and use `Ordering::Relaxed` for counters as the built-in reporter does. + +Implement `ParallelReporter` as well if your reporter owns threads. If not, mirror +`ErrorOnlyReporter` with `type DestructionError = Infallible`. + +## The status board + +`status_board::GLOBAL_STATUS_BOARD` is the shared handle to stderr that keeps a transient progress line from being scrambled by permanent messages. + +* `temporary_message(text)` overwrites the current line, updating the progress counter in place. +* `permanent_message(text)` clears the transient line, prints, and moves to the next line. +* `clear_line(0)` erases the transient line, which is what to call before printing a chart. + +Any code writing to stderr while a `ProgressAndErrorReporter` runs should go through the status board rather than `eprintln!`. diff --git a/docs/Library-Sizes-And-Formatting.md b/docs/Library-Sizes-And-Formatting.md new file mode 100644 index 00000000..006c78a6 --- /dev/null +++ b/docs/Library-Sizes-And-Formatting.md @@ -0,0 +1,130 @@ +# Sizes and Formatting + +Three modules decide how big a file is and how that number is printed: `size` defines the units, +`get_size` reads a unit off a file, and `bytes_format` renders it. + +## The `Size` trait + +`size::Size` is the bound every measurement unit satisfies. It requires the usual arithmetic and ordering operations plus multiplication by the unsigned integer types, so trees can be summed, compared, and scaled. + +```rust +pub trait Size: /* Debug + Default + Copy + Ord + Add + Sub + Sum + ... */ { + type Inner: From + Into + Mul; + type DisplayFormat: Copy; + type DisplayOutput: Display; + fn display(self, input: Self::DisplayFormat) -> Self::DisplayOutput; +} +``` + +`Inner` is the underlying primitive, `u64` for both built-in units. `DisplayFormat` is the configuration rendering needs. There is no reason to implement the trait yourself unless you are inventing a unit. + +## `Bytes` and `Blocks` + +Both are newtypes over `u64` with the same small API. + +| | `Bytes` | `Blocks` | +|-----------------|------------------------|----------------------------------------| +| Meaning | A count of bytes. | A count of 512-byte filesystem blocks. | +| `DisplayFormat` | `BytesFormat` | `()` | +| `DisplayOutput` | `bytes_format::Output` | `u64` | + +```rust +use parallel_disk_usage::size::{Bytes, Size}; +use parallel_disk_usage::bytes_format::BytesFormat; + +let size = Bytes::new(1_500_000); +assert_eq!(size.inner(), 1_500_000); +assert_eq!(size.display(BytesFormat::MetricUnits).to_string(), "1.5M"); +``` + +`new` and `inner` are `const fn`. `From` and `Into` are derived. Addition, subtraction, +`Sum`, and multiplication by `u8` through `u64` and `usize` are all available, which is what lets the tree arithmetic and hardlink deduplication be written directly. + +## `GetSize` + +`get_size::GetSize` maps a `std::fs::Metadata` to a size. It decides whether `pdu` reports apparent size or on-disk usage. + +```rust +pub trait GetSize { + type Size; + fn get_size(&self, metadata: &Metadata) -> Self::Size; +} +``` + +| Implementation | `Size` | Returns | Platform | +|-------------------|----------|------------------------------------------------|-----------| +| `GetApparentSize` | `Bytes` | `metadata.len()`, the logical file length. | All | +| `GetBlockSize` | `Bytes` | `metadata.blocks() * 512`, the space occupied. | Unix only | +| `GetBlockCount` | `Blocks` | `metadata.blocks()`. | Unix only | + +The difference shows up in sparse files and in files whose tail does not fill a block. +`GetApparentSize` reports what an application would read; `GetBlockSize` reports what the filesystem spends. The block-based implementations need +`std::os::unix::fs::MetadataExt`, so cross-platform code must gate them behind `#[cfg(unix)]`. + +All three are zero-sized types. + +### Writing your own + +```rust +use parallel_disk_usage::get_size::GetSize; +use parallel_disk_usage::size::Blocks; +use std::fs::Metadata; + +#[derive(Clone, Copy)] +struct CountFiles; + +impl GetSize for CountFiles { + type Size = Blocks; + fn get_size(&self, metadata: &Metadata) -> Self::Size { + if metadata.is_dir() { Blocks::new(0) } else { Blocks::new(1) } + } +} +``` + +`FsTreeBuilder` requires the getter to be `Sync`; the CLI's `Sub` additionally requires `Copy`. Keep implementations stateless. + +## Formatting bytes + +`bytes_format::BytesFormat` is the `DisplayFormat` of `Bytes` and mirrors `--bytes-format`. + +| Variant | Behaviour | 1,500,000 renders as | +|---------------|-------------------------|----------------------| +| `PlainNumber` | The raw count, no unit. | `1500000` | +| `MetricUnits` | Powers of 1000. | `1.5M` | +| `BinaryUnits` | Powers of 1024. | `1.4M` | + +`BytesFormat::format(bytes) -> Output` performs the conversion and is what `Bytes::display` calls. + +```rust +use parallel_disk_usage::bytes_format::BytesFormat; + +assert_eq!(BytesFormat::MetricUnits.format(1_500_000).to_string(), "1.5M"); +assert_eq!(BytesFormat::PlainNumber.format(1_500_000).to_string(), "1500000"); +``` + +### Underlying pieces + +* `bytes_format::Formatter` performs the scaling. It takes a scale base; the constants + `formatter::METRIC` and `formatter::BINARY` cover the standard two, built on + `scale_base::METRIC` (1000) and `scale_base::BINARY` (1024). +* `Formatter::parse_value(value)` returns a `ParsedValue`: `Small { value }` below the scale base, otherwise `Big { coefficient, unit, scale, exponent }` with a unit of `K`, `M`, `G`, `T`, or `P`. +* `bytes_format::Output` is the `Display` type, either `PlainNumber(u64)` or `Units(ParsedValue)`. + +`ParsedValue` renders `Big` to one decimal place and pads `Small` with trailing spaces so a column of values lines up. Use `Formatter` directly for the components rather than the string: + +```rust +use parallel_disk_usage::bytes_format::{ParsedValue, formatter::BINARY}; + +match BINARY.parse_value(3_221_225_472) { +ParsedValue::Big { coefficient, unit, exponent, .. } => { +assert_eq ! (unit, 'G'); +assert_eq !(exponent, 3); +assert ! ((coefficient - 3.0).abs() < f32::EPSILON); +} +ParsedValue::Small { .. } => unreachable ! (), +} +``` + +## Choosing a unit at run time + +`Size` is a trait and the builders are generic over it, so the unit is fixed at compile time within any one call. A program deciding at run time ends up with one code path per unit. `pdu` dispatches once in `app::Sub` and then runs fully monomorphized code. Mirror that structure; `Size` is not object-safe, so boxing is not an option. diff --git a/docs/Library-Visualizer.md b/docs/Library-Visualizer.md new file mode 100644 index 00000000..48f70baf --- /dev/null +++ b/docs/Library-Visualizer.md @@ -0,0 +1,97 @@ +# Visualizer + +`visualizer::Visualizer` renders a [`DataTree`](Library-DataTree.md) as the ASCII chart `pdu` prints. It is a struct of parameters whose work is performed by `Display`. + +```rust +use parallel_disk_usage::bytes_format::BytesFormat; +use parallel_disk_usage::visualizer::{BarAlignment, ColumnWidthDistribution, Direction, Visualizer}; + +let visualizer = Visualizer { +data_tree: & data_tree, +bytes_format: BytesFormat::MetricUnits, +direction: Direction::BottomUp, +bar_alignment: BarAlignment::Right, +column_width_distribution: ColumnWidthDistribution::total(100), +}; + +println!("{visualizer}"); +``` + +## Fields + +| Field | Type | Meaning | +|-----------------------------|---------------------------|-----------------------------------------------------------------------| +| `data_tree` | `&DataTree` | The tree to render. | +| `bytes_format` | `Size::DisplayFormat` | How sizes are rendered. `BytesFormat` for `Bytes`, `()` for `Blocks`. | +| `direction` | `Direction` | Whether the root sits at the bottom or the top. | +| `bar_alignment` | `BarAlignment` | Which side the proportion bars fill from. | +| `column_width_distribution` | `ColumnWidthDistribution` | How horizontal space is allocated. | + +`Name` must implement `Display` and `Size` must implement `Into`, which both built-in units do. `Visualizer` is `Copy`. + +## Two ways to render + +* `Display`, that is `to_string()` or `{}`, produces the whole chart with a trailing newline on each line, ordered by `direction`. +* `rows()` returns `Vec`, one entry per line, always top-down regardless of `direction`. Use it to post-process, paginate, or colourize. + +```rust +for row in visualizer.rows() { +println ! ("{row}"); +} +``` + +`rows()` may lay the chart out more than once while converging on a column allocation that fits. Render once and reuse the result. + +## `Direction` + +| Variant | Effect | +|------------|-------------------------------------------------------------------------------------------------------| +| `BottomUp` | The root is the last line. The default of `pdu`, chosen so the root ends up next to the shell prompt. | +| `TopDown` | The root is the first line, matching `--top-down`. | + +## `BarAlignment` + +| Variant | Effect | +|---------|-----------------------------------------------------| +| `Left` | Bars fill from the left. | +| `Right` | Bars fill from the right, matching `--align-right`. | + +## `ColumnWidthDistribution` + +Controls how many characters the chart may occupy and how they are split between the tree column and the bar column. Both constructors are `const fn`. + +```rust +use parallel_disk_usage::visualizer::ColumnWidthDistribution; + +// Total budget. The visualizer decides the split. +let width = ColumnWidthDistribution::total(100); + +// Explicit split: at most 60 columns of tree, exactly 30 columns of bar. +let width = ColumnWidthDistribution::components(60, 30); +``` + +`total(width)` corresponds to `--total-width` and suits a terminal of known width. The visualizer works out the split and falls back to a minimum layout when the budget is too small for the names it must print, so a small total does not truncate the chart into unreadability. + +`components(tree_column_max_width, bar_column_width)` corresponds to `--column-width` and pins both sides. The first value is a maximum; the second is exact. + +To match the terminal, read its width with [`terminal_size`](https://docs.rs/terminal_size), which this crate already depends on for the CLI. + +## Rendering components + +The submodules that build the chart are public and available for custom renderers. + +* `proportion_bar::ProportionBar` holds five counts, `level0` through `level4`, the number of blocks at each shading level. `display(alignment)` turns it into a `Display` value. The block characters are the `ProportionBarBlock` constants `LEVEL0_BLOCK` (`█`) through `LEVEL4_BLOCK` + (a space). +* `tree::TreeSkeletalComponent` renders one cell of the branch drawing; + `tree::TreeHorizontalSlice` is a horizontal span of it. +* `child_position::ChildPosition` distinguishes the last child from the rest, deciding whether the branch character is a corner or a tee. Build it with `ChildPosition::from_index(index, count)`. +* `parenthood::Parenthood` distinguishes a node with children from a leaf. Build it with + `Parenthood::from_children_count(count)`. + +For most purposes, walking the `DataTree` yourself and formatting sizes with +[`Size::display`](Library-Sizes-And-Formatting.md) is simpler than reusing these pieces. + +## Ordering + +The visualizer renders children in tree order and performs no sorting. Sort the tree first, as in +[DataTree](Library-DataTree.md#sorting), or the chart shows entries in filesystem order. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..fd35ab6d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,33 @@ +# Documentation + +`parallel-disk-usage` is a highly parallelized directory tree analyzer. It ships as the `pdu` +command-line program and as a reusable Rust library. + +## Command-line program + +* [CLI Usage](CLI-Usage.md) documents every flag and argument of `pdu`. + +## Library + +* [Library Overview](Library-Overview.md): the measure, transform, render pipeline and the central types. +* [Getting Started](Library-Getting-Started.md): installation and a first program. +* [Feature Flags](Library-Feature-Flags.md): what to enable, and what is platform-conditional. + +### Core API + +* [Building Trees](Library-Building-Trees.md): `FsTreeBuilder` and `TreeBuilder`. +* [DataTree](Library-DataTree.md): inspecting, sorting, filtering, and reflecting the measured tree. +* [Sizes and Formatting](Library-Sizes-And-Formatting.md): `Size`, `Bytes`, `Blocks`, `GetSize`, and + `BytesFormat`. +* [Reporters](Library-Reporters.md): progress and error reporting during a scan. +* [Visualizer](Library-Visualizer.md): rendering the ASCII chart. +* [Hardlinks](Library-Hardlinks.md): detecting shared inodes and correcting for them. +* [JSON Data](Library-JSON-Data.md): the interchange format shared with `--json-output`. + +### Reference + +* [Recipes](Library-Recipes.md): examples for common tasks. +* [Module Index](Library-Module-Index.md): a map of the crate. + +Generated API documentation is at +[docs.rs/parallel-disk-usage](https://docs.rs/parallel-disk-usage).