Skip to content

Add a unit argument to the length validator (utf16 / bytes) - #398

Open
ChrisJr404 wants to merge 1 commit into
Keats:masterfrom
ChrisJr404:length-unit-param
Open

Add a unit argument to the length validator (utf16 / bytes)#398
ChrisJr404 wants to merge 1 commit into
Keats:masterfrom
ChrisJr404:length-unit-param

Conversation

@ChrisJr404

Copy link
Copy Markdown

Closes #250.

Motivation

The length validator counts strings in Unicode scalar values (chars, via str::chars().count()). That doesn't match how an HTML form field's maxlength or JavaScript's String.length count characters — both use UTF-16 code units. So for non-ASCII input the backend and the frontend can disagree about whether a value is valid.

Concretely, "🖐🏽" is 2 chars, but 4 UTF-16 code units and 8 UTF-8 bytes. A form that lets the user type up to 4 "characters" of that emoji can produce a value the current length validator rejects (or accepts) inconsistently with the browser.

In #250 you suggested handling this with a mode/unit parameter on the length validator rather than a separate length_utf16 validator, so that unicode / utf-16 / bytes are all covered by one option. This implements that.

What this does

Adds an optional unit argument to length:

#[validate(length(max = 10, unit = "utf16"))]
title: String,

#[validate(length(min = 1, max = 255, unit = "bytes"))]
slug: String,
  • unit = "chars" (default) — Unicode scalar values, i.e. today's behaviour
  • unit = "utf16" — UTF-16 code units (matches maxlength / JS String.length)
  • unit = "bytes" — length of the UTF-8 encoding

The unit only changes how string-like values are measured. Collections (Vec, HashMap, arrays, …) are still measured by their number of elements, and the unit is ignored for them. When a unit is set it's also added to the error as a unit param, alongside min/max/equal.

Implementation notes

  • ValidateLength gains validate_length_with_unit and length_with_unit, both with default implementations. length_with_unit defaults to length(), so every existing impl (including user-provided ones that only implement length()) keeps working unchanged. The string impls and the deref/Cow/Option forwarders override it so the unit is honoured through wrappers like &String and Option<String>.
  • validate_length now delegates to validate_length_with_unit(LengthUnit::Chars, …), so its behaviour is identical to before. LengthUnit is exported from the crate root.
  • The derive parses unit = "chars" | "utf16" | "bytes" (a couple of aliases like unicode/utf8 are accepted too) and emits the unit-aware call only when the argument is present; otherwise it emits the exact same validate_length(...) call as before, so output for existing code is unchanged.

Tests

Added unit tests for the trait (utf16/bytes/chars, ASCII equivalence, forwarding through &String/Option/Cow, and the collection no-op) and derive integration tests in validator_derive_tests/tests/length.rs (utf16 max, bytes min/max, explicit-chars-equals-default, the exposed unit param, and Option<String>).

cargo test -p validator and cargo test -p validator_derive_tests --test length pass; cargo fmt --check and cargo clippy are clean. Also updated the README length section and the changelog.

The length validator has always counted strings in Unicode scalar values
(chars). That does not match how an HTML form's maxlength or JavaScript's
String.length count characters, which use UTF-16 code units, so backend
validation can disagree with frontend validation for non-ASCII input.

Add an optional `unit` argument (`chars` (default), `utf16`, `bytes`)
that selects how string-like values are measured. Collections are still
measured by their number of elements regardless of the unit.

This exposes `LengthUnit` and `ValidateLength::validate_length_with_unit`;
`validate_length` now delegates to it with the default `chars` unit, so
existing behaviour and custom impls are unchanged.

Closes Keats#250
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: Consider an UTF-16 code units length validator

1 participant