Skip to content

GDTF reading optimization for big files.#146

Open
VzhelevVector wants to merge 1 commit intomasterfrom
optimization-GDTF-reading
Open

GDTF reading optimization for big files.#146
VzhelevVector wants to merge 1 commit intomasterfrom
optimization-GDTF-reading

Conversation

@VzhelevVector
Copy link
Copy Markdown
Collaborator

Optimize GDTF File Reading & Reference Resolution

Summary

Performance optimization for reading and validating large GDTF files. All changes are internal — the public API is unchanged. Behavioral parity with the original code is preserved for all file sizes through a threshold mechanism that activates optimizations only for larger files.

What's New

Hash Map Indexes for Reference Lookups

Reference resolution (attributes, wheels, emitters, filters, connectors, color spaces, gamuts, DMX profiles, models, sub-physical units, geometries) now uses unordered_map with O(1) lookups instead of linear scans. For files below the activation threshold, the original linear search is used unchanged.

A flat geometry index is built once by recursively traversing the geometry tree. This allows instant name-to-geometry lookups without repeated tree walks, while preserving the actual tree structure.

Per-Mode Channel & Function Indexes

Each DMX mode builds its own channel and function indexes after channels are fully resolved. This replaces triple-nested loops (channels → logical channels → functions) with a single map lookup in macro resolution, relation resolution, and mode master resolution.

Set-Based Duplicate Detection

Duplicate geometry+attribute combination detection in logical channels now uses an unordered_set for O(n) detection instead of the original O(n²) nested loop approach. Under the threshold, the original nested loop is preserved.

Case-Insensitive Hash

A TXStringNoCaseHash struct is introduced for all hash containers. This is necessary because TXString::operator== uses case-insensitive comparison, but the default TXString::hash() is case-sensitive — using the default hash in unordered_map would break lookups where names differ only in case.

Return-by-Reference Optimization

Multiple getters that returned vectors and strings by value now return const&:

  • Array getters: GetChannelArray, GetLogicalChannelArray, GetDmxChannelFunctions, GetInternalGeometries, GetFeatureArray, GetSubPhysicalUnitArray, GetDmxRelations, GetDmxMacrosArray
  • String getters: 11 GetUnresolved* / getUnresolved* methods that return member fields

This eliminates thousands of unnecessary copies during reference resolution.

Mode Master Short-Circuit

ResolveDMXModeMasters now skips the function-level lookup (getDmxFunctionByRef) when the channel-level lookup has already resolved the mode master, avoiding a redundant search per resolved function.

Attribute Validation Optimization

CheckNodeAttributes now uses an unordered_set for attribute matching instead of repeated std::find + erase on a TXStringArray. Error reporting order is preserved by iterating the original XML attribute list.

NoFeature Attribute Handling

Early detection of the NoFeature attribute is performed once during index building, regardless of whether indexing is active. The getAttributeByRef function checks for NoFeature before searching, avoiding unnecessary scans.

Threshold Mechanism

Optimizations activate based on file complexity:

Threshold Value Controls
kIndexThreshold 128 Fixture-level hash maps (sum of attributes, wheels, models, emitters, filters, connectors, color spaces, gamuts, DMX profiles, geometries)
kModeIndexThreshold 64 Per-mode channel/function indexes (channel count in mode)
kDuplicateCheckThreshold 64 Set-based duplicate geometry+attribute detection (channel count in mode)

Files below these thresholds use the original code paths — zero overhead, zero behavioral change. Files above get O(1) lookups with emplace() (first-wins semantics, matching the original linear scan behavior).

@VzhelevVector VzhelevVector self-assigned this Apr 20, 2026
@VzhelevVector VzhelevVector requested review from AndriiVoitenko, ILane-VW and adragnevVW and removed request for AndriiVoitenko April 20, 2026 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant