fix(abstract-substrate): blake2_256 prehash signablePayload over 256 bytes

[abhijit0943]

Summary

Substrate signs the raw encoded ExtrinsicPayload only when it is at most 256 bytes; for payloads larger than 256 bytes it signs the blake2_256 hash of those bytes instead (Polkadot.js @polkadot/types/extrinsic/util, mirrored in the HSM firmware).

The signablePayload getter in @bitgo/abstract-substrate always returned the raw payload. For large extrinsics — e.g. POLYX switchValidator/nominate with 6+ validators — the BitGoJS user signed the raw bytes while the HSM signed the hash, so the two parties signed different messages and TSS G-share combine failed with InvalidSignature.

This fix makes signablePayload return the bytes that are actually signed:

• length <= 256 → raw bytes (unchanged)
• length > 256 → blake2_256(raw) (32 bytes)

The decision is extracted into a shared utils.getSubstrateSigningBytes(raw) helper (with the threshold as the MAX_RAW_SIGNING_PAYLOAD_BYTES constant) so the rule lives in one documented place and gives the future DOT fix a copy-paste target. All downstream consumers (sdk-coin-polyx, recovery flows) then get the correct signableHex at the source, with no coin-specific logic needed in WP.

Testing

• sdk-coin-polyx nominateBuilder: small nominate extrinsic returns raw bytes (≤256, not 32); 8-validator nominate returns a 32-byte hashed payload.
• sdk-coin-polyx batchStakingBuilder: bond + nominate with 1 validator stays raw (≤256); with 6 validators returns a 32-byte hash. The batch wraps nominate and crosses the threshold sooner, so it is covered explicitly.
• Boundary test through the helper: exactly 256 bytes stays raw, 255 stays raw, 257 hashes to 32 bytes (off-by-one guard).
• Existing POLYX MPC/TSS unit tests (including "should add TSS signature") pass with the updated semantics.

Notes / follow-ups

• sdk-coin-dot has its own standalone signablePayload getter with the same raw-only bug, and the DOT wasm path (wasmTx.signablePayload()) needs separate verification. Both are out of scope here per the issue and tracked as a DOT follow-up; the new helper is the intended drop-in for that fix.
• This SDK change only fixes prod once WP bumps @bitgo-beta/abstract-substrate; the local blake2 workaround in WP polyx.ts should be reverted at that point to avoid duplicate logic.

[linear-code]

SI-911

[abhijit0943]

@claude review

[github-actions]

Claude finished @abhijit0943's task —— View job

---

Code Review

Reviewing the PR changes for the blake2_256 prehash fix:

Todo List:

• Read and analyze the main transaction.ts changes
• Review test coverage in polyx test files
• Check for potential edge cases or issues
• Provide comprehensive review feedback

Analysis

This PR addresses a critical issue where BitGoJS and HSM were signing different messages for large Substrate extrinsics, causing TSS signature combination failures. The fix correctly implements the Substrate signing protocol where payloads >256 bytes are blake2_256 hashed before signing.

Main Implementation (modules/abstract-substrate/src/lib/transaction.ts)

✅ Correct Logic Implementation:

• The fix at transaction.ts:547-553 properly implements the 256-byte threshold rule
• Uses blake2AsU8a(raw, 256) which produces a 32-byte hash for large payloads
• Maintains backward compatibility for small payloads (≤256 bytes)
• Added proper import for blake2AsU8a from @polkadot/util-crypto

✅ Documentation:
The updated JSDoc comment clearly explains the rationale and references Polkadot.js implementation, which helps future maintainers understand this non-obvious behavior.

Test Coverage (nominateBuilder.ts & batchStakingBuilder.ts)

✅ Comprehensive Test Scenarios:

• Small extrinsic test: Validates raw payload returned (≤256 bytes, not 32 bytes)
• Large extrinsic test: Validates blake2 hash returned (exactly 32 bytes)
• Both test files cover their respective transaction types (nominate vs batch)
• Tests use realistic validator counts (8 validators) to exceed the threshold

✅ Test Quality:

• Tests verify actual byte lengths, not just existence
• Clear assertions that distinguish between raw bytes vs hashed payload
• Good comments explaining the threshold behavior

Security & Correctness

✅ Security: No security issues identified. The fix aligns with the established Substrate protocol.

✅ Edge Cases Handled:

• Boundary condition (exactly 256 bytes) handled correctly with > comparison
• Import dependencies properly added
• No breaking changes to existing small transaction flows

Minor Suggestions

1. Consider adding a constant for the 256-byte threshold:

   private static readonly HASH_THRESHOLD = 256;

   This would make the magic number more maintainable across the codebase.

2. Test edge case: Consider adding a test for exactly 256 bytes (boundary condition), though the current tests adequately cover the core scenarios.

Overall Assessment

This is a high-quality fix that:

• ✅ Addresses the root cause of TSS signature failures
• ✅ Follows Substrate protocol specifications correctly
• ✅ Maintains backward compatibility
• ✅ Includes comprehensive test coverage
• ✅ Has clear documentation

The implementation is technically sound and the test coverage demonstrates the fix works correctly for both small and large extrinsics.

Recommendation: ✅ APPROVE - This PR effectively resolves the signing mismatch issue and is ready for merge.