fix(net): reduce memory usage in transaction queues

[317787106]

What does this PR do?

Reduce memory retained by incoming transaction queues and avoid unnecessary serialization allocations in TransactionsMsgHandler.

• Strip transaction-level unknown fields and redundant ret entries before admission. Both smart-contract events and ordinary transaction tasks retain the cleaned transaction.
• Drop transactions whose serialized size still exceeds 500 KiB after cleanup, before either queue retains them.
• Calculate transaction IDs through TransactionCapsule, avoiding the full transaction byte arrays created by temporary TransactionMessage objects during request checks and cleanup.

Why are these changes required?

Queued transactions could retain removable padding or oversized payloads until worker processing and downstream validation. During a backlog, these bytes accumulate in protobuf objects and, for smart-contract messages, cached serialized data. Cleaning and checking transactions before admission reduces this retained memory, while avoiding temporary message construction reduces allocation pressure.

The cleanup preserves signed raw_data and transaction IDs. The size check is an early admission guard; full transaction validation still runs downstream.

This PR has been tested by:

• 25 targeted tests passed locally, including four new tests covering both transaction paths: padding removal, oversized payload rejection, the exact 500 KiB boundary, and redundant-result cleanup. Assertions also verify retained message bytes and transaction IDs.
• checkstyleMain, checkstyleTest, and git diff --check passed.

[3for]

[NIT] check() and the request-cleanup loop each create a new capsule and compute the transaction ID. Accepted transactions later compute the ID again in the worker using another capsule. Oversized transactions undergo at least two raw-data hash computations before being dropped, which could be optimized.

[317787106]

Updated in bdafbed. check() now returns its existing ID set for reuse during request cleanup, eliminating the second raw-data serialization and hash computation. Cleanup still happens only after the entire batch passes validation, with regression tests covering this behavior.

Worker-side hashing remains unchanged. Given its low CPU overhead in load tests and production, and expected throughput of at most 10,000 TPS, further optimization is deferred to keep this PR focused on reducing memory usage.