OPDATA-6745: Fix memory leak in Requester

[dskloetc]

OPDATA-6745

Description

There is a memory leak in Requester.

If requests come in frequently enough that the requester queue is never empty, even if it stays at a small size, the recursive processNext function will chain Promise objects forever. Only when the request queue is empty, does the promise stack unravel to release the memory.

This happens in por-address-list, the-network-firm, tiingo and to a lesser extent finnhub-secondary because they have multiple endpoint making multiple requests. This means that even though one endpoint waits for all its requests to finish before sending a new batch, there is always a batch from another endpoint still in the queue to keep the promise chain from unraveling.

This PR fixes the memory leak by refactoring the Requester, in particular the queueing process.

Instead of dropping requests in a queue and separately handling the queue, the code is more procedural with each "thread" taking care of its own request and a queue that only coordinates how requests wait on each other to satisfy the rate limiter.

Changes

1. Add turn-queue.ts. This coordinates the different "threads" waiting on the rate limited sequentially.
2. Add tests for TurnQueue.
3. Refactor Requester using the TurnQueue.
4. Update 2 tests that were observing bugs and 1 test that seemed to be observing a bug but has behavior we want to keep.

Testing

Tested locally with the-network-firm with all 24 feeds from RDD.
We'll confirm that it fixes the memory leak when we test in staging after merging.

[github-actions]

NPM Publishing labels 🏷️

🟢 This PR has valid version labels and will cause a patch bump.

[github-actions]

🚀 Successfully created version bump PR: #813