Refactor replicate object with async await - #2780
Conversation
Hello sylvainsenechal,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
| /** | ||
| * Runs an async task for each item in a collection, up to `limit` at a time. | ||
| * On error, no new tasks are started, but already-running tasks are awaited | ||
| * before the function resolves. Returns [firstError, results] so that callers |
There was a problem hiding this comment.
Return the first error only : Keeping same behavior as before with mapLimitWaitPendingIfError.
Could've returned all errors as a list but not needed now and prefer to keep existing behavior.
Cannot aggregate errors neither, as we need to keep the original error to check its properties to determine if its retryable or not
There was a problem hiding this comment.
but already-running tasks are awaited what happens if an error is triggered here ? Can make sense to stop new flows on error, but for the current one we can miss an error ?
There was a problem hiding this comment.
Ok I saw your comment. I'm not sure it's the right approach and we should return all errors ?
There was a problem hiding this comment.
Mhh its replicating initial pre migration behavior, I think its fine anyways as replicateObject will try to delete all orphans as soon as a single error is found
Although maybe there is a question raised about reusability of this function, maybe its built to be too custom for replicateObject and should be more generic (return array of errors) ?
There was a problem hiding this comment.
refactored with async await, in the file runTaskWithConcurrency
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files
@@ Coverage Diff @@
## development/9.5 #2780 +/- ##
===================================================
- Coverage 75.43% 75.42% -0.01%
===================================================
Files 201 201
Lines 13935 13934 -1
===================================================
- Hits 10512 10510 -2
- Misses 3413 3414 +1
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
406a3d4 to
7f5778f
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
7f5778f to
5680800
Compare
5680800 to
64afb79
Compare
There was a problem hiding this comment.
Migrations here aren't the main point of this pr, but were kinda forced as MultipleBackend extends ReplicateObject.
I did the minimum required migration on this file
| await this._deleteOrphans(destEntry, destLocations, log); | ||
| throw err; | ||
| } | ||
| return this._handleReplicationOutcome(null, sourceEntry, destEntry, kafkaEntry, log, done); |
There was a problem hiding this comment.
_handleReplicationOutcome(null, …, done) is called from inside the try: if it throws after calling done, the catch calls it again and done fires twice. We can compute the error in the try/catch and call the handler exactly once after it. (same at line 909 and in _processQueueEntryRetryFull)
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
The following reviewers are expecting changes from the author, or must review again: |
64afb79 to
7467f4b
Compare
|
Sorry I took some time to respond, you can review again next week @benzekrimaha @maeldonn |
7467f4b to
c031658
Compare
DarkIsDude
left a comment
DarkIsDude
left a comment
There was a problem hiding this comment.
Please try to create dedicated commit for prettier format as mention in the best practice of the migration 🙏. You can also use "Hide whitespace" on Github to ease diff if you want.
CI is red, maybe you should have a look ?
| // Sets up source clients using the role from the replication | ||
| // configuration if the authentication type is as such. | ||
| return this._setupRoles(entry, log, cb); | ||
| return this._setupRoles(entry, log).then(() => cb()).catch(cb); |
There was a problem hiding this comment.
This is always weird to read.
| return this._setupRoles(entry, log).then(() => cb()).catch(cb); | |
| return this._setupRoles(entry, log).then((_result) => cb()).catch(cb); |
_ is a convention in JS for unused parameter. The linter should not complain, what do you think ? I guess this can be an error if the callback trigger an error ?
There was a problem hiding this comment.
And I think we have the same issue here with .catch
There was a problem hiding this comment.
yeah its nicer to have _result, I also see i messed up some stuff around the catches, its tricky
edit : no : Error: 1190:51 error '_result' is defined but never used no-unused-vars
| actionFunc(_handleRes, nbRetries); | ||
| } | ||
|
|
||
| retryAsync(args) { |
There was a problem hiding this comment.
We usually prefer to keep the function retry and the last argument (done) is optional ?
There was a problem hiding this comment.
Yeah im moving this at the top of retry
There was a problem hiding this comment.
Did you get a try to some library to do that ?
There was a problem hiding this comment.
I asked ai to create this function, although I also looked online at how people do it and the solutions were very similar.
Definitly there are libraries for this, but taking a look rapidly I only small non popular one, can't find a proper "do parallel stuff with js" thats very popular and battle tested.
Here we also have some specific behavior, about continuing ongoing task when one error is found, so not sure the available libraries would offer that.
I think its fine to not use a lib
| /** | ||
| * Runs an async task for each item in a collection, up to `limit` at a time. | ||
| * On error, no new tasks are started, but already-running tasks are awaited | ||
| * before the function resolves. Returns [firstError, results] so that callers |
There was a problem hiding this comment.
but already-running tasks are awaited what happens if an error is triggered here ? Can make sense to stop new flows on error, but for the current one we can miss an error ?
| const errors = []; | ||
|
|
||
| const worker = async () => { | ||
| while (nextIdx < coll.length) { |
There was a problem hiding this comment.
Let say
Awhile (nextIdx < coll.length) {Bconst idx = nextIdx++;
With 2 workers you can have A1 B1 A1 A2 B1 B2 and then trigger an error as coll is only size 2 ?
There was a problem hiding this comment.
no I think this is fine, its js single threaded code even if it kinda looks weird, this wont run at the same time 🤔
| /** | ||
| * Runs an async task for each item in a collection, up to `limit` at a time. | ||
| * On error, no new tasks are started, but already-running tasks are awaited | ||
| * before the function resolves. Returns [firstError, results] so that callers |
There was a problem hiding this comment.
Ok I saw your comment. I'm not sure it's the right approach and we should return all errors ?
| outcome = err; | ||
| } | ||
| const result = await this._handleReplicationOutcome(outcome, sourceEntry, destEntry, kafkaEntry, log); | ||
| return result === null ? done() : done(null, result); |
There was a problem hiding this comment.
This is what we don't want to do in async migration. done should not be called inside the async function 🙏. We did a skill to ease migration (and a confluence page) where we explain the why
| next => this._refreshSourceEntry(sourceEntry, log, (err, res) => { | ||
| if (err && err.name === 'ObjNotFound' && | ||
| sourceEntry.getReplicationIsNFS() && !sourceEntry.getIsDeleteMarker()) { | ||
| next => this._refreshSourceEntry(sourceEntry, log) |
There was a problem hiding this comment.
This will runs the next waterfall stage synchronously, and that stage is not async — metricsHandler.rpo() can all throw synchronously for example. A throw there lands in .catch, which calls next again; async.waterfall's onlyOnce then throws "Callback was already called" as an uncaught exception in the queue processor, masking the real error. Use the two-argument then so the success path can't feed the rejection handler
next => this._refreshSourceEntry(sourceEntry, log)
.then(res => next(null, res), err => {
There was a problem hiding this comment.
aiya its more complicated than i thought to make these refactoring
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
The following reviewers are expecting changes from the author, or must review again: |
449cafe to
05fd0ff
Compare
I didnt run the linter though, the diff is just re indentation mostly 🤔 |
05fd0ff to
03574e0
Compare
this commit is mandatory as multipleBackend is a class that extends replicateObject and some functions are overloaded Issue: BB-803
03574e0 to
6dd6d10
Compare
4 participants
Issue: BB-803
Review hints :
Imo this refactor is super overdue and not so complicated to do.
Refactor replicate object with async await