Add rclone copy#12337
Conversation
| def transfers = task.cpus | ||
| def checkers = task.cpus |
There was a problem hiding this comment.
Are these supposed to both be the same?
There was a problem hiding this comment.
you are correct. I've modified now to respect the ratio between transfers and checkers set by default in the tool.
https://rclone.org/flags/
|
|
||
| input[0] = [ | ||
| [ id:'test' ], | ||
| ':http:' + test_file_path, |
There was a problem hiding this comment.
This is a little ugly, can it not directly take the https: path?
There was a problem hiding this comment.
Thanks for the comment! This is what I have found to make it work with https but please let me know if you have any other solution.
https://rclone.org/http/
There was a problem hiding this comment.
Fine.
One thought would be whether you could add the http: inside the module...
There was a problem hiding this comment.
okay, I thought it would be best for each pipeline to handle this outside but makes sense. What do you think now?
Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
…ules into create_Rclone
3 participants
I added a new module for the RCLONE suite: rclone copy.
This module uses rclone copy to transfer files from the source path to the destination path, while allowing extra arguments to be passed through task.ext.args. This makes it possible to configure options such as logging, stats, certificate handling, chunk size, HTTP URL support and dry-run behavior from the pipeline config.
Notes:
For the nf-test, I am using the same input and output location together with --dry-run. The module would not have write permissions to perform the actual copy in the test environment, so running it this way allows us to mock how the command would be executed without requiring remote write access.
This is related to nf-core/datasync#45