I have a case right now where I'm running taskgraph graph on a slurm cluster, which requires that I request the number of CPU cores that I think I'm going to need.
Looking at glances or top while I'm running the graph, I can see that the most tasks that I'm actually executing in parallel are about 5 at a time. It would be really great if taskgraph could warn be about egregious over-allocation of workers, like if only 2 or 3 processes are ever used at a time but n_workers was set to 20 or something, then we could see a warning in the logs.
The specific case I'm encountering is that there are 20 CPUs allocated, but the most I'm ever using (as far as I can tell) is 12.
I have a case right now where I'm running
taskgraphgraph on a slurm cluster, which requires that I request the number of CPU cores that I think I'm going to need.Looking at
glancesortopwhile I'm running the graph, I can see that the most tasks that I'm actually executing in parallel are about 5 at a time. It would be really great iftaskgraphcould warn be about egregious over-allocation of workers, like if only 2 or 3 processes are ever used at a time butn_workerswas set to 20 or something, then we could see a warning in the logs.The specific case I'm encountering is that there are 20 CPUs allocated, but the most I'm ever using (as far as I can tell) is 12.