Workload hangs in D+ state during graph initialization (demotion issue)

[mshcherba]

Hello! Thank you for open-sourcing the PACT artifact. I am currently trying to reproduce your experiments on a CloudLab c220g5 node.

I've run into an issue where the workload hangs completely during the initial memory allocation phase. Specifically, when running the bc (Betweenness Centrality) workload on the Kronecker graph (using 8 threads), the memory pressure spikes during the graph loading/building phase.

Because ./run-pact.sh sets echo 0 > /sys/kernel/mm/numa/demotion_enabled, the kernel cannot demote pages to the slow tier. As a result, the fast tier runs out of memory and the process gets stuck in an uninterruptible sleep (D+) state before the actual computation phase even begins, causing the entire experiment to stall.

My question:
What is the intended methodology to handle this initial bulk allocation pressure? Is there a specific configuration, boot parameter, or script adjustment I am missing to prevent the system from deadlocking during graph loading while keeping demotion_enabled=0 for PACT's manual migration control?

Any guidance would be greatly appreciated!

[huaicheng]

We use vmtouch to load the graph into memory first before running PACT with the graph workload (the runtime phase where PACT does proactive migrations). Will take a look at the scripts and see if we missed any key steps in the doc.

Thanks for reporting this!

[mshcherba]

Hi @huaicheng,

Thanks for the quick reply!

The workload still hangs despite vmtouch because gapbs does not use mmap to read the .sg file. Instead, it dynamically allocates anonymous memory (new []) and copies the data into it via read().

Here is what happens:

1. vmtouch correctly preloads the file into the page cache on the slow tier.
2. bc starts on the fast tier (CPUs 2-9) and allocates ~4+ GB of anonymous memory to build its internal graph.
3. This anonymous allocation immediately hits the fast-tier cgroup memory.high limit.
4. Because demotion_enabled=0, the kernel cannot demote these anonymous pages. The process gets stuck in a direct reclaim loop (D+ state).

Since vmtouch only populates the page cache, it doesn't prevent the out-of-memory situation caused by gapbs's anonymous memory allocations.

Since my goal is to reproduce the experiments from the paper as closely to the original setup as possible, I'd rather avoid inventing my own workarounds. How did you handle this initialization phase in the original experiments to avoid the deadlock?

[huaicheng]

What's the available node 0 capacity in your setup? It should match the target split ratios (eg, 1:1) based on a correctly profiled RSS for the target workload. PACT would turn on kernel demotion when it runs and send signals to the kernel for triggering the actual demotions, etc.

Could you share your kernel cmdline/params, vmstat info about page migrations, etc so that we can reproduce this? We will look into this more closely over the weekend.