HyperQueue not submitting allocations

[svatosFZU]

I am using hq-nightly-2025-06-23-3020ab5c8bca3548b2afbc04623bc191c33550bc-linux-x64 to test #886

Since I started testing it yesterday, I have noticed that the HyperQueue is not submitting any batch jobs. It submits one to test when a new allocation queue is made but nothing more, even though there is enough work to fill multiple workers. I tried allocation queue with and without --min-utilization=1 but it makes no difference.

I see no errors in the debug log: https://www.fzu.cz/~svatosm/hq-debug-output.log

[Kobzol]

I will take a look, but note that autoalloc probably won't function correctly at all right now if you use >1 CPU cores, until #934 is merged.

[svatosFZU]

Thanks for the info but that does not seem to be the cause. The log above is from Karolina. I have the same situation on Barbora, where I have only single-core jobs now.
debug log: https://www.fzu.cz/~svatosm/hq-debug-output-barbora.log

[Kobzol]

Hi, could you please try again now that #934 has been merged? We have extended the logging to make it easier to debug what's going on, and now the queues should estimate resources in a smarter way.

[svatosFZU]

Hi,
I tried with hq-nightly-2025-06-29-aecde5c53dd41843cc566d87e6fe8036ba26a8fa-linux-x64 . The new log from Barbora is at:
https://www.fzu.cz/~svatosm/hq-debug-output-20250701.log

[Kobzol]

Thanks! Seems like min utilization isn't working properly, could you please try without it?

@spirali Seems like min_utilization should be ignored when we have a partial worker query?

[svatosFZU]

Hi,
I removed the allocation queue with min_utilization and created a new one (without it). Jobs are still not being submitted. A debug log (continuation of previous one) is at:
https://www.fzu.cz/~svatosm/hq-debug-output-20250701a.log

[Kobzol]

It seems like an allocation was created and a worker has connected from it?

[2025-07-01T10:16:01.883Z DEBUG hyperqueue::server::autoalloc::process] Determined allocations to submit: [1]
[2025-07-01T10:16:01.900Z DEBUG hyperqueue::server::autoalloc::queue::common] Running command `sbatch /home/svatosm/.hq-server/001/autoalloc/2/001/hq-submit.sh`
[2025-07-01T10:16:01.908Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Sbatch output: Submitted batch job 1723243
[2025-07-01T10:16:01.909Z INFO  hyperqueue::server::autoalloc::process] Queued 1 worker(s) into queue 2: allocation ID 1723243
[2025-07-01T10:16:05.141Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:05.141Z DEBUG hyperqueue::server::autoalloc::process] Handling message RemoveQueue { id: 1, force: false, response: Response token }
[2025-07-01T10:16:05.145Z DEBUG hyperqueue::server::autoalloc::process] Removing queue 1
[2025-07-01T10:16:05.146Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:07.599Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:07.599Z DEBUG hyperqueue::server::autoalloc::process] Handling message GetQueues(Response token)
[2025-07-01T10:16:07.599Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:08.131Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:08.133Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:08.239Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:08.244Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:08.355Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:08.355Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:08.456Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:08.456Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:08.563Z DEBUG hyperqueue::server::client] New client connection
[2025-07-01T10:16:08.563Z DEBUG hyperqueue::server::client] Client connection ended
[2025-07-01T10:16:13.481Z INFO  tako::internal::server::rpc] Worker 2 registered from 10.32.4.178:47062
[2025-07-01T10:16:13.481Z DEBUG hyperqueue::server::state] New worker id=2
[2025-07-01T10:16:13.481Z DEBUG hyperqueue::server::autoalloc::process] Handling message WorkerConnected { id: WorkerId(2), config: WorkerConfiguration { resources: ResourceDescriptor { resources: [ResourceDescriptorItem { name: "cpus", kind: Groups { groups: [["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17"], ["18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35"]] } }, ResourceDescriptorItem { name: "mem", kind: Sum { size: 191851.8476 } }] }, listen_address: "cn178.barbora.it4i.cz:43711", hostname: "cn178.barbora.it4i.cz", group: "1723243", work_dir: "/tmp/hq-workeraKltCH/work", heartbeat_interval: 8s, overview_configuration: OverviewConfiguration { send_interval: None, gpu_families: Set({}) }, idle_timeout: Some(300s), time_limit: Some(43140s), on_server_lost: FinishRunning, max_parallel_downloads: 4, max_download_tries: 8, wait_between_download_tries: 1s, extra: Map({"JobManager": "{\"manager\":\"Slurm\",\"allocation_id\":\"1723243\",\"time_limit\":{\"secs\":43188,\"nanos\":0}}", "ProcessPid": "3593507"}) }, manager_info: ManagerInfo { manager: Slurm, allocation_id: "1723243", time_limit: Some(43188s) } }
[2025-07-01T10:16:13.481Z DEBUG hyperqueue::server::autoalloc::process] Registering worker 2 for allocation 1723243
[2025-07-01T10:16:13.481Z INFO  hyperqueue::server::autoalloc::process] Worker 2 connected from allocation 1723243

[svatosFZU]

Right. When the new allocation queue is submitted, one worker is made and it is running for moment. But no jobs were running in it and no workers were made since.

[Kobzol]

I see. Yes, the worker has ended because of idle timeout, and it wasn't re-created again, because the automatic allocator figured out that this worked is unable to execute the submitted tasks.

The problem is that the connected worker doesn't have enough memory ({ name: "mem", kind: Sum { size: 191851.8476 } ), because the task asks for 192000. Try lowering the mem request of the tasks to e.g. 191000.

[svatosFZU]

Thanks, I will try that. So, I was going to cancel these jobs asking with a lot of memory. When I tried that, the HyperQueue crashed. So, this is just FYI:
https://www.fzu.cz/~svatosm/20250701131502-hq-debug-output.log

[spirali]

Thanks, I will try that. So, I was going to cancel these jobs asking with a lot of memory. When I tried that, the HyperQueue crashed. So, this is just FYI: https://www.fzu.cz/~svatosm/20250701131502-hq-debug-output.log

Thank you for the report. #949 should fix this crash.

[svatosFZU]

Hi,
just to confirm, the new jobs with lower memory requests (which somehow worked before) are being started and run.

[Kobzol]

I think that nothing changed about mem. However, before, the auto allocator was essentially dumb, and would spawn allocations and workers if there were any tasks without an assigned worker. Now it's much smarter and only creates allocations in queues that can actually fulfill the resource requests of waiting tasks*. This is what exposed the mem issue.

* Only the first allocation will be performed in a "semi-dumb" mode, because we need to figure out what are the precise resources of workers spawned in a given queue.