From 1f0055515a2af565aa59e96c31770b9fb4457d22 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0001/1150] CI: retrigger From 6018983d1b0f30922c1e8b571a392686aa951a98 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0002/1150] CI: retrigger From 8930fd3dddf294bbd0c0b400ccddcf81952b5f18 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0003/1150] CI: retrigger From 2933202da5b4bc6b3c689b0b7a314e162b96a476 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0004/1150] CI: retrigger From c212caceb0fc9bab08689b8a639d5a83d094900f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0005/1150] CI: retrigger From 4988e952793b0e8c0a5676f86a5449acb2c572c8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0006/1150] CI: retrigger From 2fc8afa0fbff6d3238d7a2f4e10bc2fb46433052 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0007/1150] CI: retrigger From c22fdf3b5d90a03812a3f662f6345b99956b0811 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0008/1150] CI: retrigger From 536a571c12f07d3830865412149aa88c6579a9da Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0009/1150] CI: retrigger From ffe02f02802d60b635080be99ca3f37d7550e8c7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0010/1150] CI: retrigger From 127d1d2d37cd4e6d27c286041c6caaa241c2ff7d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0011/1150] CI: retrigger From e07ad6ac5ef31995a2341ac5396f791d9e81a56f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0012/1150] CI: retrigger From d58bcf5e2edef945b49c0056d49343af2da2ac18 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0013/1150] CI: retrigger From 3fc44c4186520c6e149b93309616977523fc43a3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0014/1150] CI: retrigger From ed8d808c76b4d06f19b3b6557745abb6dc51bd9f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0015/1150] CI: retrigger From 4fbd57100f9bb0408978ba31ee94db177e7eae0c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0016/1150] CI: retrigger From 98857b42befa9d965806d5f922b9b8cb6c4004f2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0017/1150] CI: retrigger From c539ff562e3cf4e16ce616614770002d799cb624 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0018/1150] CI: retrigger From 5409cf92b1b07bf76cd45031306273f3cfe46404 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0019/1150] CI: retrigger From b274b013f37115d47e3c3df79e024f44496d7723 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0020/1150] CI: retrigger From c95ba6028a1a139b290254bde12c8cb598c1e8b3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0021/1150] CI: retrigger From 209797addde6548fda2f0bf9cde67ced150cd0ed Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0022/1150] CI: retrigger From dfd06c5258c4056ded516f6d511137a1a0f3de22 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0023/1150] CI: retrigger From fc585ca2113d5a45200b2a8cc4db3b1cc280f117 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0024/1150] CI: retrigger From 9df2778dd3fea8d1416d8b1f30e4996c85dcda2d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0025/1150] CI: retrigger From d68516c214905745fab95e25a0ee27b5ce8b4ad1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0026/1150] CI: retrigger From 46bd476a6c4066fdf1cc7bd73e93c1ed9ce7a62d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0027/1150] CI: retrigger From a57b4baccad109839c925767d0759375cb1faa3e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0028/1150] CI: retrigger From 9bf42e33301068962519a58eb0027bfdbb41bda9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0029/1150] CI: retrigger From da6fcd11e7133af2010334b890034ffdee730b47 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0030/1150] CI: retrigger From 21df4650a5abea002d5b5f5eb628085ae9bace4a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0031/1150] CI: retrigger From ac10552ae68beb246e3e4bec82118f05137350e4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0032/1150] CI: retrigger From c6f5582d04a4d7675fcc426eb4f727f85d481f31 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0033/1150] CI: retrigger From 0e054c1859d538ffe6317e676a9c51a88aaf6dfc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0034/1150] CI: retrigger From 38d3167081c862439693f08e79a1c6ef060ebb2a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0035/1150] CI: retrigger From da0818394d2adbd8d4f3be14c70b6373efc42702 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0036/1150] CI: retrigger From 6c60c462ba2e5ea5ca69eb180ef13111e9dc67f3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0037/1150] CI: retrigger From b6f0471407d87920872c85640cb1fcc30d41f40f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0038/1150] CI: retrigger From 0fa124f77024955e26512f5b9159fe571bafc7d8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0039/1150] CI: retrigger From 14749aa2a5a4a72bf5735184fd7cf27f82bd904a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0040/1150] CI: retrigger From 9c28b0e53bd8e0b0b6a5f5c9e5a5b88fd0da4aa7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0041/1150] CI: retrigger From 27380f1b9c77dd07e592e05b7db6bbea70278fbe Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0042/1150] CI: retrigger From b88a66fae8f66892ce34d04bc1d13581f44d1a3e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0043/1150] CI: retrigger From fcb2cdf17b6889b8e9282495f40c18a1d5dca17f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0044/1150] CI: retrigger From e4d161a945b4214da6e5d8c2d56ea3a47ec9a3bc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0045/1150] CI: retrigger From 1d2eb727d95d31f1a522901b714fd01a354a827b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0046/1150] CI: retrigger From 9cdb37d513c3513cb6594e4c19d02c1151ec7243 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0047/1150] CI: retrigger From b6defdf153dc1a0509913bfe5a54689a6f9e2580 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0048/1150] CI: retrigger From 4377cfd7fe99e44d27cfc95601f421138375d8d6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0049/1150] CI: retrigger From 239f13453eb6693b24f35c3b47d8a0d316dc7d3e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0050/1150] CI: retrigger From d56f6acc419a2b66c724cc2a608da4807dbd95a4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0051/1150] CI: retrigger From 03733100ddb00864edfd86b372fec589eb9fa9ce Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0052/1150] CI: retrigger From 933400343e728645951c406c984e5097f760a32b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0053/1150] CI: retrigger From 5013e928c8b3e142b1f5aebaf1eebdf71408688b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0054/1150] CI: retrigger From 025f6e084289a26c54bd038e879a02b0c048731c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0055/1150] CI: retrigger From cf653ab56e54682ff1a7e5718c6f0c81903cbbd0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0056/1150] CI: retrigger From 9b3594da90895b563db7f084707fb4457131cbac Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0057/1150] CI: retrigger From 895704b3100872b17b5dd7d84aabe77769850371 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0058/1150] CI: retrigger From 9fb2adada184792f7e49573ebc5535965e0e84a1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0059/1150] CI: retrigger From 9c642e1dbba0e0e968f37881129401420e5f7510 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0060/1150] CI: retrigger From 2298734ee7095ba17d62bf1492ca720b3fd86ef7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0061/1150] CI: retrigger From c1d8f761b63ec2987da6b1b07a012b86ed78b4f6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0062/1150] CI: retrigger From 48939500238795c360cef93c3b11279fc0e01b7e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0063/1150] CI: retrigger From 1407b6a27ccc8332527f5cceffc92a282a97f549 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0064/1150] CI: retrigger From 6159ca6ac22fc9e93dcd845ecc465a7de40aa239 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0065/1150] CI: retrigger From f6aa054586ef1f77a8787654798a7f7a97107987 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0066/1150] CI: retrigger From 5189d063582b9dc67c8095a62b48eecc04e0351f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0067/1150] CI: retrigger From 26b03cb41a85d89b6525228ecf996ab9724bfb0b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0068/1150] CI: retrigger From dd839ae117e5a054f4cc1f8178e49d1dd17a27a8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0069/1150] CI: retrigger From 2f58bca27bb4e07b8aabe2955535daa85e44a9c4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0070/1150] CI: retrigger From 9f83e3b4587f0fd72b495daec74559a66d89600e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0071/1150] CI: retrigger From 0ac08ea479e4cb1faee6ffc8075f6607b4f6863e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0072/1150] CI: retrigger From e10ec311433994f027066f3c804d65c0176c8ef2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0073/1150] CI: retrigger From b44e6c6b055f1ddb44ecde7ed86d0848f64cad24 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0074/1150] CI: retrigger From c0848b3fbb55c470d5a9b2de8a04eca38e8ac950 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0075/1150] CI: retrigger From 641b44514c520f3a3bb736c17b75193c4501aebb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0076/1150] CI: retrigger From cd857e44745120433b0afc8b66c272cec57d11b2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0077/1150] CI: retrigger From a6735d4fe6f1d44a7b99557352804c4446314fd3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0078/1150] CI: retrigger From 420915414c40be6891c1f86ae032d71554b72913 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0079/1150] CI: retrigger From 910ea79e58d6084eddb3bb3eee2c19956ddacb2c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0080/1150] CI: retrigger From cbef7b947af1f6aebe7492276761ffb0207dedde Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0081/1150] CI: retrigger From a17ce0cccab92d96e51762ad90e50827cbcb9be6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0082/1150] CI: retrigger From 58e639dacc28160b52a09b94ac3c3e027898283a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0083/1150] CI: retrigger From a2da07b2af5eaa58711c333f89a365878570b2c5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0084/1150] CI: retrigger From c4cf9361043e6575e7cf90c2932588cd99eded2a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0085/1150] CI: retrigger From 30301aa1331b8c2eefd35bad780ed569b8a60bba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0086/1150] CI: retrigger From c9707de79ea155e4b9c955bc44b2624f188ed43a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0087/1150] CI: retrigger From 07ac41cb4c60a6ae633f9adb119c460507d2b506 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0088/1150] CI: retrigger From 4df1b4ba2a816af7306db87d1798a9b3f6382b9e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0089/1150] CI: retrigger From 6418ea43b968bf51cfc6b37af8bacaf0f5b66702 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0090/1150] CI: retrigger From e7015ba76cf8f9a033cde631e088241b0fe4eb12 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0091/1150] CI: retrigger From 3b37e1acd8ba2a73b46732e7d0244a7037e2043f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0092/1150] CI: retrigger From dc4c25b0def178b99952e889ea96721e4319485f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0093/1150] CI: retrigger From fbba2fdecf9c43fbdad7beadf43186472c1690c8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0094/1150] CI: retrigger From d4234acb8a034b1f2773a6d8b56a8d83fdffc28b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0095/1150] CI: retrigger From 3c747b26847a23a2a89c53d4ca2665ed6fa18742 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0096/1150] CI: retrigger From 13dff77ca580a736c94e02210f77d192e47fffa3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0097/1150] CI: retrigger From 253ba23ff9974760ec5831517b3a822cf8b441fa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0098/1150] CI: retrigger From 52d62f6f64458a376520496506917489c77e84fd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0099/1150] CI: retrigger From 073b89d220055f0cff4f57d6763448b991cf3883 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0100/1150] CI: retrigger From d1679d31ca6a291124a19f73a5ba3cabadce22ac Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0101/1150] CI: retrigger From d933822959a80b93703feabf5ff29117bd8bd021 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0102/1150] CI: retrigger From 31a256771c0fab4d1e11a53f4671fe867e2fbfa9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0103/1150] CI: retrigger From f9b2ffba9562d850f2c458115ff56bbd3e175a9f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0104/1150] CI: retrigger From 41959a83b40908b196970136dc5bf11f0561903b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0105/1150] CI: retrigger From 20b3dc0ccd3421bbb7c64b617641f7341566d388 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0106/1150] CI: retrigger From a669bc922c2eab80ba4a8f8ae4652245d404d61c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0107/1150] CI: retrigger From cd39ed2160f3c3f3e349439372cf8b2c8719a9e7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0108/1150] CI: retrigger From 078f75d8460e1435abe220e4e2345a8e1e0e548d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0109/1150] CI: retrigger From 031d1075c184263f91fc8c9876bb56cbb8b86a5f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0110/1150] CI: retrigger From ed750b666d449f40f87c763f438c52cac1c94c10 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0111/1150] CI: retrigger From 414c16f038946ae1f4b7d1b9217a7e76cc02005f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0112/1150] CI: retrigger From cfe9b940338828436a848be8e2942f10ba4be24c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0113/1150] CI: retrigger From 7f4cfa4f8cfc2848b11b844cb86d917848f7d252 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0114/1150] CI: retrigger From 8508459d345608f21e0a0b142ae8d04401921e65 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0115/1150] CI: retrigger From 77e91f71de64cadd74f61207d64b0e83356ce858 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0116/1150] CI: retrigger From 12618fc246f4eb66aee4755eec3880416385af51 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0117/1150] CI: retrigger From 6ac55cf0b82b1d239d1dcd0700c2f9b0e3f509b4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0118/1150] CI: retrigger From c17af7672f7679e1054fe9ae6be440d43c940789 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0119/1150] CI: retrigger From 87678158f7baf6abca6977505fc44bbb656c963a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0120/1150] CI: retrigger From 51b99f8dae03c9b17e1401aa4596c1005296d5af Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0121/1150] CI: retrigger From 6688240b260897ba2b2e717d801c755043376da9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0122/1150] CI: retrigger From f16d3272badce402442826e239f0ab64b56de1d2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0123/1150] CI: retrigger From 47d962fc96d47c7312ad350ad2df56018f0a2870 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0124/1150] CI: retrigger From 5ebfd579d41b9f78142fd59834415bffc101980c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0125/1150] CI: retrigger From ba18049b5a5fab60772e99e2dd18374d3ea8bec8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0126/1150] CI: retrigger From 63b2bf37f91e78a957293d5487e5152d4be2edc7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0127/1150] CI: retrigger From 8e0dab67a8ee2ed6567b5982c59b26e3f87c00b9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0128/1150] CI: retrigger From ecd9e136d843df7c67d15456457c1fb27a371ccb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0129/1150] CI: retrigger From 2f503802240fecda3df5034fe7d0786621879352 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0130/1150] CI: retrigger From 63e7816499a0fda5c00bd847cd896b0c8bc43d8b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0131/1150] CI: retrigger From fa39da7f72b61d2477eb6c71770822b981a6c885 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0132/1150] CI: retrigger From 978be7f9ac8d92bdec2c0482df943de0e0b3c30f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0133/1150] CI: retrigger From f8d3393fb213959b08fa3ebe302616ffd3b97877 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0134/1150] CI: retrigger From 3353ddcda521157c4a281c245d424e834bc6c5bc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0135/1150] CI: retrigger From d1f988d314ddbdec28eae23e74be0f417302b10f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0136/1150] CI: retrigger From 1946ac053141b52862252665944645227b0017b6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0137/1150] CI: retrigger From 588ac09098b8d625c071632ddb0522768db4f677 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0138/1150] CI: retrigger From 4e130e39d71cc6159226bd2f403bf7c3e009971d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0139/1150] CI: retrigger From d2e27921ee594f3d61bcbf9fd423bab0c5146956 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0140/1150] CI: retrigger From 3a7fdb94865fea03bab4f59716df01f70d312eff Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0141/1150] CI: retrigger From f861c20b960d3ae05eddcbaf17c41f27f07a279e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0142/1150] CI: retrigger From 16e4c050b7f67aecba4595b34a21ec25876c3092 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0143/1150] CI: retrigger From 65cff222ab9087d0c0fd0168b440eee8150ecead Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0144/1150] CI: retrigger From bd74b89c8f4f784797469afd6a755855e9ed2bc9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0145/1150] CI: retrigger From a11be800073097906c8c9729fe8ab65d2c1ce417 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0146/1150] CI: retrigger From 36ebaab88e411918220306eeb69257fa7e64e433 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0147/1150] CI: retrigger From ed8fce289bbe5e2fc3f65a290d15613f6ee63b8b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0148/1150] CI: retrigger From 7304d0765dfcfdee6784804685ab320faf2214c2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0149/1150] CI: retrigger From ae2534c2c37ab0c53c1e76a83a57d054c3279aa9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0150/1150] CI: retrigger From ecd052d71069b05306011d4b35be454974911d3c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0151/1150] CI: retrigger From 1e6d453281a5c99fd67aea12f3c22ab85def18bd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0152/1150] CI: retrigger From 278e5b3f38f17faa511f10d8835deddf4d5b32eb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0153/1150] CI: retrigger From 1dd1f7613d1e0b378a90a35bdf9119382f992993 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0154/1150] CI: retrigger From ec3ceeeba814da6a5a36992a5775d813a926bdf1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0155/1150] CI: retrigger From 134d97cc4c19c73ee56818c4884e8286b895a2f0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0156/1150] CI: retrigger From 71c80576af7ac374ca7598a011c62d708c254d1d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0157/1150] CI: retrigger From 8387e8cfb83b7faea7835c2da6accb5b4207b1cb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0158/1150] CI: retrigger From 1afe9aa7622a50dfe071eeb19d010e4d58c4dc39 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0159/1150] CI: retrigger From 78b59b132a0cbba500663c799217c4f6c79f7c7d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0160/1150] CI: retrigger From f8d2364812c8e82adcf4db3c5023304bf5029bf0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0161/1150] CI: retrigger From 9158e159180abf59dd51e3a5530da03a2e417b1d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0162/1150] CI: retrigger From 83f8d91a46e2484d960289b5210b7ac39b555350 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0163/1150] CI: retrigger From 8a0748e2657d0b57e2ea4f3a4fcbe85293b169b4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0164/1150] CI: retrigger From 7c6c2b0bfe5124f260d8a06aea2f9066e91907cb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0165/1150] CI: retrigger From 5be962ff7d94abc5fddb080c5a4ebf0e76292199 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0166/1150] CI: retrigger From 5cd4c60cd20e47e7c725c38e74030267f0ae92b0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0167/1150] CI: retrigger From 48b380bc23e7d5f65f33c08730afef7cd460a4c4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0168/1150] CI: retrigger From 023ed4619bd4aaaa18f2db6d680b2b46a43f634b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0169/1150] CI: retrigger From 227221693433263928c0302e202716a10c60d7f6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0170/1150] CI: retrigger From 1f0e927d711505a2d697796565208b8ed4e62a29 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0171/1150] CI: retrigger From 9d52c8c0eb1142227137a11a9433bd8e447d1d2f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0172/1150] CI: retrigger From 0238822352970f76044b30ba1a83bd4780804dcf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0173/1150] CI: retrigger From b2cb0628da840b3daa98e074012e545db6d33b99 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0174/1150] CI: retrigger From d53fea9562dfce85683f314dceb6e2be8c36e10c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0175/1150] CI: retrigger From 9b038308ae3f798314322cb56da8a1c7989fd25e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0176/1150] CI: retrigger From f2a163579dc3cb524d8c0c647b0b1c9ba9d2c18f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0177/1150] CI: retrigger From 4b6a9c97da98fab5f82dbf7fd61da49af089bc34 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0178/1150] CI: retrigger From 01e575417a8ee4a40f53ebf3c1af7b6b165168e8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0179/1150] CI: retrigger From 8dd830cafc6a9e6025b57602fd6741a5cf9148e2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0180/1150] CI: retrigger From 2d7ec35e89d3ad97dcfc12b769be69b7ed5c5903 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0181/1150] CI: retrigger From 40a8f61197efe2df6779b2795bd806f804698447 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0182/1150] CI: retrigger From 53a84883406d838019e1a6a2dcfa8d87335c71a9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0183/1150] CI: retrigger From ca3425b8755dde70e419f74e698612041049bc63 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0184/1150] CI: retrigger From 59776759386e4e4c6f556bc779573f19c4feeec6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0185/1150] CI: retrigger From f5030a0e3f6875076e893aebf9e0c5f9665603c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0186/1150] CI: retrigger From d51ad39a471c8be87c18de56503e891bc3d7bd1a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0187/1150] CI: retrigger From 94dc301d0f84442c6d08ff94e96bcf699533b6e6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0188/1150] CI: retrigger From 3f50b69ed1ba4ada115350f35c672ac44e6fda9a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0189/1150] CI: retrigger From 1737fc5d696831b175f45eb48d37882126badd3f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0190/1150] CI: retrigger From dc8195fe0e1da572018a9a84e3db954358b6fbf5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0191/1150] CI: retrigger From 6547af0d683a6c5d7d95580ef8fe3d6bb5d3f777 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0192/1150] CI: retrigger From ecc1312cbcafe78983e0a318df0057ff6a9884fa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0193/1150] CI: retrigger From 10160caf3a40fa18156a679a52d2313783ce97dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0194/1150] CI: retrigger From 352447ed58df7b843d9f4d8b49e44004a1d9355d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0195/1150] CI: retrigger From fda608bc935abdeb332d714eef905e753ae19886 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0196/1150] CI: retrigger From 8f970f3d4e303c68b133de7f3178b8d72ad16ee1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0197/1150] CI: retrigger From a30beae77da5f7f19eb80d590f41262aa1ccfc5e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0198/1150] CI: retrigger From 0b04f5866ee080441f7ea39ee4367090eeaefa87 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0199/1150] CI: retrigger From 858eaa32fa9c019728f3419f800eb96bdf466b34 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0200/1150] CI: retrigger From 79b4ac38098bf3c9e37b57cdfea968a47c677ada Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0201/1150] CI: retrigger From 084efb9fd7aae31cca0c941b37784df73797a3f7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0202/1150] CI: retrigger From 9ecffba366f8f5e0ab81b22b229e101b46694bb0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0203/1150] CI: retrigger From dae77f10c309d805399e1ba9e45b11a00935d532 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0204/1150] CI: retrigger From c130eb0b5ea7f6abf500bd306343e509247857a7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0205/1150] CI: retrigger From 12b0033aa24f51fcbb74eb43fc8d7f98fa125f15 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0206/1150] CI: retrigger From 59d4cadbe4ffc2d4394cb16ebf664206ae80cdac Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0207/1150] CI: retrigger From 7f66d90a1287cc605ea6048ba0045a6f54f830fd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0208/1150] CI: retrigger From e5d540d6e3f0836ba235fa1eece3040043667599 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0209/1150] CI: retrigger From 12203d9874714d6714a38fc69f97adfa70b5ed37 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0210/1150] CI: retrigger From 7916e93595af782e00f45603803c158d8af42ff9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0211/1150] CI: retrigger From 5968faea15ebc1dc8a448b2dbca45797c5a972fa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0212/1150] CI: retrigger From 4dd05b3b241f4cd7c22cb660df79791f902d60e7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0213/1150] CI: retrigger From 345194b121c2147283add186676b241c152c454d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0214/1150] CI: retrigger From 9fbe28a145739cfb834f8e40e146f4189f4a3198 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0215/1150] CI: retrigger From d9c6a08a9424eae20a69391994ae1ba4f5d0a88f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0216/1150] CI: retrigger From 8810ad19c77a48bad576c71cfd84574ebcbddcc8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0217/1150] CI: retrigger From 6ff08d84b71896c785f5b817667ef5849225e0b9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0218/1150] CI: retrigger From 090d2a1ae64407354a2e0d9f057a59fd6fbef02f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0219/1150] CI: retrigger From 2687fde871613cfb151f8bc308ca8cff01396581 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0220/1150] CI: retrigger From 14758411983cbbbda97c4ee927b74f66d5033c33 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0221/1150] CI: retrigger From 6e0ed0077932a06cc35f74a15067c0b7f8760681 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0222/1150] CI: retrigger From d0470d09268fab3252d80ef917f2b4e5ee3f284b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0223/1150] CI: retrigger From ad8e6cccc71e018b0e6e421111b8eef8329493a8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0224/1150] CI: retrigger From 0451e9ce765c04f65aeecf756914308e92d3cc8f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0225/1150] CI: retrigger From b82aed50e321873deae75064219a872a93c31950 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0226/1150] CI: retrigger From 0864adf2b45dfe26bcd76eebbfd8efba945b63d9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0227/1150] CI: retrigger From ae646c86e441631d0ea31b678ac7648460fb8ed5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0228/1150] CI: retrigger From c1d618c3a32b119f8aa5c513f9b1fce5b78fcd99 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0229/1150] CI: retrigger From 4c818aa7ef12fc44735e1dfcecaa1879f8a8ead6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0230/1150] CI: retrigger From f2115c6d8a9f3ab06cbc1d51c4f0eb368496aeb3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0231/1150] CI: retrigger From 1cda1b95699df02a3e0220595302ea5225e71bbe Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0232/1150] CI: retrigger From 41891d70fd17d3772b40ca8111afcf443add1820 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0233/1150] CI: retrigger From 8d0ac37ab63fd5b06f76268454592655a6a2466f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0234/1150] CI: retrigger From 212618d40184673f4431c17f38c3f38884099e12 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0235/1150] CI: retrigger From bc88770f13118359300cd4fcb7d428095c02cf04 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0236/1150] CI: retrigger From be73cea5da6c0b694d760153976507e4f7d5f283 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0237/1150] CI: retrigger From 1eadd7d49185beac2c75a13a712ad49f244dc0aa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0238/1150] CI: retrigger From 6dc9ca4912108dcb7f0b5b2e0ff24ac3ef61bb4d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0239/1150] CI: retrigger From 7a97aee407b6f6cd1d04dee87bc4f709119d11a7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0240/1150] CI: retrigger From acd8b468d3ad3995536d25626c94717548f52b2b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0241/1150] CI: retrigger From d74ad5c00806688ad260d56987478f5e2a0c93b0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0242/1150] CI: retrigger From 6ced5401408e0caa5f0e1e1e290214b7b50e6f95 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0243/1150] CI: retrigger From 66ba8dec22e028d20c40bbc00aa5f5520aaa399d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0244/1150] CI: retrigger From 5bf5c0e5767d56b4162e095643cb1ff5848097b7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0245/1150] CI: retrigger From a42f491d273f9752fd165b759e021f0026e78c88 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0246/1150] CI: retrigger From 6adf3cd08754a694ed9077856d4ef341aac285c4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0247/1150] CI: retrigger From 3fcd97472f58a7932751b0775f4835f96ef96eb2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0248/1150] CI: retrigger From 93fb54fc4f2f418d1205ceb6fc1160532e3a1a89 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0249/1150] CI: retrigger From d43775eaf8fdd4d00a895ed3abdba1d3d4c00755 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0250/1150] CI: retrigger From 36190e6373410a924908dc751d8c53ee70af0fd8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0251/1150] CI: retrigger From 8817c4f05631b3a9a3c0a3e20b354cfff7dc65a2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0252/1150] CI: retrigger From 14003d6187c5e452dc14d74adb99200be855dd81 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0253/1150] CI: retrigger From a0f2d5476c789971d3ceefb6f1d2d7405d1ea4b8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0254/1150] CI: retrigger From fbe742f48734aca465bf0db05fde167d99f588b9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0255/1150] CI: retrigger From 79c9c426047e1f417dc3bfdfe0dac4a5748143ac Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0256/1150] CI: retrigger From d999024046081184e664d21fbcc72b2201412a0e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0257/1150] CI: retrigger From 3ef1f0e3c98431e763f784903546c4764d16f568 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0258/1150] CI: retrigger From f0fe99c29aa4cee242e4ad676d70be4dcede55fa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0259/1150] CI: retrigger From a4bbff06e25d8de61dd42b9cd1100e95890e7276 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0260/1150] CI: retrigger From d29e0d87ce479af15d1526aa7e6a8bdb8bb50a30 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0261/1150] CI: retrigger From 3f086b807031345d0ba4e4d85b65a7184e5505c5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0262/1150] CI: retrigger From 52f3bf96b56fa55bd0d7d4d75fedad905defa737 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0263/1150] CI: retrigger From 7d84190fd207c0fb86dd89e3d736a06196da7197 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0264/1150] CI: retrigger From 4056ac19f8d69645335f36b6689bc35687198c87 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0265/1150] CI: retrigger From 4a989c8347c405475f5f0233a7656fb45b185424 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0266/1150] CI: retrigger From 90d2ef10091b3d8eb52375c7063554e86017fdfa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0267/1150] CI: retrigger From 1f62e8d4392fe526aa1734987974afa625437de5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0268/1150] CI: retrigger From 97e2df624709af10ab344b0a91a4d16cf3ce0087 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0269/1150] CI: retrigger From 63109c3866e5e07bf5c7d31a4ff12d90e5839cda Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0270/1150] CI: retrigger From 3be8feab1b8af3d284938167d3a2b55453ab951e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0271/1150] CI: retrigger From a7c960ebe5c760d9191c87b2501031286f11530f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0272/1150] CI: retrigger From 1a8ccca18bff325287f64508e6b529b61adb1d0c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0273/1150] CI: retrigger From 466e6212bc5ee57328e4a6ba29103c2206ade026 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0274/1150] CI: retrigger From a61e5fe79cca7ddc47db432b2e30121c166c0940 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0275/1150] CI: retrigger From e28f2157f2fb2b5e837457fcdabcc46100494252 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0276/1150] CI: retrigger From 34546a8406ea29bb4f062d9f762484843c1b7293 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0277/1150] CI: retrigger From bc26ca77d6f42dfab7b85a69739a34aaf5c48904 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0278/1150] CI: retrigger From 0dab9a63467c6207ec41c25fdf10b1212fe33a4a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0279/1150] CI: retrigger From 7407d396c27a13e2def74f7fc9d2ea0a90fa7e6e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0280/1150] CI: retrigger From 478ac7afef7dbccac6456357879bbee051c1178d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0281/1150] CI: retrigger From 9914383d7364282750a3e1e7790fa925bf6406df Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0282/1150] CI: retrigger From 87a8e240efdf45be1c281a992e3e2724f42f544e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0283/1150] CI: retrigger From daeb79c63e61a722f2c74f259691ea4e93ee72e2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0284/1150] CI: retrigger From 6b58e1eeb4f5844ce1d1cc36530ba9c325035c1e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0285/1150] CI: retrigger From 702f8434d9427c3e9a625d58d6841dc36ca15c92 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0286/1150] CI: retrigger From a342f5533f5ccf006ac319af8c78566400bc71fd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0287/1150] CI: retrigger From 3bd1bc954a9c80e5dee7e91105f70ae8d11cd31f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0288/1150] CI: retrigger From ba1e515a24d4ce732ec9e5a6f2de0f7944f58f98 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0289/1150] CI: retrigger From 5690504a95d38e6b55d912fdbd798ec9953cd606 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0290/1150] CI: retrigger From 5759d018cc78989bafa1be7af41c85545b660bf2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0291/1150] CI: retrigger From 23eb1f60afd62cc551fc450c27da7e068d6f26ea Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0292/1150] CI: retrigger From bdaf9a6d67c0b85872d59ce2389073701f91c3a0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0293/1150] CI: retrigger From 3530d0f3ac5ed24ed30cf6dbaf2fd484b21c7505 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0294/1150] CI: retrigger From 6aae22cd3cb6858c3be0eeb21ed66676ec88d342 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0295/1150] CI: retrigger From fa5a79575f75faf5c011baccf42eb9644c8d8fbc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0296/1150] CI: retrigger From b3ca0293354e87bb06634ddd50ff632b6817156d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0297/1150] CI: retrigger From 933e3274781ba8d02e4e6ec50b0d1959292494bd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0298/1150] CI: retrigger From 3ff44db7238257f58c4179a21c1cbb2e516faf3b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0299/1150] CI: retrigger From 0f3093357eb16e0901b8841a6e4cd55186c66071 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0300/1150] CI: retrigger From 40351b51f7a48cbf8b24152fb636fcaee41c759e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0301/1150] CI: retrigger From 464e9efb825dd0c5f60ae3647b6b69937128dda8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0302/1150] CI: retrigger From 46293fc92813671c3643d7a343ef907bf868bec4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0303/1150] CI: retrigger From 16d132b63e3ed3460d0f7ccc0ec213e4cda6831a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0304/1150] CI: retrigger From c1d5e62dc9982ff8f63c9c093d8eb327b95d0d24 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0305/1150] CI: retrigger From f55f95949cc424902098aaad8289786928074004 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0306/1150] CI: retrigger From 74ddf12ac2449a5397c9e76df866f57068ca11dd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0307/1150] CI: retrigger From ac870543b580f3670bfa08ab15e813a7503c932a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0308/1150] CI: retrigger From ddce083eeb858faff73f9d92d5520e1d8054e15c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0309/1150] CI: retrigger From 62ede2e5d23dd12cba91a4a0d341d523dee7c8a5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0310/1150] CI: retrigger From 91332d9e97010372e73d561fd80a44c83ee88cbd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0311/1150] CI: retrigger From b2be9353d3c53cc7de5352c523e766f12c1d7636 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0312/1150] CI: retrigger From 9e732dbb77f1acfd1e7448723cff5d0c2f492a9f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0313/1150] CI: retrigger From f9109803161135c9b0ffc8b40f2d772c0c28fc5e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0314/1150] CI: retrigger From 0d200a2ccfec375e54e6043fcc9f0eed163121a6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0315/1150] CI: retrigger From 0981c83f30ee62d3637427407fcbb0438326ab4f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0316/1150] CI: retrigger From e4dbca2ed979c396e96b4c1bb03a86e5d2a735f8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0317/1150] CI: retrigger From 8b97a4b7230945271481a8baf7dafe40a6b87e19 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0318/1150] CI: retrigger From 720847b4aff597b84655ad286f0d0a68d45bcd75 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0319/1150] CI: retrigger From 766725923b4ddf33aadfc82729b9dfb431445ca8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0320/1150] CI: retrigger From 918b3d42322889b86e52a4d43ef8f8d4a470bd77 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0321/1150] CI: retrigger From 7d7804fe5f0e772f4e30de6ec7dbdada0964b39a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0322/1150] CI: retrigger From f4f0705ed9607798a36f209a74127f23128b09a6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0323/1150] CI: retrigger From 3edc4bccc01aca127465d5841174e2fae41d89cf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0324/1150] CI: retrigger From c46e4ff9ee458d8b438ad1f7be978f93fc90d67a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0325/1150] CI: retrigger From 5035638c9c47f66efc88d6920f15251faf69edc4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0326/1150] CI: retrigger From 2ad91d63ee9268a75fc8786b2a389431cde83885 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0327/1150] CI: retrigger From d2a345dae0b05d9dcf657a12a87d0670d336258f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0328/1150] CI: retrigger From dfea24bc1a856b62ee5e6fe98c195e2470478a26 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0329/1150] CI: retrigger From 7a950d1d9e1c69eb0cbe89a9716e2a31832ab9c7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0330/1150] CI: retrigger From fe11896d5e189cade79b58db681a62394418e417 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0331/1150] CI: retrigger From 990681b3a04d831886b149a3adc21a8cb9160957 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0332/1150] CI: retrigger From a5a82633554af3047ff1fefbd01678af617fa2ff Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0333/1150] CI: retrigger From d95acbbf5fad825850bc322f086ac78c38e8b3de Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0334/1150] CI: retrigger From 3906c6072a0ecfef2c299c4cf9d8bffb5d152690 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0335/1150] CI: retrigger From 3b0ce37a9de05010136cd56f0be072054c5822a9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0336/1150] CI: retrigger From 2e2ae53a6aae759c815e3d997f61ac6191f26bb6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0337/1150] CI: retrigger From 7bcaf791b77e7ec1102985b9e2174a83dfedb723 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0338/1150] CI: retrigger From 0d09ee6fdb3903f66c18396f0512ca37b0bc6232 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0339/1150] CI: retrigger From 73ece100d2fd1364a5f2e9b3eeb78ce26e408125 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0340/1150] CI: retrigger From c22653438cef963d9b17b9dd22271a6b6a5f435f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0341/1150] CI: retrigger From 1386a3ba919af5150399ecce24734d4fae540ee1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0342/1150] CI: retrigger From 4ff7704088893c4265420e3b9e61a49f873420f3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0343/1150] CI: retrigger From 9085bb60750d2199da665337485d793fc13a44dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0344/1150] CI: retrigger From 3002a09b0f273cca5153a9889df09e7ca7f5d40a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0345/1150] CI: retrigger From 04c7da5c60f0b05b75e97b53776415270356a810 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0346/1150] CI: retrigger From dca654895eefa80d80cf0b2c89c033d956402cb2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0347/1150] CI: retrigger From 08f3fa6513768a8b742185c4f8cc7936a105cdbf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0348/1150] CI: retrigger From 90ebbc51ffba2b934dc32f719850b4902d548b06 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0349/1150] CI: retrigger From 9a440bbed8df421270647ffeb6d5425c24e876a3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0350/1150] CI: retrigger From 55f0170bfa30d129e23657c7336559161d2cbd6a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0351/1150] CI: retrigger From 1ffadf393aae6e37d79b7559ddd74d24ca8f39b8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0352/1150] CI: retrigger From cf913a1f66247102169e536402a91c6f7a629a64 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0353/1150] CI: retrigger From 378b4283aa17117bb01b3ce2c25d7013fde616e8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0354/1150] CI: retrigger From 847f7103c8ccf8c80cc49dc57f7869029b5f59c5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0355/1150] CI: retrigger From 5dfa673f342326b2e37264f5120c692cfdf1a0de Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0356/1150] CI: retrigger From a5ee80d97282b2f82f86c6827f279ceb2710834e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0357/1150] CI: retrigger From 98023eeeaaff15dd9e4accc68c1ab3399fc54214 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0358/1150] CI: retrigger From 4bacd6b463d1e1ea86868c79bd5f6754fc9d822a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0359/1150] CI: retrigger From 5e86b270802ae4b69abeaf5c02d911f7bf54ecaa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0360/1150] CI: retrigger From 7c2aa665c908286c328cbd14b77df9cc07dba215 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0361/1150] CI: retrigger From 5ed7c88f18f6b6f1a3574d23a65c3cdf9d216e18 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0362/1150] CI: retrigger From aa6c366992bd7519360d1855671e5c457b11447f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0363/1150] CI: retrigger From 4714a99c86a0999b5bab603b0d71f9d2e3b7c49f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0364/1150] CI: retrigger From db70a9c25eb0bc3d184242b0dbee6f20a4b903d5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0365/1150] CI: retrigger From 4395ce2e74d6f8729d841dd8768cd5fb2083f4ba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0366/1150] CI: retrigger From a97a0f3e319a2f66d816dfeff220cebaff44df6c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0367/1150] CI: retrigger From 3e1cf8ce00a3e087e6fd99bf7e518ac28f3954a8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0368/1150] CI: retrigger From 0010a21fb2f3010f6ab18d3b14e59d144ec6cb36 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0369/1150] CI: retrigger From 1fb7266f76e44394074b63968d0167d91fda5c79 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0370/1150] CI: retrigger From dfc8e5a72c86d89796f7eb0248e867e9c8ebd089 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0371/1150] CI: retrigger From 2b41d48e1c8258c08f9ee59e502209e6304d5a83 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0372/1150] CI: retrigger From f59801744b8c66b58be18a7f5cd60fdf89b7c528 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0373/1150] CI: retrigger From 67cba8a06ee9b56c264110757b1a413959191957 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0374/1150] CI: retrigger From 872a0cbddbbbd19862247430933f15976ca180ae Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0375/1150] CI: retrigger From c7471bcf7ec02b615a49a15101caa9ba261097b0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0376/1150] CI: retrigger From ddbf4ccc22b6f546f73aeb428cb95ef095c48eee Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0377/1150] CI: retrigger From 9777df731360fd407a2ab2cf2f316e3a1ac8dbf2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0378/1150] CI: retrigger From 8994fe0382ac100fe4a7e48867d955aac2df1c0f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0379/1150] CI: retrigger From be3dc3db73568393334d38769d445676cbb7e4d8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0380/1150] CI: retrigger From 2bc3803838942c0a077a2d8ea8a971bedd55d737 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0381/1150] CI: retrigger From a083159ba77785a85f5ef57ace26f15417eb7c37 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0382/1150] CI: retrigger From 1ef9f9e0f97ffe729d63b4077c87c4d809667743 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0383/1150] CI: retrigger From bff4b31af84a96d5a94937f1e41e0154647199aa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0384/1150] CI: retrigger From 1c97f7f59cbea9fd145bc6c3121c3a95ddaa0e5b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0385/1150] CI: retrigger From e3bfd3cf7a0b63d8c0131c13947526c40f4b8f5b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0386/1150] CI: retrigger From eaa9e656d30e07530872ba57592ce49923e9a10e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0387/1150] CI: retrigger From 86478c6437b7171c94da03ee1a11c3c203e81c70 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0388/1150] CI: retrigger From 46522c7f3e1a8e04e188fedae3dcd47706ad9808 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0389/1150] CI: retrigger From 4f25ced69d95b8f8731cc442d26525f58796fe11 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0390/1150] CI: retrigger From 9f2b4ad52e4b4db46e422dd7d41c7f1b17c6e0b4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0391/1150] CI: retrigger From 9c6a2327eb1deb7036339acf74eb048ed3a373b9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0392/1150] CI: retrigger From 335557b6caf63af396fbd93f562f3f9fb46ebc1b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0393/1150] CI: retrigger From 310756421fe2a479f21310cb9648009fe47160e7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0394/1150] CI: retrigger From 55eab1df459508fa7238607852bcaa1198fe491b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0395/1150] CI: retrigger From ad8fbaf402faba82e03f31d918d985b3cc13200c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0396/1150] CI: retrigger From 15a404a850ed652b7edf6c383b83c0c5437d2775 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0397/1150] CI: retrigger From f3126a5eea61550b25172f77ebf184d7ed05f830 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0398/1150] CI: retrigger From d44ca6d0e0a7433bbb0f07fbb75d4ce1f9a0cb9a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0399/1150] CI: retrigger From 95502d1d309390fe493a53c8d3935797666dd515 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0400/1150] CI: retrigger From 4fa3aab75ca514c0519f651a224b0b4b6137a130 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0401/1150] CI: retrigger From 57eca9648e5baefe1dd56a6d294b15564bda7517 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0402/1150] CI: retrigger From 3fc8f7caac6ad795f8fd60b062bf18067c0d5705 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0403/1150] CI: retrigger From 2021c20532bcc9f006c03b037ad96195c05b9639 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0404/1150] CI: retrigger From 1ce845279eae9ea5abe196543e29b2236073cbc2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0405/1150] CI: retrigger From 2c8369aa96578d0d68f62925c8cfc72dbca3f24e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0406/1150] CI: retrigger From 45d03288ed2ad444b053754f0e6fb8b45526ce35 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0407/1150] CI: retrigger From af19f6a187a3964f647da69eb5a0d89449500c8a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0408/1150] CI: retrigger From 380c54e12cb7bc99f4d00512491c5616c6b102df Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0409/1150] CI: retrigger From e4aabbc4cb25ce54c63c3673c6c88fe514b68e3a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0410/1150] CI: retrigger From c8e2197ad258c48108f6cd063a2b523fc1fc66ac Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0411/1150] CI: retrigger From bd3a6bc8e7bc37250b0d80969e5fcd025aa397e9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0412/1150] CI: retrigger From f35209f3a5e014f9812eee3c956dcfc2807391b1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0413/1150] CI: retrigger From f2adcdcb39626f4303b2df22b4e6ae5396e642be Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0414/1150] CI: retrigger From 77d20e6660677c692bcb4643aed38c86c1b35a91 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0415/1150] CI: retrigger From 7db4325f6ced49ff80780155c3342a41a5774195 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0416/1150] CI: retrigger From a02e09e2a7ce517573e2585a14efe9647761ae49 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0417/1150] CI: retrigger From b77c93f0ccd61bb887c2e9f533f8190df90e3030 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0418/1150] CI: retrigger From 79f0f15723fc2e4fc06ed9fdf5f967f9f7098bbd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0419/1150] CI: retrigger From 35f33e7a3bd3b565781d320c5f5aa308da7cde5f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0420/1150] CI: retrigger From e4e0a83e1b7900e1a75fe7f38561025a03595739 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0421/1150] CI: retrigger From e252d06191b7e78d5bb5d6de6c17b7f3ee3b59a6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0422/1150] CI: retrigger From 0852fe50bed270a40a789551dc83b4598d1aefe0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0423/1150] CI: retrigger From bd46d6aa234d9e2408c433cdf826c16ac95a9d71 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0424/1150] CI: retrigger From d3c01922aec8805009d790c093d9f0b4d173ab6e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0425/1150] CI: retrigger From 1e919a1318fb87a07ec12b1a6824cd7158170b91 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0426/1150] CI: retrigger From 4cbf48ba6255cb0a3a83ff49ea37ed351895809d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0427/1150] CI: retrigger From 3d1de5f02bd54fc6ebde00ef2567dc8125052828 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0428/1150] CI: retrigger From 019a11d62a8d924e6eeec7093252566b1784051c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0429/1150] CI: retrigger From 731a5d138efe6efc8ffcf2474996be1af4dfd634 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0430/1150] CI: retrigger From 7ccdf7db860092a708ccd911072b1a80827013ea Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0431/1150] CI: retrigger From ff344790f6fa5857e9a2499f3837bd98ad0512c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0432/1150] CI: retrigger From 9208224999e0bcb1bbd6016eab2c4ee462e73dc2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0433/1150] CI: retrigger From 322036437332795538ca4b0ba366ad6be303210d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0434/1150] CI: retrigger From 08ad52e188913dac227da9197105a16ea43721c9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0435/1150] CI: retrigger From 601a77a3e018395e740f3cc932c079d99d30e1ba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0436/1150] CI: retrigger From f2995a4497ecf40387688f244d845771121d7e16 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0437/1150] CI: retrigger From 5ac1cbbd74affe8efae447a494dc734932349107 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0438/1150] CI: retrigger From a1c6af57d9a2485bb29835c9e81b15337b98bb6f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0439/1150] CI: retrigger From a93687e7b497584d112d41781d145daf95f75944 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0440/1150] CI: retrigger From 92b84efd40ecbd223804ac0548e3e040013389f7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0441/1150] CI: retrigger From 96b71ee31765c57f4f9c91955ce4cdf7ae9d0930 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0442/1150] CI: retrigger From 516fdb4ce7e7f762a121a99602619281f897610a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0443/1150] CI: retrigger From 5a4feb84dd58dfc986762b36eda4fb58a5d9f0c0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0444/1150] CI: retrigger From 58084a33807a45f1a4640cb21c4f1ecf60a28403 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0445/1150] CI: retrigger From 88b4e3c5249ce1d6273bf5632a14f23edf0c5474 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0446/1150] CI: retrigger From a1ce3ec789de51158b6cbd73a01ae6882db1a00f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0447/1150] CI: retrigger From f161d740cebad4ab89a1eea5ad7016333ee95842 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0448/1150] CI: retrigger From 69f60e125acc2411901309b1b29bdbbb095faaa0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0449/1150] CI: retrigger From 88078f7922172bd553be49cd43c87ad188721366 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0450/1150] CI: retrigger From 1f2c5070d0e94e4b4c10e7682c841c167a70ca8a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0451/1150] CI: retrigger From 000584301c13059a106f011f6d9a22c1cd9685c9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0452/1150] CI: retrigger From 7f280b968ed456b3ba786456d2ad86b948cba7f2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0453/1150] CI: retrigger From 9b2cbce56b63c436cf457a131516b340b5631344 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0454/1150] CI: retrigger From 82168d9ee39cee371776b55d6c5115bd23358930 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0455/1150] CI: retrigger From febd0194493f531d32ef87a3f5a24f0857d86b52 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0456/1150] CI: retrigger From 8deec7a80936e3855f18cf8c0fe6915bb859ee63 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0457/1150] CI: retrigger From 2c55003e0da6c323fc8a80573c57721522a6f4bf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0458/1150] CI: retrigger From 9f9605abbcac06ad19be741badc0a12c6cba913a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0459/1150] CI: retrigger From 437537501fb95ff8fbe502f84d37936ce2724288 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0460/1150] CI: retrigger From 0268cbf5210904c0b6424a453ee4348000a4a272 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0461/1150] CI: retrigger From 89d5f2ea5f1d1cb83480df299dbcd5d64593b18a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0462/1150] CI: retrigger From 5d97b3a7e21eff18f87d80752e6f4c0976cd80ce Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0463/1150] CI: retrigger From 5460d8c559f2a3c9062e0f86563017b5d9a6831c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0464/1150] CI: retrigger From 5283132af6e323c3131437a76227b8aa2eafaaa2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0465/1150] CI: retrigger From e8dbc360b3fc5310b5b937e34bf41510c5bf18c6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0466/1150] CI: retrigger From 1198c6d0262164785719be5ff789dc7849f71130 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0467/1150] CI: retrigger From f6c5f52db304d9ae411e8ff78c5b57c88d8c61ef Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0468/1150] CI: retrigger From 99e217fcc11091e99fad90d3a13e5e085b54f038 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0469/1150] CI: retrigger From 77dfbd0cc2387e4ef1fae2a8ce48f0175cb6326c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0470/1150] CI: retrigger From c56a6887f1ec37bf1944f233df91ff3615ea35a9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0471/1150] CI: retrigger From cf8c558b1443647e64ba9fcd43d62d13ea1169dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0472/1150] CI: retrigger From 3871ef464c50543fcc337960664fddb655137f84 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0473/1150] CI: retrigger From 3cf9ad36ddaba4065e3fe28fac7646f5a0b4fe13 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0474/1150] CI: retrigger From d64c928063b1ada389acf81bb66fa05358cc8ad4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0475/1150] CI: retrigger From 11d0dd0daf4bb7bf6c6d82736aa23024082c2b46 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0476/1150] CI: retrigger From bf71d97024958249e1eb92b1911480ac0e7a78c2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0477/1150] CI: retrigger From daadd3f30b19620144052bc746483dc8ea420ee7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0478/1150] CI: retrigger From 5e320d40e9e9e9ae2d8fad83b1c644f564cf79cd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0479/1150] CI: retrigger From c7a3ddbd0584666d406ab4ac50282218a482fd8d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0480/1150] CI: retrigger From b2e0ec273c7d240cb8fa3e54ad911b0eac13f9ed Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0481/1150] CI: retrigger From 02871d4d92a2924f0d1211543065e9a45024bd68 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0482/1150] CI: retrigger From d79a7757be5654f7cdad0f8b6e78ffcdb1c07d17 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0483/1150] CI: retrigger From 74d29eeb4decad84f343c00e14ca1c1938f8a3d7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0484/1150] CI: retrigger From 99a998193d72d20e46fed02067bdfab6037e594c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0485/1150] CI: retrigger From 3d7c85aee52394eb8522c2213647560c70960f5b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0486/1150] CI: retrigger From b6a64333ca4a66b031f4c1874d20260e6eab4ea8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0487/1150] CI: retrigger From ad1ff31ef51d8f480a53d65208e59f00c10e3335 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0488/1150] CI: retrigger From 6c4adae14639dc390aa7996a3d449856d2b68180 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0489/1150] CI: retrigger From 4cfda4a56deabcb9807909e3a310c87de4393019 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0490/1150] CI: retrigger From 2478e4fec795ab79456d7af99d2a2aa642711dd7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0491/1150] CI: retrigger From d4f25c2adcaba0e7cff79ad0324c461b37dc99ef Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0492/1150] CI: retrigger From 6e8c7d10a389346f1133f9f1c32ff1b5050d7cee Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0493/1150] CI: retrigger From 2beb754f01a428d06750c6427d8f3fbad130392d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0494/1150] CI: retrigger From 4efdd82bb024611867f65d594662123d7e281d8e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0495/1150] CI: retrigger From 9cc3c1ea1c864714a7f43b016e00f01253eda14f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0496/1150] CI: retrigger From 20923c663be6d32b7d87515b4863753938424d4a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0497/1150] CI: retrigger From ba6417cde0bf0a4d37dc4dd268025e2ec97e1de2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0498/1150] CI: retrigger From 14dd3b394a0408a6dac3d5ad49199dbf620021e3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0499/1150] CI: retrigger From 5eb150430d7e5db4406615df6c9b7a99fbcfbd8c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0500/1150] CI: retrigger From 826de8fb457212e2bbbdb0b840eb68c1dd7f0ffd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0501/1150] CI: retrigger From a044d700759da3cd18fb46badc945fceec77f440 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0502/1150] CI: retrigger From 7ea5e678265b801ec34a5b65302c7ed3c5a0647e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0503/1150] CI: retrigger From b44d829940ab285a11d184c850947ca98485fbb5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0504/1150] CI: retrigger From 94058a4e9f84773eb1d3eb51d3b5e3995b643c64 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0505/1150] CI: retrigger From c2ddd40d2fd343504af64ec27685e766c1a35827 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0506/1150] CI: retrigger From dcd1dc9ac3a28f3edeb40091b8879c1ad3ec4b0a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0507/1150] CI: retrigger From 1909f073170ea045f7266b806075ee562a1e657d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0508/1150] CI: retrigger From b4af382113c482ddf6b1c3370a221d2855b60c90 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0509/1150] CI: retrigger From 2b8a24d7357ea5598fc97aa8d19814108ad7fe94 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0510/1150] CI: retrigger From 1656cd50dfd42e2d5b654ac45a54891645fd5c10 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0511/1150] CI: retrigger From c01a01e31c9707e737aef1ae300555b407381e8e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0512/1150] CI: retrigger From b00cb73901fa6437f3ffc8d177cd530c5803de92 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0513/1150] CI: retrigger From 2ff2bab7ca085c51fb1254bc671d8bff6fa38cb9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0514/1150] CI: retrigger From 1e248ebb29521ff31373f9f6f865093318f90904 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0515/1150] CI: retrigger From 7e1e622f10c0e4ee2aa95d684ee78fb4a59fbaa1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0516/1150] CI: retrigger From 177a855168f896b9b66a092bffff6ae207dc96c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0517/1150] CI: retrigger From ebf22976372357f30a279d2410752586f5e3d727 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0518/1150] CI: retrigger From f2ed73d538e82d6a1c268f4e747860cdd2215d0a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0519/1150] CI: retrigger From c6d9351ba0605116403e7bc4a2d85a90a4fde817 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0520/1150] CI: retrigger From e10457ca3f4cb6fd69284dd45a91dc5938a5ea70 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0521/1150] CI: retrigger From 39099ce4d1b5c3a1ef9d5eae6d75210bf2b46c68 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0522/1150] CI: retrigger From 6dcc16b5252125c897a70bb40e4bca0f0a22e1d4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0523/1150] CI: retrigger From 14d08c7569a3f5fe5b74b69e2d4eee94833b781e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0524/1150] CI: retrigger From c4d61c71307734f9a09a5151bbe381b069ba84db Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0525/1150] CI: retrigger From d010e2a365c670fdb475635da533e4e1edc17661 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0526/1150] CI: retrigger From 6f1ddfc5f5c2817561af13d00e2de55cc6c316dd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0527/1150] CI: retrigger From 98025a754ba1971ae5a246b16ae44417f72f513a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0528/1150] CI: retrigger From 34e28d82ee8998e8df50d909acbec7d0e036d61c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0529/1150] CI: retrigger From e4224a5e599e1bfda77717c039570958946a2a31 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0530/1150] CI: retrigger From 28d2d2a5dfe39fd1e962ee8e27c8d0f4add736db Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0531/1150] CI: retrigger From baf33699509a79a84a9fe788a920a5596134dbb0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0532/1150] CI: retrigger From acd9ba3385f7403e9f3e3d81e254866aa86537c9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0533/1150] CI: retrigger From b4a035a3a536101bee52c2b87a65e2216cfb5f6d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0534/1150] CI: retrigger From ddde883c69ce28cfe64d8da029eb0823b6fd5dcd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0535/1150] CI: retrigger From e9ad1427ab58f2de3971c47ea2f30f85421268ba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0536/1150] CI: retrigger From 84c46dc4365aa6167a1f012014aa5a77bd3a9438 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0537/1150] CI: retrigger From c0553868fbaedae834071dc984766d211bd7ef2a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0538/1150] CI: retrigger From c61e425ed3c439462f2500721c07504133bd7b85 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0539/1150] CI: retrigger From 8cb8d61f372d6b5f29778b2cc16a9966f53b4ea0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0540/1150] CI: retrigger From 9f546b006ab503930a61e840dfb682674db9983f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0541/1150] CI: retrigger From a782b75f82962c6dc85be1d1c5030aada7b901b7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0542/1150] CI: retrigger From af22b3ea4f5558ec6eaac912665b56ef4a5cb6d5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0543/1150] CI: retrigger From 204ff3a364c6355374e3901b364a22da4d6ece1d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0544/1150] CI: retrigger From ba2f916287f8fa37ba88b61fc70a10db34091c0d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0545/1150] CI: retrigger From 46dc7e7f28cc83c04ce8396dd382f9b7e80e9b90 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0546/1150] CI: retrigger From e48b0ef303fe0b40b65fc24aeb1000ae89fd4575 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0547/1150] CI: retrigger From 31143dc3a52e2705609c07f0fd2c5a973f0106d0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0548/1150] CI: retrigger From 2155119a54193b3d995fc9538bf5f0a8e6c5e364 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0549/1150] CI: retrigger From c2eb371749b6d76e7e82c8b88a835da84fa13ff9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0550/1150] CI: retrigger From 311ca407dfe45e9a727bee05e2a709bbc7424450 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0551/1150] CI: retrigger From 450e6a584062bbaef1eb37f90de991b1022244e9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0552/1150] CI: retrigger From 5620c260cc70f49caddbc463572856a69c984aa1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0553/1150] CI: retrigger From 3d65fe19ea3bf249b23ec199b1d01c2517073a6d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0554/1150] CI: retrigger From f6b885f867bef9be5e74ff9f81697c48bbc4ca00 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0555/1150] CI: retrigger From 2428d9ed426223a8f19794e085a914722259fdcb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0556/1150] CI: retrigger From f8831674b9a004b3bb32cc151b2459b23aa401e8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0557/1150] CI: retrigger From be78313b4ede5792659e06697965428b2128511f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0558/1150] CI: retrigger From e5ba867abb8ad382bd91f2f37d2ab7949927fbfd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0559/1150] CI: retrigger From c298216eee828e7981d25951486f1f6799f1e634 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0560/1150] CI: retrigger From 710ce4442880cb3652304065dfa0df529928652f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0561/1150] CI: retrigger From 9e5fc24644a96122c7b813cd1bda0ddcdeab4ca9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0562/1150] CI: retrigger From c272a4cba7f702fafffe26bf605d1cfca3d36461 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0563/1150] CI: retrigger From 6187e454ebfba61462f7c5be4d16b2e11cd131d9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0564/1150] CI: retrigger From 178169ef5aeaf920a81882380111f10f54d1c376 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0565/1150] CI: retrigger From fd2acc812e1ab908c217d4e40a63a6917fd03c32 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0566/1150] CI: retrigger From 9f3d3fbe4af0a313639557715e41696840eeb4a4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0567/1150] CI: retrigger From 90d40df2d48bbcbe81ba06bd1476657472d40e98 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0568/1150] CI: retrigger From 3539087a6eaf1330a324f28827c99c375d611a26 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0569/1150] CI: retrigger From c7d22b5637630cbf7cda120f9bd728f81faa5f5b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0570/1150] CI: retrigger From 41750f56880dfe7a0c729387a9c89fca7fd63442 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0571/1150] CI: retrigger From dd08acdc7b085ab8d3b0a1d17ece48c869fc3ff7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0572/1150] CI: retrigger From c304a67bfa61c09d5bcd59c9f267986c4e2694c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0573/1150] CI: retrigger From e538bbbe36b07adf27544678ade6180a0062caa3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0574/1150] CI: retrigger From 404177109ef274c3bf89e8b7600338877de08642 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0575/1150] CI: retrigger From 96f5e75fbec22d08cc74de2c0c8b1512186b74b4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0576/1150] CI: retrigger From 78971d1a50f3022438e4931e8074421c0e6d8d0c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0577/1150] CI: retrigger From f9f446b905864a6668086767a8eaba8246011a7d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0578/1150] CI: retrigger From 4ee3b7917b36673be170a0b0007757b870307c87 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0579/1150] CI: retrigger From 8e4854384bdbf20bb924fc4015981ed139fb9cfa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0580/1150] CI: retrigger From 8529e2e4ce8c16a83e0562cb2ac8e0ff82f8996e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0581/1150] CI: retrigger From a6ed5b969702c8e38bccb4e73e2ef544aa4f3a54 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0582/1150] CI: retrigger From d8cae98bc48e2b92f8e7fd81c308e6fd9e62dfd7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0583/1150] CI: retrigger From 3ec2bb862313d6f969ac124a66e7edf06ae491d2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0584/1150] CI: retrigger From 0dc1cedf4764ac4f2e906d271d46c75326f26f5d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0585/1150] CI: retrigger From abeeb61d5067e50e95b9440134b037cc5a22aafc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0586/1150] CI: retrigger From d5079c27e4abf32d46d20e2b908ecddb29051920 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0587/1150] CI: retrigger From 75d4deea01522f1898f574fc3ebd0497cd965102 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0588/1150] CI: retrigger From 5a451ce76e4b7bd1bda7ba1f56d7d0a7516d30fe Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0589/1150] CI: retrigger From 920c43859a2c1e7bc3fdddde298875c12494fcca Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0590/1150] CI: retrigger From 2922415086f883f95fdae14db6363a7471650498 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0591/1150] CI: retrigger From de0a2605f5e7990711633d7f612c1ec1a453927b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0592/1150] CI: retrigger From 275481923bfa369af3a5efdd6f2284ddf9260c55 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0593/1150] CI: retrigger From df83ce10bf0b70416bc3f30cb56b05add625c1aa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0594/1150] CI: retrigger From dcc748915334c0d1046ddbed5a445f0a34ac8f40 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0595/1150] CI: retrigger From 8ef3264265d7afed774b8197ff4c703b913c5374 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0596/1150] CI: retrigger From 00b03c63f3313138eec8834ebbf874b65fd34b4c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0597/1150] CI: retrigger From c9d4ccf3f5d68eb2ca804017d8f06bd735ac6227 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0598/1150] CI: retrigger From c5b09691ee10862489771072ae2f56e1cb2c379e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0599/1150] CI: retrigger From becca1be151974a771a1262926329c4de5dec634 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0600/1150] CI: retrigger From 9e8f91c1764823f1abfb79e008ac1688314651b9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0601/1150] CI: retrigger From b204091e0050192317bc5aa5844dc8eebd727a32 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0602/1150] CI: retrigger From 3a09b5936b301cb0705596bacab17c5d2ff0a06a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0603/1150] CI: retrigger From 00d55637069b2d10bbea9090e7f8ac017d6582f3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0604/1150] CI: retrigger From b99f5ca9b351fe65685f177a9337b99a4e060b4c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0605/1150] CI: retrigger From 9d65e085d61146c7770e004149ad239a54ace671 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0606/1150] CI: retrigger From 63caf5ac4c0bc694cfe931f22d85f8b1c6225f35 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0607/1150] CI: retrigger From 45964e0928b13e5840434d108e9a6bd0f7d756dd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0608/1150] CI: retrigger From 433fc878b78435c3eab2171c1c8c58710c73e4d3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0609/1150] CI: retrigger From bec8579d30eb2a078af4bff1802557f230cdd7fe Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0610/1150] CI: retrigger From 1a1952222578dbd4c066b953247a8762c0945bb5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0611/1150] CI: retrigger From f578fe47a5de391be2b6a58466ed000f9c8ddea8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0612/1150] CI: retrigger From a2e505e730cd65b4d50cc3bc7c8a7e8a72b46145 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0613/1150] CI: retrigger From f48abe47455470b00ce99c3f3ad6193331b47a1a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0614/1150] CI: retrigger From 06861cc75874b1a9b709a62a2be7477466853288 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0615/1150] CI: retrigger From 28f9c6ed752249ff2eb520a333c3774af6a9f40e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0616/1150] CI: retrigger From 16df65ac09ff16f392934bf9d64892ec30ff4804 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0617/1150] CI: retrigger From 8547a25124bbe8c2689fc5dc68ffe86a66635bf3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0618/1150] CI: retrigger From 71fb81fc92855a66f23ef3571403f9439dfba54e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0619/1150] CI: retrigger From f5c77bf9c58f9c7d51cafadd76531f064c775ef9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0620/1150] CI: retrigger From f5f6e00391d0c63a94f69ee0c33f6529eb29ff48 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0621/1150] CI: retrigger From 77bcaf97f2caadc5ebb9743a56b684c337d31216 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0622/1150] CI: retrigger From 707aeb501dc94e32ba9816e3023ef543413e5e3b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0623/1150] CI: retrigger From 6da8b9ee6cb14e6335a42fa4be057b1e344f5ae2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0624/1150] CI: retrigger From ffbf8eafbadc5d0c8a51f9130f2ea87471d2ba72 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0625/1150] CI: retrigger From 0f550f84c0a122cc8dbfe8bd0f77e73b39a474d1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0626/1150] CI: retrigger From b008368e875bcf24a828ee12c10f98abcfc8285d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0627/1150] CI: retrigger From 2ecb2a41a8784040cc1950a3def6180fd096301e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0628/1150] CI: retrigger From 7c6cbf2213812357aadf405428a102af173406f9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0629/1150] CI: retrigger From 080c221afde63b32577b973134b4dec5b509890e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0630/1150] CI: retrigger From 63b62ba44049a7f7cec6a1c72c34edd46be1b176 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0631/1150] CI: retrigger From 3df91630f04c992b3e38c496cc2600a44e37c275 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0632/1150] CI: retrigger From 94a199e64188a007a7bd4374942a53d083f35bdc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0633/1150] CI: retrigger From 90e46cbad6e45840a0daa987d72abd3fd7af9f5d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0634/1150] CI: retrigger From 192203c1578eb605491556887da0ae4cded88d29 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0635/1150] CI: retrigger From f6125c752e13fbc66cc624d68f1f3d034f1a4c26 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0636/1150] CI: retrigger From b983a42b9842f1511e1e46da041889b3e3fcb01f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0637/1150] CI: retrigger From fa3233bcff2d91ff8d6f62f27b19405bf963fc94 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0638/1150] CI: retrigger From 23283972523efe2c527a0e769ed7d7d04de8cf55 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0639/1150] CI: retrigger From 05c6cd31b23751705e2e40a20f8587512ac6a319 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0640/1150] CI: retrigger From b73859037e78b02d0789c4a31db98f270360cb68 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0641/1150] CI: retrigger From 02b8d0b76fcc33c1b64e3fe31abe681dc807e2ed Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0642/1150] CI: retrigger From 0f619315545036b0ea8ca8097aa44186fb84a99e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0643/1150] CI: retrigger From 566bebd91752c51ba314666fd4158aa6a5a3010b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0644/1150] CI: retrigger From 51b31874a626d0c6b7ff7895b86d70cabe0535b5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0645/1150] CI: retrigger From 4b34aa266fa879ba1a69b6edfd9f5c06a36013eb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0646/1150] CI: retrigger From 98b3a6f3a0234d9d6326634407619299bd1aeef2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0647/1150] CI: retrigger From e65adb89aec18bc94ac69a3cd82ef4f658f50521 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0648/1150] CI: retrigger From f036e86e433befed4583d693f2d7b85aa55e2199 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0649/1150] CI: retrigger From e5772f38986a3cb73770045b3c23a59dfdd4efb6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0650/1150] CI: retrigger From 2867a7e88ff3636dd262b61b67df4c3ae18cbf30 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0651/1150] CI: retrigger From 624b2773cbb61b3d970f9a242424eb63154f18a7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0652/1150] CI: retrigger From 02293f3b090ae4ac9ab613ffe2f80481c6277c32 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0653/1150] CI: retrigger From a8f5b0a2208754efd6b77d22fa821e68394d7683 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0654/1150] CI: retrigger From 98fb5762fddde90a175f6687f43f0521d7614895 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0655/1150] CI: retrigger From fbfcdb34255f88282089247f3085334059ffaa9d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0656/1150] CI: retrigger From 3fdef8901f23388db7b38f45906220fd44d1a13f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0657/1150] CI: retrigger From 010cdffae0f1f46a24fd85a3208d33ea620fddf2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0658/1150] CI: retrigger From ecceb80e945fd5e5ea12fcd30cf8900692bb892b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0659/1150] CI: retrigger From c67f428fb170dcb6acb382dd2424669ab7c6887a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0660/1150] CI: retrigger From 2f20e1fcc706f8483311ed4cd78e65ec9035e4d8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0661/1150] CI: retrigger From aa14b9e73476b6e198238eb0562b83d335505555 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0662/1150] CI: retrigger From 330f78c8a98e2abe86a42f6157780cc351b01e09 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0663/1150] CI: retrigger From d32cdf78b38c3fab198286db2e73e556d24d982c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0664/1150] CI: retrigger From ec61071a7c9ebc7c3515e5142e67ce1ad1d5b4a3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0665/1150] CI: retrigger From 2540920ba7ac70695024566c8bd7c7343e693b1b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0666/1150] CI: retrigger From 702ab45a2c330cf0e93edcd21d76b06beb65cef4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0667/1150] CI: retrigger From e114539e8a2a9de94c389f2f30f601bd37114650 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0668/1150] CI: retrigger From 66bb496d24c619272b8226fafa696df0fa33ea96 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0669/1150] CI: retrigger From 3c24a4e43780554ad799c4ef121cc13a25f7ddfc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0670/1150] CI: retrigger From e584ded24d1f896519fc9b77db7a9c7fd4362d60 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0671/1150] CI: retrigger From 8520096bdc06c5efc015a7b0ce8aa9d8d58ae8d6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0672/1150] CI: retrigger From de83881db43a7b34f4632234ac99180093ec338f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0673/1150] CI: retrigger From 14eec9b40b593391dfd17c79367f596c0758d083 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0674/1150] CI: retrigger From 634300f43905b1f00670e0dcf3dc3fc1c4139e84 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0675/1150] CI: retrigger From b7eb1ee43f74be5600864f8c1b06826fa1769513 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0676/1150] CI: retrigger From a4d0811de7e2c46d8e9278332c033cee9ebcdb14 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0677/1150] CI: retrigger From e3d7b1279dfcf19b7a8ad2b9fb2551d0957a3261 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0678/1150] CI: retrigger From 51d182cd3b560b026054440d08e18ec5cfb7d282 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0679/1150] CI: retrigger From 93a179727723a034da7ec7ca0f884c1b3953d8e9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0680/1150] CI: retrigger From c09bbb8147c692f6a268b4ff689b566ee7ffed77 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0681/1150] CI: retrigger From 4dace4aa8f451a4dafc15ec48008eca80bc440ca Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0682/1150] CI: retrigger From 192202d2e9b770f613778909e4bc0bcb00d6f3a8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0683/1150] CI: retrigger From d1477ac5804256191ccfa14ff23cc9bcd4af0c5c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0684/1150] CI: retrigger From a7fa82a790f673b1af69c3be1fc3dcb09fb26951 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0685/1150] CI: retrigger From 48f082861f0afb3baa577d4c0969d17afad09719 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0686/1150] CI: retrigger From 33c9dbf148aec11bddcc9d0d1ac7ccf18ef4c9c3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0687/1150] CI: retrigger From 15007da4ba16e9fda86bbea433e9af0763afec45 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0688/1150] CI: retrigger From a7af3747fdc5f05114a76ecbe4404129289a1b07 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0689/1150] CI: retrigger From c985c8a96e1fa14733c260995a258908bf6afaad Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0690/1150] CI: retrigger From 4414c8006f47665f35d3963c2bf00510e302ec83 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0691/1150] CI: retrigger From 20dde265def459a9c04cce8a53295f4a1b0f2b79 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0692/1150] CI: retrigger From c4e1c31c7da15e53d3d702ff2439f574017fc1b4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0693/1150] CI: retrigger From 4c8abd580d938cfb5e60f0140db30980cfdc5a2e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0694/1150] CI: retrigger From bf57b7ad70e392b05ce17c3853a79313ac7fc8e7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0695/1150] CI: retrigger From 6127e0803e22e5d11a28057fb64423475ff59832 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0696/1150] CI: retrigger From 3a04d7a1fbcb82cc1f571770d8360baf7f5bb8bd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0697/1150] CI: retrigger From cab92599436de168ef2033738555afc502a0435e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0698/1150] CI: retrigger From 742eddc1de199f0a79fc8d0df80ac1fe6c3a8cf8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0699/1150] CI: retrigger From cf712a5b67b601e54a8c3f244f045bcebdc71660 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0700/1150] CI: retrigger From 91fb92e3e7c5a67c001e58c3c482982833be7dca Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0701/1150] CI: retrigger From a7a67dcee7d3e9725bc03a372a92d51c4c284f8d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0702/1150] CI: retrigger From dab0fcadf3127816a94be134a6a07708e424db99 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0703/1150] CI: retrigger From 4cadd5b09fc354b08cd2b0d567a068706706c85c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0704/1150] CI: retrigger From b193ac38c19b835ad69489546bdb767c043449a6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0705/1150] CI: retrigger From 95d92e99fb42f781bba651c0882399cdb98e2e14 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0706/1150] CI: retrigger From 0e73fe767a50b7b65bfc60be0f69b2b19ab08f3e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0707/1150] CI: retrigger From ae91085755f241259362fbe699a0f0da99e31612 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0708/1150] CI: retrigger From b1585167923d275923d3470c2511c3aed74551c7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0709/1150] CI: retrigger From 3b0c4ee54101cbd3e09891f82503d4ea6f40cea3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0710/1150] CI: retrigger From f152dd4f0b3172c46c4fdca18a4d5d863fdf03b5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0711/1150] CI: retrigger From 9165866b867b0ee8eedb47e9afe5802ae52f8d72 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0712/1150] CI: retrigger From d855bda7183a8175841aedeb06bd6bb029769287 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0713/1150] CI: retrigger From e0034714e3b715871f6d124790ae8a4fa817abda Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0714/1150] CI: retrigger From 4b8f5238ba300f48b98f156d36f534f76c9259eb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0715/1150] CI: retrigger From b556d0cc75902aea1a651f255f91202319906a5c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0716/1150] CI: retrigger From bd69e1beb6ee6b52eead7bcee2980f84e407b737 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0717/1150] CI: retrigger From fedbf4a44c2746cb33bdccd5c9182f98debb2973 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0718/1150] CI: retrigger From 31d83271e0d6de0d32d665500ff857c3a108eaa0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0719/1150] CI: retrigger From b6d8f2680286136c08daf4718d7bee0fc640e738 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0720/1150] CI: retrigger From c8a2a361272e7a712ffca7df55df33022b68a688 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0721/1150] CI: retrigger From 730295e9f7a658f844fc8d2265f777f9f75ffb32 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0722/1150] CI: retrigger From 0c1ef26b7a5970562732f328f20fe6458a2fec42 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0723/1150] CI: retrigger From 6a8659c29996582cbbcabfde562957d44bf2826a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0724/1150] CI: retrigger From cd51a7979d6b7714a7daef7dc1915ee39d2cfb1a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0725/1150] CI: retrigger From 8ea59b093cda4359bb22e241fe1fea4cf81e0675 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0726/1150] CI: retrigger From 8743350eaf650722360e353f326e737d20fe9e64 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0727/1150] CI: retrigger From 3e06cac9368ecd66bd5a12dd2e6b788ca9a61dd6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0728/1150] CI: retrigger From 09a8462a889af3de7611afdfce4c3cf3e2a71c58 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0729/1150] CI: retrigger From e44c2dbf1ad77940044af5bbdbd66d5d7612bec5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0730/1150] CI: retrigger From 9daf9fbdb6624b8772007368b537d6490be1787f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0731/1150] CI: retrigger From f1512c08411e057e3da733b2dbc8f803575d686e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0732/1150] CI: retrigger From 36a9dd7320247808b10ca3d018d81a470610b9a8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0733/1150] CI: retrigger From fdb7082adf494d4a582b4fac98d27f565f9c19ad Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0734/1150] CI: retrigger From 6e94dcacf111abf80ecf4a88a964f27056a00613 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0735/1150] CI: retrigger From 26a3e938c6e44968a4282be673387c34f3f92b1e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0736/1150] CI: retrigger From a69061e9696b7007c7fa0b0b495147aabaede5fd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0737/1150] CI: retrigger From 5be9e00a77a6c9e223b49e2ec7565485f7cce997 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0738/1150] CI: retrigger From 2dafd8acbf0bd02ff5c1f5da21efa2317ecd36ab Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0739/1150] CI: retrigger From 8dca56d3b8314526acd5fb4df95f777983beb49b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0740/1150] CI: retrigger From 69c4d2570ab3ef539c7a3ba709f3b5cd5f6b851c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0741/1150] CI: retrigger From cb5b84f0e42c9cfaee5ef6c1f8a343c23a60ba47 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0742/1150] CI: retrigger From 73e47364833f96ccefeac9aee1a5247a3adce34f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0743/1150] CI: retrigger From d2450626209fcd8a010b2244973bb3589e3bcac7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0744/1150] CI: retrigger From a807d76c00b1bf3800f728f102f2a66a32160030 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0745/1150] CI: retrigger From 9a6a085c010ae66a1176ead3de3bc37a03e335ea Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0746/1150] CI: retrigger From 3f5a079a082302bfa2d484170010440362ee4ba2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0747/1150] CI: retrigger From 37d23b828e4605acc39c5896098369d036672341 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0748/1150] CI: retrigger From c9cc41f4188950aa6f631f33e1178014bc461bb5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0749/1150] CI: retrigger From 1ab3563ecf4c7201af29b8ad1c6c9ea79e7915ca Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0750/1150] CI: retrigger From d1da1b0f50e9a6cc17aeb1de8968aa1db973655d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0751/1150] CI: retrigger From ed10b7908d36acf7fe9ac26a94f4fc0f94cf100a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0752/1150] CI: retrigger From c4c152843df91ff1e1774f786b51ca516672c58b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0753/1150] CI: retrigger From 70f6d1b5ef5196e2b6eda856dac297968c062139 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0754/1150] CI: retrigger From 23cb8b473d1e01fee80e06dec54f0ffd2f5864d7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0755/1150] CI: retrigger From e82d5b58b9db67e64fa43c7f973f2c86aa568cc0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0756/1150] CI: retrigger From 167b41e8de130211e2b1968cf9a5f7eca1fc6cf0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0757/1150] CI: retrigger From ec59b7cf5a7d61f38cc5fe49b77abb81943b0931 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0758/1150] CI: retrigger From 3833b1e2e751f325fc8aa3a2e432743abaef03f8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0759/1150] CI: retrigger From ae4680edecd27884bb98ff3d0c1291ddfc9747d1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0760/1150] CI: retrigger From 349a7e955b4a27c65c7c23f46dd5212e4d5328d1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0761/1150] CI: retrigger From 729f3b9b7a809220a970a8701d892781e4b16100 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0762/1150] CI: retrigger From f802758c72cdd762b92d3bcb36d604860960f6dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0763/1150] CI: retrigger From 0e4c475f313f3e2eb1439ca042689f7380509303 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0764/1150] CI: retrigger From 3aceca3023d88def99e7931ea26d46e100f138d9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0765/1150] CI: retrigger From 6650c1f99ce9fc6f99d9139f3fd2c1239ffe4462 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0766/1150] CI: retrigger From e06550d269ca6d017aa92543abf4aa994ec1dfdd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0767/1150] CI: retrigger From fb66f57e0d9691a9ad08477a4eed0a30a80a20d9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0768/1150] CI: retrigger From f83e4a1aca7eed4f7059cfe3d8afb9a00dcc87d5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0769/1150] CI: retrigger From b2966cce20ea8c1134bf8c4dd1fee32ae71d788c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0770/1150] CI: retrigger From be1b34633bc6873a774842919164cf3a5ca7d041 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0771/1150] CI: retrigger From 92f4ef9c1b888ae6bb7237b6c31587e82856fc40 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0772/1150] CI: retrigger From e92e821c437222f7b117e78843200a9e4c78ecbb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0773/1150] CI: retrigger From c7735da0cfa599e3a856ac2b5f6ad25933d72f44 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0774/1150] CI: retrigger From bb015b0b0a66801124f0d8b6c3c7ade05434b6f8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0775/1150] CI: retrigger From 579128f74fd0325695ce1246a2f8344db7fb0d71 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0776/1150] CI: retrigger From 4c1fffeab438dbd74734ebe9479bca1c31f3b2a4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0777/1150] CI: retrigger From 86684bfaaf1067453e48e71aa1d5fb6cb42ad290 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0778/1150] CI: retrigger From d9deb592203eae96cecda58d031ee66aaf1b30ab Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0779/1150] CI: retrigger From a1a4a4888c3c7dcf5fa58947b9c3cea665499313 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0780/1150] CI: retrigger From 63826abf8da55d0a3a73d6765d4456e69791791d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0781/1150] CI: retrigger From 9a44c2be5d2f0783b7b50f9919e109ff5826a910 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0782/1150] CI: retrigger From dbfed401489081837705bc165cc000c499c49e31 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0783/1150] CI: retrigger From 5a43cbe327f1fe84284d4c4b6947cd44782d3fa8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0784/1150] CI: retrigger From 7d35c4297a924b24eb4315ff213d66e537ed10ff Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0785/1150] CI: retrigger From a8a958e039118c2e0049fd69e3edd3644473a14e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0786/1150] CI: retrigger From b5c4ced5c6234a9d3148e3ef2f39fd40ce477064 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0787/1150] CI: retrigger From e2f88778aee097eda808dd9f6c29a37cece6ce04 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0788/1150] CI: retrigger From 9cc3275e1915bef5995de6eb3a974c14d011a8fb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0789/1150] CI: retrigger From ec83daea1e6d10210beaa3bc9cdb0f792162b2dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0790/1150] CI: retrigger From 452c78d77e9d22b6078608b87b7f6e0410f32d12 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0791/1150] CI: retrigger From 72bf38abd3b259f1df2357601b801b4ff1be41e0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0792/1150] CI: retrigger From 5ce4a341ad577041f605e0d5440ba05ecb3aa981 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0793/1150] CI: retrigger From a30b44db9cd03e0cdd045ee1451e1e6cf5bf9592 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0794/1150] CI: retrigger From 58542c6c26bb679b5ebe4aaa2f559d8ba85cfd58 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0795/1150] CI: retrigger From dc4dff6d043a0de77eae2698ec941c9fbd969f2f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0796/1150] CI: retrigger From b56b4f34ba36ce2dce16efbc95236a6ab8e7331d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0797/1150] CI: retrigger From 96229859da12286e584ac75e5b796617a9c8e9f4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0798/1150] CI: retrigger From 95a2efdced6513c2eaa2888afac5090a0ba03425 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0799/1150] CI: retrigger From 70a5df9f29322500dc48229c200152cb7b029dd3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0800/1150] CI: retrigger From 2f37e99907ed515524648eb38c26f8b2d17bd9e1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0801/1150] CI: retrigger From 7763193098c2299ddfb80d115251b15621b9e78e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0802/1150] CI: retrigger From ea18b759d64288a05261188f731f98c286b960a0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0803/1150] CI: retrigger From 3e22a7bd88795a905a84940e2499d7bc41c9e2d2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0804/1150] CI: retrigger From 5276b6bbd093ebe412d79b24971be8a39bfbafd0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0805/1150] CI: retrigger From 1219a71b89c910912187232bbdcad901fec715d7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0806/1150] CI: retrigger From 7040888d060970801cbb15d9b12255bf8107d8e8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0807/1150] CI: retrigger From 9c2bfc4ad8af43b13594df31dc14f5a01e513da6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0808/1150] CI: retrigger From d1f46a7cdd902f01d622acfe8c8bb73357b63a67 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0809/1150] CI: retrigger From f7a74f95356f821ec41ff8d87bee301fa5914f6a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0810/1150] CI: retrigger From 6a40b0a9cce2a5ffc08ec0ab2433c2f0454fad09 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0811/1150] CI: retrigger From defccaf174b5b26ca398738e33bd20fd91904f8a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0812/1150] CI: retrigger From 67c0c76be418205cbe373f2cfb71d1600cbf359f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0813/1150] CI: retrigger From bf5de42d35d725cbe9a03a3daa7cd565af682926 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0814/1150] CI: retrigger From 2a96889c7f737a31c85a65391b3fe8fa35e3ca3c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0815/1150] CI: retrigger From 4ed1106bae956ba9a8a89f8fb6d7bf2ebaa3bfba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0816/1150] CI: retrigger From 9a4cd79327b1fb5a7242461fd7c1154ec3b18309 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0817/1150] CI: retrigger From fc4e3931273089697540921177ba95b315aa6c07 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0818/1150] CI: retrigger From c7a4c85bc143639270eb3cf7be10dafed3b6769f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0819/1150] CI: retrigger From 79fe59aa8dfa45d977d29c1dd99a207402a8e235 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0820/1150] CI: retrigger From be1df9320245f3481e8fee40b228fceb0bed536b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0821/1150] CI: retrigger From 81e70ed8cc5eb0556d86f1c27fd7207df1b5f292 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0822/1150] CI: retrigger From cba2c13267cf0cebc371ac68f9c1bd984f79493a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0823/1150] CI: retrigger From 66cc4e7bba981da476ad40ae63e76296faf216c2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0824/1150] CI: retrigger From 87ae6c5e52a9b89aacabc8a0c57f437caad867ac Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0825/1150] CI: retrigger From 13e4eb2767cca5a656b13780fa4715c9a78ca72d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0826/1150] CI: retrigger From 10a48d7beee57206693dcab2c532e53524b971a8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0827/1150] CI: retrigger From 712bb936644b62321f2f239b99a30e04efa7701e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0828/1150] CI: retrigger From 872a27338314758303e74c5f2a63d6c072f5ff1e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0829/1150] CI: retrigger From 50f58691c44bd700eddd63e347806ce64055e7c5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0830/1150] CI: retrigger From 1317078de382be88034584c5cbba488753a9df1a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0831/1150] CI: retrigger From b558cf31aaf38893fc26c67e265e59e6861ac562 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0832/1150] CI: retrigger From 17d774022441f2f4e7614e25dd15721e8ed95890 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0833/1150] CI: retrigger From 0ba208c71d54b71fbd48e238eba4d66f709c7dbb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0834/1150] CI: retrigger From f1b1abf6f78b1678f27cf5eeb8b33b7c181be436 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0835/1150] CI: retrigger From ea34aaedd081ca335355ee8c7a64d2c8ac3ed0b9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0836/1150] CI: retrigger From dff8f85790705a3c31cc430044aeca02ae53cc68 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0837/1150] CI: retrigger From b41e169adcafaf8efeba339ee47ef9e1abfa0498 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0838/1150] CI: retrigger From 4ba2478f23c719a55b448420397ba32f680144e2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0839/1150] CI: retrigger From 2eae6affdab1fddd8353e7b27e06376a0da2b8aa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0840/1150] CI: retrigger From 48525d98ad3a55f9a31d93adfac0badd0965a136 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0841/1150] CI: retrigger From e3b32641e1d38207f07590dd6fef093dd0622501 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0842/1150] CI: retrigger From a6699370ac94889ec395f1a0a6d59b2a53fb1008 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0843/1150] CI: retrigger From 8f08ceaa8826bef4b4ea0b73997f1a85ebd8da21 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0844/1150] CI: retrigger From 18e369fc1da2e50b4f6acfce861312aa4555e369 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0845/1150] CI: retrigger From 8e618fa1946b822d610ff84c9327493f6ec7bcd2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0846/1150] CI: retrigger From 5c14a1c7e238a8065d46049e1e10fcb1feca05cf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0847/1150] CI: retrigger From 0862224c1669dbad696cacf3bc3b57a023eb4547 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0848/1150] CI: retrigger From 2beb00225300cba457a0685985dd24004d0ec1a7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0849/1150] CI: retrigger From 54b583472f0b475c369988b087e46634ce4863e6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0850/1150] CI: retrigger From ef32b495cc0f2115e4c9d98c6c6ec482b4770ec1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0851/1150] CI: retrigger From 1a6772e1f82725824e8c1de79d78ed561a489f6e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0852/1150] CI: retrigger From 52558069fd00fba80574f1b77a34af516598a739 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0853/1150] CI: retrigger From f5fd5aae93b7ded685d76885f128fc78656116d1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0854/1150] CI: retrigger From ce3d5c05445dcd78f432e87653d10a3679bfa604 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0855/1150] CI: retrigger From db475d535ea38b0ba36997bdb8bef59c1ed551d3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0856/1150] CI: retrigger From 2fac47beff8f07e9ea22c2175ccdfd49dcff6121 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0857/1150] CI: retrigger From 271c92ba32efa2c5cb60ebe9fabe0e80efb20731 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0858/1150] CI: retrigger From 0df6b7aac67e3faef9fbdd60d72752eee0a4f2c7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0859/1150] CI: retrigger From ddbd6e02fc7165be139dc551fe8a66048cd14ce8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0860/1150] CI: retrigger From 604f4501649152f130b623598e091c198ad124ff Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0861/1150] CI: retrigger From 0ac51bf65ce14889937d99b620a8d1ae6d7cd4e9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0862/1150] CI: retrigger From d11bb9cbb9ced8a87e964aeda76b35d7acd451a5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0863/1150] CI: retrigger From 34a81f5ee8bd34ca9d21e284dafa81e50e8be48d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0864/1150] CI: retrigger From 49dee9ba1ef03a59f276e2464879729833257817 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0865/1150] CI: retrigger From 4bb9cc1f3877ee70c0842f2ab369e5780814ff43 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0866/1150] CI: retrigger From 8ddb7ec3e0f2e346825cc74b9f819bf9f75c18ab Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0867/1150] CI: retrigger From f90abab2c6e10cb61d3f95a2a49bc7468203a202 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0868/1150] CI: retrigger From 2b54a2b15bfd36b2b5ae4f8076b67d1f6581cfe8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0869/1150] CI: retrigger From 64f7ce17e1de37d44b279745c012dd89277c27ff Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0870/1150] CI: retrigger From e1357b0a22ba90e8875bbfdf16dbe766117a9473 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0871/1150] CI: retrigger From a9ec268ae5b90e0537573f35b07a256c013d8d2e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0872/1150] CI: retrigger From 43913911e441529047324989a42877f9c44915e1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0873/1150] CI: retrigger From 5cda5dd4f8264289ee39c1e7b13137f5c9040d5f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0874/1150] CI: retrigger From f4a80faef06d926be8c4fe8ba90c57ad3cc01eaf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0875/1150] CI: retrigger From cf3d2963c7df8740e6e92f36a0d708dd405a3ec9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0876/1150] CI: retrigger From d42da9cd54c0a6b43899245533645978c8a59fe2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0877/1150] CI: retrigger From 1ed9a41967f61eb5e35ded60e5dd8910c7624ac9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0878/1150] CI: retrigger From b49948986032d366a3f790679e7dae0d00163f6c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0879/1150] CI: retrigger From cba7edd6ef9dd6e1588343ae079395ca6b3580bf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0880/1150] CI: retrigger From f4f2c2850c52275889f1c48b4527b0325ced11d0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0881/1150] CI: retrigger From 4a9dcaf087bfe08d5dc39bf70743679b36f84e87 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0882/1150] CI: retrigger From 986a4382da67901b3168be6f0a2ffa487cf0b58e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0883/1150] CI: retrigger From e280e97636a2757d5b1ec94056903669a34649cb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0884/1150] CI: retrigger From 9152c2326141e0837279b660afe5aecf82f50c07 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0885/1150] CI: retrigger From b42e03e1c9694bdcd5e77c4b22ce85dea44ec354 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0886/1150] CI: retrigger From 48657d6f6df70e0cded8d716933ab79c33d621f0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0887/1150] CI: retrigger From fc7ae96d3f20280ca82391736039b6caa5041f37 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0888/1150] CI: retrigger From 2e441f6e4ff7fb57a87430880bb444946079c66d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0889/1150] CI: retrigger From 35ac471d0634a6d83fedd479867405879d6f24a2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0890/1150] CI: retrigger From d1cb669f1934405d675e2056d1a2f1ea872f917f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0891/1150] CI: retrigger From 4767fb699cc599334ac599d48c25fc6e9f27ac9a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0892/1150] CI: retrigger From 4ccd1d1c745eede35bd47420a19b7b93907a6f15 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0893/1150] CI: retrigger From 2c37e4a114b9ffa8e9068dc3357e9963b358744f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0894/1150] CI: retrigger From 85ba24ef7485b2d9f35b6f072bebc9171f4279dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0895/1150] CI: retrigger From b12042fb7a619373dffeb49ee6be6082014b7565 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0896/1150] CI: retrigger From cdf2f369860d16e7be66897dfc74fffb844a4339 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0897/1150] CI: retrigger From 284bec91ff00623b245dd65d2678ee1526f6df10 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0898/1150] CI: retrigger From ae7315747450562961b2db76490a91576937626f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0899/1150] CI: retrigger From 3b2a56ee4897419b1b15edf0a2ee491c21bbe738 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0900/1150] CI: retrigger From a218a4c8a0ed48827d49342174e49eed89728365 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0901/1150] CI: retrigger From 85089819ba0b80654b03b30e12b8afd8ba379105 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0902/1150] CI: retrigger From 8c3650b93b0401a1deb90c815017bb46ebf0897d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0903/1150] CI: retrigger From b84518b751c557432ef6d1e3c5841641f9f535f8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0904/1150] CI: retrigger From baae09987190587c08ae07f68e72d74801fca4c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0905/1150] CI: retrigger From 9319e4864e6a6d26c3aa1f469b09f20464a9404c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0906/1150] CI: retrigger From 4dc328194bbf1ad8fc31be643afc6669ba7c7d7a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0907/1150] CI: retrigger From 29ef9ddf0485ea98d018f6d803c7a586dcd054b0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0908/1150] CI: retrigger From 3ed99cc3f6c720cd645001d04bd839d05293e120 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0909/1150] CI: retrigger From 28734498561c114b07f4c6b6badada9cf82b35a6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0910/1150] CI: retrigger From a920d349d9d8613586a30e1c568339c760cb8893 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0911/1150] CI: retrigger From ecdee2d4531b70dfb5cf3d2a688db330be1f7d6a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0912/1150] CI: retrigger From 91d20add692fb0e66a0b6b495fedb95f6467218a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0913/1150] CI: retrigger From 71b418a97e1edcf98a2dea8f5d0f7d3688b56841 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0914/1150] CI: retrigger From efd905fa2b5b785e5841e2aba22e9524e8083815 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0915/1150] CI: retrigger From 637238898726095e592368ee9e6760deb5373312 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0916/1150] CI: retrigger From 5b22456e37f2ec186e80ed0d80fb0db5e482ca0e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0917/1150] CI: retrigger From 76d833ef6cc80960ec5586a9f10f7fa581da57f8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0918/1150] CI: retrigger From c0ca729b5fcdb2120aa6b5d4b41643b256584b57 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0919/1150] CI: retrigger From 8b8fb74e1f7da9d6a5ff235b053e6b767b10394e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0920/1150] CI: retrigger From 973db7e4d251a854bff5c250d180b92d56e67537 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0921/1150] CI: retrigger From f928cf68ea19d335ff469398278bd69ee2128a84 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0922/1150] CI: retrigger From 072a20db6764e6441a74872a8bf32f1a3caea07a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0923/1150] CI: retrigger From 0859558d7d339a57adb8ae7cbf509d6daf220d2e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0924/1150] CI: retrigger From c63e97e5bf5c43f2b3bc57f4762cd8b48b259c17 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0925/1150] CI: retrigger From 67ce8e09db53983363454f539bc1c3355d9a264f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0926/1150] CI: retrigger From e6714498c3a6b6a83c281f8a6776143d99cf632a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0927/1150] CI: retrigger From f451559c826817da0e2fd684ecf887aa7d82f491 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0928/1150] CI: retrigger From b574f1040ab020d5dc4720b1b9268165e716cf16 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0929/1150] CI: retrigger From 3348d6c46b8179adf54421c9b1bbee324ec38d1b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0930/1150] CI: retrigger From b9ec079a44dbfc89f11a01aae638f7b720b17fd6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0931/1150] CI: retrigger From a5152f02eeebe779e9673d4386293f30bc0c6bce Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0932/1150] CI: retrigger From b0be4045d57cf6e48521f55de321d2088e04839b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0933/1150] CI: retrigger From 8b3480150fb940ba8d4f8541fbb626c06779aee2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0934/1150] CI: retrigger From a8e63fec59ccac3925b41420688e94a9c370d26d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0935/1150] CI: retrigger From 977666431fba51b44692491608168b524356b483 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0936/1150] CI: retrigger From acd4ccf11e8e9068c56cbced695b4a7ea4c4c89e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0937/1150] CI: retrigger From 1ac086cd0f959aed1e424ed4b1e69170089ac384 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0938/1150] CI: retrigger From ec8860851445f782731d32d9648611e4cbdd0be4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0939/1150] CI: retrigger From bc5dab5bcb1b4a3dd057a10dd3fd0426ace6ad80 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0940/1150] CI: retrigger From 6acdd785bfc95d21d6c11e21950e1fd9fdb3d4b3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0941/1150] CI: retrigger From e01b127e7bc2e3a66e1f57fca27f5cf1a4376d58 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0942/1150] CI: retrigger From 85f942aeefba1fe77e0fa9e9fdc5c090f8798935 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0943/1150] CI: retrigger From 860da0a77cc2072104fb899bab2b032fb2675e18 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0944/1150] CI: retrigger From d17ff0853a3429fa166ac211cc2136befc3b15fe Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0945/1150] CI: retrigger From 59f979acd62b3ec3320e88eac82d807f1226cfcd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0946/1150] CI: retrigger From dc4e254db54e5a7dd200c8e7df9551c15ad1249f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0947/1150] CI: retrigger From e24db19436a1403122d81a0e997d758de6217d49 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0948/1150] CI: retrigger From 75a1669af4c738b02adb2ca2534e2ca5d16ed62a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0949/1150] CI: retrigger From 9919c439a2a9a8af0aabbfc9c008c0e36b6ab93a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0950/1150] CI: retrigger From 8a857fc68071706d322a83a30eb0e23e27b72df4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0951/1150] CI: retrigger From 2f868fd17190b225c32495fc1c91af6954f12b97 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0952/1150] CI: retrigger From 266f0216309e8ed4dcddbbb1e55c15109bba0af7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0953/1150] CI: retrigger From 05c84de59968685ed64bee3bbd55006dc1463d4f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0954/1150] CI: retrigger From 1430875c10db54a6d651c9eb47d0a0f857790e8f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0955/1150] CI: retrigger From 9fa1a9270e8edf850e192f4ad3136a5a5e156f22 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0956/1150] CI: retrigger From 6af6d61672e801a954b4de385f3033a054dfc702 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0957/1150] CI: retrigger From 298d6955144eddc95888eaf601224f60c3bb0c79 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0958/1150] CI: retrigger From 3b6f83eaf6f859537372e6132c924df62b0f4967 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0959/1150] CI: retrigger From 1b2192c931352d677a302f1e97332126ff1d49a4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0960/1150] CI: retrigger From 729bca6c129748be6dc624a093b33b25a79b8054 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0961/1150] CI: retrigger From 6e0839673f72ec23513e2aa23163aec64ab78ceb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0962/1150] CI: retrigger From 8b12b9d2dc93d5721fbe701b4feee51d17d0433d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0963/1150] CI: retrigger From d156ac20fd4bd4c2022575624ae8803f1ce25e3f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0964/1150] CI: retrigger From 59e2b84e2a763fc873738cc84987ffc9d1bccf80 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0965/1150] CI: retrigger From 3e2081a1b01f097a5bfcb8f493657c01d0f9d445 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0966/1150] CI: retrigger From 7ba2c9e9d99c51580f7fbd3131cebada81ea0180 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0967/1150] CI: retrigger From 91aca7f7b418680ac761d04244116f5ba88af67a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0968/1150] CI: retrigger From 5ae99c5bb19e18a29e14770cc3a3d35e565b7000 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0969/1150] CI: retrigger From 2cd75cd4ba84c7a4628af2ade5b48732046e32cd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0970/1150] CI: retrigger From 45eb19bd9baa2e33fb1363069ebc6d87c04736dc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0971/1150] CI: retrigger From 0fd77fe203f6ba70807bcf7118ab89d0ff8514f6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0972/1150] CI: retrigger From ac73d927f3443d39c49b8eb12514284bf0d57f28 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0973/1150] CI: retrigger From 173595663f21201a57a69582fe142aafa3dedba7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0974/1150] CI: retrigger From d8c3f8d32e792b79ed4113e100505105505e9e3e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0975/1150] CI: retrigger From 6cbcf70a127927afea36802ba81e400ef8bc0cb1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0976/1150] CI: retrigger From 5a6465892dbdb482eee532c8bfd618e934bc0429 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0977/1150] CI: retrigger From d62ae1cea7e6d78c76b174d26cba62941e5740d5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0978/1150] CI: retrigger From fb12eba8033f51872116a33b22a8d9aacdad3217 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0979/1150] CI: retrigger From 459c6b2594cb690551e9e6e133aae45125936a61 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0980/1150] CI: retrigger From 874b60a47399a2d415446b19ae4a83e0b901b90d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 0981/1150] CI: retrigger From 414d10e76dfc21255b82efb87623d05ac5780ef4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0982/1150] CI: retrigger From 6ec1c78b637d56cb85ba8e4e9fb2530652cfb84c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0983/1150] CI: retrigger From 1f1e5e94de91c1ed5dd8e86cb6a83c9306fab4f7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0984/1150] CI: retrigger From 88853fe7639886be54b4b4e80bf81b9bcc017a86 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0985/1150] CI: retrigger From 7c8ffaa94a80ad65bbd823db8e7e2e8ef21cb503 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0986/1150] CI: retrigger From 217118c770b95bef815800d51a49ef85d95411d7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0987/1150] CI: retrigger From a5ae46f609a7cc84ebf10d0ab3ec481d1293ef49 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0988/1150] CI: retrigger From 49befb92bbac86246939db9a9969180abd54fe84 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0989/1150] CI: retrigger From d1158aa2e70dea3f61ebea5b6de225a95e4801ab Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0990/1150] CI: retrigger From 74f121b44544b9e14491cea071fdd67a5b82fc71 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0991/1150] CI: retrigger From eb30a9b26a5e4e7d2521d319439ba892cbc2f73b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0992/1150] CI: retrigger From 96698e1f7612ec4819a981ea0132b58ae5f1ddc1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0993/1150] CI: retrigger From db8904e43ac2bf7e7772c6250c4a92914dc81d85 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0994/1150] CI: retrigger From c94dd1596ee05a0b6488c2b22f68d1f8a0add64a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0995/1150] CI: retrigger From 87d7ab4626180aa2b0732f9c287e6e8523c755e6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0996/1150] CI: retrigger From fd62a2c4eb6c93181505ca5d3ed04b25ca2c60d9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0997/1150] CI: retrigger From 4f262dc99437d07e6230f7ba8df481b09e39df44 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 0998/1150] CI: retrigger From 9532415ddcfa4ed05fee305b2bf9dfc8eb194b29 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 0999/1150] CI: retrigger From bf3454013f5086820f780cf760f77f3ba283bcf5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1000/1150] CI: retrigger From 6e2a63146c275eec7f640e31c2470d1d2643e1fc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1001/1150] CI: retrigger From a8b4008dc7e907de692a35f2377dda1792681a11 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1002/1150] CI: retrigger From 158c826f0a1bb8785a835cdcfe7b62d3e031c659 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1003/1150] CI: retrigger From 7c6814ac95c6b3d3cac2d5db067f6f7fea5df6ab Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1004/1150] CI: retrigger From b04d165f366ff20a080e6923416ed67f567b112b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1005/1150] CI: retrigger From 335171070596a86987b498ceb8d6f728a886a3a4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1006/1150] CI: retrigger From df3014f1ea72944f2e6c463d7c212d8b9e6665ab Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1007/1150] CI: retrigger From 55be3b1314aba7b30bfd41389d95f1d2d0c7bd2e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1008/1150] CI: retrigger From 1f8d5e2146d79cd4784f546feda22b58b09f55f6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1009/1150] CI: retrigger From 9e8cec5f9772306add554f665fa8da8f30e0ec15 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1010/1150] CI: retrigger From 896986f14b2feed2d0549dcd197b479923ea1c0b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1011/1150] CI: retrigger From cda090c0d63c81917b9a6d411eadee0a7a6e562d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1012/1150] CI: retrigger From 1b1bb6538eeeaf751974fcda42e351d99be1a44b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1013/1150] CI: retrigger From d559feb6633b2259e9b4f8eee66f69a10c4d83e9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1014/1150] CI: retrigger From e6ad5745214a66225b22bf1533f3ee74affcddb5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1015/1150] CI: retrigger From bae82584aa547673edf23f1104d2ab10c5aa52a1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1016/1150] CI: retrigger From 480fb85ee6c89fdce89ae4c0effc36f5e4dd34a5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1017/1150] CI: retrigger From d623832c8a71bd155b3c92c57de5b49ec477ac64 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1018/1150] CI: retrigger From 728a00766283c8db3e3e5786097ab0ac2812fcb4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1019/1150] CI: retrigger From 9b0524c30a8bbb3f69efb8943f1fb69b46ad4f95 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1020/1150] CI: retrigger From 629521ff7f912516027c279b64213171ddcc2ea4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1021/1150] CI: retrigger From 0fb79014ebf7cddb91d896700577050693c22703 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1022/1150] CI: retrigger From 652a9165d97d8485c46ee27a8fae6616f4b411f3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1023/1150] CI: retrigger From d9cba717d16f0bffa79e58f58b2876d4e367a835 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1024/1150] CI: retrigger From 1d59fd1f192d5e7729aef017c4d339df8beca837 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1025/1150] CI: retrigger From fc9f2695941e720ed6c0eea8b0f63a408ed44612 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1026/1150] CI: retrigger From f982d37e6fe184f544bfc1d1a8dc4f603a7aa56a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1027/1150] CI: retrigger From 632a7ea56499d6711e433d9209093c7fadad9cba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1028/1150] CI: retrigger From 6a28e09bd50b760a10f6366361f4dd04e6d56c56 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1029/1150] CI: retrigger From 7f0dd128b5dd5a9fdcd967f67a1689aded65933c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1030/1150] CI: retrigger From 8235927db22b714b1b853d9f6b4abe7fe9bbf1d4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1031/1150] CI: retrigger From 6e81aa94b404122288360842b54e575315348e85 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1032/1150] CI: retrigger From 654b39bdedef3606d3ca08b49d320c17b81e5779 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1033/1150] CI: retrigger From 1ef9de8bb1d4a1cf8b8b25761729092ab54151ce Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1034/1150] CI: retrigger From b4eb3eb24e64e0fbdc58bf79666477f6b3567f27 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1035/1150] CI: retrigger From 0465a25fd0876f20517c5986de829769cdd08d5a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1036/1150] CI: retrigger From 91dd8480639c60b10d585b1a0950aadbb8ee9942 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1037/1150] CI: retrigger From c860d82eceaed9149b0b4756aa46d837ec419fd6 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1038/1150] CI: retrigger From e95c701e54c5d140643f32484d2784db5678e410 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1039/1150] CI: retrigger From d1bfe2fadeeef6b0e923fc249fc73e21319b8197 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1040/1150] CI: retrigger From 780376c1ebb32bd3ccd5a57a118ae31bc2463904 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1041/1150] CI: retrigger From 533c477c884a33fd88ca30c70f1f7669271e2480 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1042/1150] CI: retrigger From 5f3344c465f497e967cb896927d130fc533fc572 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1043/1150] CI: retrigger From 6bee19b42e57092033b20917a708e5eb03b0e9ae Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1044/1150] CI: retrigger From c9dfeb79681cee6e032dcce81b3538d114a35f61 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1045/1150] CI: retrigger From 8eb3514b204b2ccfbf687b63e8f4c47e0ac4bb21 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1046/1150] CI: retrigger From 120b60fc28f12247712b54a3d0396db635c406db Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1047/1150] CI: retrigger From 93aa2208b63c87e5fec012250e2c2a6f3f12c7a0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1048/1150] CI: retrigger From 4a80d64a95bf096f1a318c2e1b454ec34312443d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1049/1150] CI: retrigger From 602f9c12c2c68518b8ecf804671e163458feaf04 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1050/1150] CI: retrigger From 142d17e1ecfa4801220ca7541f3d8e3585756ea3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1051/1150] CI: retrigger From 9e2a1020d6ac655c46e25de6b272d2244525ca23 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1052/1150] CI: retrigger From d91b7f888cceb2885647ef260da41e26f2b1b061 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1053/1150] CI: retrigger From b8b6bf3d09d3d4168272deffe27f5d0867754ece Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1054/1150] CI: retrigger From 26159fc9ec52ecf9d129c70a80ddbe5b4f5fb542 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1055/1150] CI: retrigger From aaeefa840ca0578303f5af7aaf571eb89e0900a2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1056/1150] CI: retrigger From 770bb0eecc3333604b677e4272ab19088f9137e8 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1057/1150] CI: retrigger From b4bb2016b0a702dd4f8d72cddb97e35aacfeb509 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1058/1150] CI: retrigger From ad6046501243720043ecadd0737943a887fd4d31 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1059/1150] CI: retrigger From c27a9d421ff81e640c06de8db5222a1c019b9032 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1060/1150] CI: retrigger From 66427a39f8436877bb827c100d93f22b47c39a9f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1061/1150] CI: retrigger From 6050e3e0f71d1004ddcd891c0c25dd9d76da4f77 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1062/1150] CI: retrigger From db8b49a1552110e40d4f6321c2c843db65f659cc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1063/1150] CI: retrigger From 152c8a56637a5cd3ef89e6e420741c220a2e9f00 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1064/1150] CI: retrigger From 99469c53cdaaf6769afc967896ad22ff6e22f64e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1065/1150] CI: retrigger From 49c464b332a061bafd574a604da6188d017a04c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1066/1150] CI: retrigger From 0369febc4f2d3572fa143872d85e46adaf731632 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1067/1150] CI: retrigger From cf3628ab7a73534ba34edc358b78efee2184ec11 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1068/1150] CI: retrigger From 709287eb2b82ed2fb28c453dfb46ba33c663db8e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1069/1150] CI: retrigger From 318225d81ebcdaa78374e77a8fd6e25d23fc65da Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1070/1150] CI: retrigger From 5ad387d4db1006d974c24cfdbe1102825243082a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1071/1150] CI: retrigger From 6f4f2e3b39dbe757bf2713cec61a3666d39ba875 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1072/1150] CI: retrigger From edb2d84d7c62472276eef9f1f3dcd90506df3fcf Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1073/1150] CI: retrigger From 80a9a0e9d76d7be2173fbba48c37e4d079e26c28 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1074/1150] CI: retrigger From 31a360612678eb6bc5066adfa5303bbd0e3f4c67 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1075/1150] CI: retrigger From e85cbab294c4cef29da66abf3aba2e2c60d3a946 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1076/1150] CI: retrigger From 3ee0dbcff0afb7b6b0d5776a7c8e2e2ac47e40c1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1077/1150] CI: retrigger From 0a8125908751e964ed48224323267440ab2bfc7d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1078/1150] CI: retrigger From d6e2e785a2c121045300e325ce46f295981e782f Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1079/1150] CI: retrigger From 6d4ce41964fca1ceaeb12bb8bd31f558ab12d05a Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1080/1150] CI: retrigger From 203617df527eabb69088fef9196bb17deb176374 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1081/1150] CI: retrigger From f2d6b6ee50a6085bc14ccd88f0b73986ff4a573e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1082/1150] CI: retrigger From df93bb00e49323138b925c17500c7f44f38b378e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1083/1150] CI: retrigger From 196fc4aae51d91e16e37f4d8d290628b38636874 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1084/1150] CI: retrigger From 6fde0b82ab13d8b6b0bfb66c00b77ea973e1b3da Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1085/1150] CI: retrigger From 48354e11cebd30069fb5dafa3de24c421d96d625 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1086/1150] CI: retrigger From e1737082f8c95c46db79b2083b366f16708be60e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1087/1150] CI: retrigger From 14a0d48300428358ded532166afb941d026abcc5 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1088/1150] CI: retrigger From 1f81c3b7189762bf49adac1465b8f25c61200339 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1089/1150] CI: retrigger From 37afeb49a4be90f09250c1ca0d93bb2bc2f7c388 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1090/1150] CI: retrigger From a5eb1cb42e76f4530195623bdb24434fca1dfcd7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1091/1150] CI: retrigger From 2238367438280e3ab4fd12431eebc1c73c38e6aa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1092/1150] CI: retrigger From bf6f92dbe4319cc1881f4573cc57d402807f7dfd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1093/1150] CI: retrigger From 929fad43080f5408cba1d6145fb4e6f651feb827 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1094/1150] CI: retrigger From a0980f15732da6d6f621d2dc93b7811559d22217 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1095/1150] CI: retrigger From d95de8b2caefe9272b5f6ae6dd6f551bcddb0167 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1096/1150] CI: retrigger From 701cd1b1c67fb35d4e6e1470d7aff96e0617ac6c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1097/1150] CI: retrigger From 829d697418b598863b8a4f77df921c029673ec5d Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1098/1150] CI: retrigger From 77b0ac04ffb34a1399e51fd6d8a5205d67993ce1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1099/1150] CI: retrigger From e8e34f259cb1e2a6a3f6940eadfa6b211d6eceb2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1100/1150] CI: retrigger From aa88a66c3f2d1ca2a4d0e80d68d8d1f28dde9b1c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1101/1150] CI: retrigger From 941e79c73647e42dbc28f89f0c29c80cd26289f7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1102/1150] CI: retrigger From a89164885464e320c6930fb915c48f955e85246c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1103/1150] CI: retrigger From 2ad180dea9d303718dd19c0bb9abd719a6b1a7a7 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1104/1150] CI: retrigger From 7dba8f85d4af8e43f6f8b024872d21ed727aedaa Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1105/1150] CI: retrigger From b50f2ae4caf18fda6efd496efefdc0ac5f91e938 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1106/1150] CI: retrigger From 937ed2f94fd72b816bee6702d381ef08b7562333 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1107/1150] CI: retrigger From 7e9d2487a5a57f208a477df5b64bd4db6c1857fb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1108/1150] CI: retrigger From 80d933f7abe9652fa18b9a10ea2f902232cd6b0e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1109/1150] CI: retrigger From e0575039ba42a2b6bf2661072d0f7267348a5f2c Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1110/1150] CI: retrigger From 708d6fc9de829ccdebb33f6318fc8595273fe3b0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1111/1150] CI: retrigger From fcd3dc159f64ab0cbd7c776f027d02c358cd3680 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1112/1150] CI: retrigger From 34da1119730d9ef1db4e098610fb1f281459b8c0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 13:21:58 +0200 Subject: [PATCH 1113/1150] CI: retrigger From 773792f680bd9f422ba1c2557c33cb0b7fa58ef4 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1114/1150] CI: retrigger From 5cb9e1ef3524444cdf60523a7a279a1493e1dd63 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1115/1150] CI: retrigger From dc94ffe5fbf2860c6dbf0dd7761a89b4d2a42282 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1116/1150] CI: retrigger From 201887bae8aee5ae6139e989d95e54bfc3b81408 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1117/1150] CI: retrigger From 92d5680786292d6da6b14aee4b3018eeb361f68e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1118/1150] CI: retrigger From c9930ab3db67f4e7a7448ac428ff3b943a682280 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1119/1150] CI: retrigger From 6f0d62292b7bdaa7d09e6f7a15119bf7cd5ce537 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1120/1150] CI: retrigger From b69586e39b9aefbdbca2f69ebe84493551e3fe2b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1121/1150] CI: retrigger From f12a351a751adccd49014c8c5fd6367afcc99a62 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1122/1150] CI: retrigger From 40b7d60cd2e21665a37ba6e3a11c435ab236e6ff Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1123/1150] CI: retrigger From 9c9e8f6a3cbb97134d4a979d45b99fe7a7ca50bd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1124/1150] CI: retrigger From 5c57ce9baa4c6c1abd887e32e00cbd80d0d16bdd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1125/1150] CI: retrigger From fd6e719d19d1af5c1b05728075dc18945ffdcdb3 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1126/1150] CI: retrigger From 8dfabb32fccd4bb9a964c861676e81d1bfe174df Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1127/1150] CI: retrigger From a5f2f89b13cff4fb91324b2a632c1831db1d6cbb Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1128/1150] CI: retrigger From 0bd360363d659b309f385b825faa34d50cc8873e Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1129/1150] CI: retrigger From e62a588289260b20d919015961b30e6744671ab0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1130/1150] CI: retrigger From 228c2aa65db2ebec1e3fff9cf4f8d6c56999c5c0 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1131/1150] CI: retrigger From e0e504731b626962c4f21479528acb3a4cb8f1ba Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1132/1150] CI: retrigger From 38dd15f50618c2e00514b8334e4cafa592580a3b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1133/1150] CI: retrigger From f0a90b22346169a3186a53fb38b3793c972a6dd9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1134/1150] CI: retrigger From 617351b2974e1bd909facb79a4d9c36132312bb1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1135/1150] CI: retrigger From 2d0f828c4ee2c1a986e60c2255e505635cab29fd Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1136/1150] CI: retrigger From a0d0c88bb5c056d6df8d7be15fa59e75cd01e179 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1137/1150] CI: retrigger From dcae354472e8bc49cc1deb1a15e5e9eb899f97e9 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1138/1150] CI: retrigger From 6658a75ed1ae01de6e86c6ad1859524321e2a5a1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1139/1150] CI: retrigger From 0b657ef6926d6a140b6d3089abdc6c37b7102a5b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1140/1150] CI: retrigger From 19f40e62429e2df95525c31f49309aa3fc877d85 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1141/1150] CI: retrigger From baaa8f7a75026980193cdeed9bad1495f406f665 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1142/1150] CI: retrigger From e3a04da675bd9172ce0c6bf3a212cc66b17b2c51 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1143/1150] CI: retrigger From 203bc7cb46d591a539037b661ce0085b1393065b Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 08:04:24 +0200 Subject: [PATCH 1144/1150] CI: retrigger From 99221de10196339f7f0c3855809c3ec276d64009 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1145/1150] CI: retrigger From 0098a4d2f1a32dc6df70633382950ebd442576d1 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1146/1150] CI: retrigger From b46f35f4d6b4e83da824bc6d5deddeb41a359927 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1147/1150] CI: retrigger From 072a0c7954ac4aa386ecf818c099e2a53a492bb2 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 07:52:20 +0200 Subject: [PATCH 1148/1150] CI: retrigger From 626004396ee9b9744cb03684218a87660dc935bc Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Thu, 17 Sep 2026 23:33:13 +0200 Subject: [PATCH 1149/1150] Add reference-chains design and architecture docs --- .../LiveHeapReferenceChains-Implementation.md | 410 ++++++++++++ doc/architecture/LiveHeapReferenceChains.md | 610 ++++++++++++++++++ .../ReferenceChains-SignalsExplained.md | 366 +++++++++++ doc/reference-chains-collection-summary.md | 87 +++ doc/reference-chains-design.md | 259 ++++++++ 5 files changed, 1732 insertions(+) create mode 100644 doc/architecture/LiveHeapReferenceChains-Implementation.md create mode 100644 doc/architecture/LiveHeapReferenceChains.md create mode 100644 doc/architecture/ReferenceChains-SignalsExplained.md create mode 100644 doc/reference-chains-collection-summary.md create mode 100644 doc/reference-chains-design.md diff --git a/doc/architecture/LiveHeapReferenceChains-Implementation.md b/doc/architecture/LiveHeapReferenceChains-Implementation.md new file mode 100644 index 0000000000..b016cd10d3 --- /dev/null +++ b/doc/architecture/LiveHeapReferenceChains-Implementation.md @@ -0,0 +1,410 @@ +# Live Heap Reference Chains — As-Built Implementation Reference + +**Status:** matches `jb/reference-chains` as of 2026-09 +**Jira:** [PROF-15341](https://datadoghq.atlassian.net/browse/PROF-15341) + +This document is the detailed, as-built reference for the reference-chain walk +engine. It records the mechanisms exactly as implemented, with the rationale +that lives in the code comments distilled into one place. + +Companion documents, each with its own scope: + +| Document | Scope | +|---|---| +| `doc/reference-chains-design.md` | The original design decision: why bounded BFS-from-roots was chosen over the alternatives | +| `doc/reference-chains-collection-summary.md` | The four-component summary: detection signal, resumable walk, latency budgets, rotation | +| `doc/architecture/LiveHeapReferenceChains.md` | The full architecture: frontier/tag lifecycle, triggering, termination, data structures | +| `doc/architecture/ReferenceChains-SignalsExplained.md` | A guided tour of the *scheduling* side: cadence, leak-signal gate, pain budget, PID controller, canary backoff, OOM ramp, abort path | + +This document covers what none of the above covers in one place: the pass +structure, root discovery (including static-field roots), candidate-scoped +reach, the canary tagging mechanics, the rotation tiers, the as-built +pacing/budget arithmetic, and the `LivenessTracker` admission boost. Where a +topic belongs to the signals tour (scheduling behavior), this document +cross-references it instead of repeating it. + +--- + +## 1. Data flow + +```mermaid +flowchart TD + LT["LivenessTracker::selectLeakCandidates (population-slope ranking)"] -->|"ranked klass candidates"| PWT["ReferenceChainTracker::pollWatchedTargets"] + PWT -->|"pre-tag each candidate's representative with a distinct marker tag"| CHASE["canary candidate chase"] + BFS["BFS thread: threadLoop"] -->|"shouldRunPass gate (see SignalsExplained §4-§8)"| RP["runPass"] + RP -->|"every pass"| MW["runPassManualWalk"] + MW -->|"cadence-gated (>= 2s apart)"| RE["IterateOverReachableObjects seeds roots"] + MW -->|"when the loaded-class set changed since the last completed sweep"| SF["admitStaticFieldRoots: app-classes-first, chunked, resumable sweep"] + MW -->|"candidates open: before any breadth-first work"| PRONGS["candidate-scoped reach: walkCandidateThreadLocals + walkStaticFieldAnchors descend walks"] + MW --> EF["expandFrontier: batched array-holder FollowReferences"] + EF --> FT["FrontierTable: FRONTIER / EXPANDED / EDGE / ABANDONED"] + MW -->|"reserved rotation slice"| ROT["rotation: leak-accumulation (Tier 1/2) + stale-EXPANDED + root-kind + static-anchor FIFO"] + ROT --> EF + CHASE -->|"heapReferenceCallback prunes a marker-tagged candidate: chain link recorded"| PWT + PWT -->|"buildChainEvent (representative, pruned marker, and up to 8 auto-discovered instances/class)"| RC["cacheResolvedChain: one entry per klass id, cap 128"] + RC -->|"Profiler::dump, snapshot without clearing"| DR["drainPendingChainEvents"] + DR --> JFR["datadog.ReferenceChain / datadog.ReferenceChainAbandoned"] +``` + +Every pass is driven by the manual walk (`runPassManualWalk()`): pure JVMTI +heap calls, which run inside the `VM_HeapWalkOperation` safepoint. The walk +reads no raw oop — JVMTI's iterators apply the active collector's own +barriers, so the walk is correct on every collector, including ZGC, where +concurrent relocation would corrupt a raw-oop reader. The phases below run in this order, each with its own +deadline slice (the per-sub-operation deadline is reset, so an earlier phase +cannot eat a later phase's slice): + +1. root/stack-ref enumeration (cadence-gated), +2. candidate thread-local descend walks (when a chase is open), +3. static-field sweep (when the loaded-class set changed), +4. `expandFrontier()` (ordinary breadth-first progress), +5. static-anchor descend walks, +6. rotation re-expansion. + +A rotation slice is reserved up front, before any of the above can spend the +whole pass budget. The reservation is capped at half the expand budget: a +pacing-throttled pass degrades both sides proportionally instead of starving +ordinary expansion to zero (or rotation to zero). + +## 2. Root discovery + +### 2.1 Root/stack-ref enumeration + +`IterateOverReachableObjects` walks the roots and dispatches +`heapRootCallback()`/`stackRefCallback()`. Root enumeration pays its fixed +root-walk-and-dispatch cost in full on every run, regardless of budget, so it +is gated by `ROOT_ENUM_MIN_INTERVAL_NS` (2s): it does not re-fire at the +per-second pass cadence. A budget-exhausted truncation here retries on the +next pass; a frontier-cap hit abandons the search (below). Note that root +enumeration alone never discovers a root's transitive children — the +callbacks are given no oop, only a tag pointer — so `expandFrontier()` is +always needed for further progress, first pass or resumed. + +### 2.2 Static-field roots (`admitStaticFieldRoots()`) + +An object held only by `SomeClass.staticField` is not reachable through +`IterateOverReachableObjects`' root/stack-ref callbacks at all. This is +precisely the leak shape production pods showed: a growing `static final` +collection. The sweep: + +- Repartitions the per-call `GetLoadedClasses()` array app-classes-first (any + non-bootstrap classloader), in place, every call — `GetLoadedClasses()` + gives no ordering guarantee across calls, so the sweep reprioritizes each + time. A likely leak source is reached within the first chunks instead of + after every JDK platform class. +- Advances `STATIC_FIELD_SWEEP_CHUNK_CLASSES` (512) classes per pass through + a resumable cursor (`_static_field_sweep_cursor`). A single un-chunked + `FollowReferences` over every loaded class could not finish inside one + pass's safepoint deadline on a JVM with tens of thousands of classes. +- Retries a truncated chunk on the next pass; a lap that truncated even once + is not marked done (`_static_field_sweep_cycle_truncated`), per the + subsystem's "no silent truncation" contract. +- Only runs at all when the loaded-class set has *actually changed* since the + last completed lap (`_last_static_field_class_count` differs from the count + `resolveLoadedClasses()` refreshed this same pass). Otherwise it would + re-pay a stop-the-world walk over every class on every pass, forever. +- Admits `STATIC_FIELD` edges always; caps non-static class edges + (constant-pool, interface, superclass, classloader) at + `STATIC_FIELD_SWEEP_NON_STATIC_CAP_PER_CLASS` (32) per class, so one outlier + class cannot blow the chunk's deadline. + +### 2.3 Candidate-scoped reach (descend walks) + +Once a candidate chase is open, breadth-first progress alone can leave the +tagged instances queued behind the ordinary backlog for many passes. Each +open chase therefore gets bounded *descend walks* — `FollowReferences` from a +specific anchor, up to `DESCENT_HOPS` (16) hops below it (raised from 6 after +pod round 7: the static-`ExecutorService` → queue → task → accumulator → +list → chunk shape is ~7 deep; the walk is deadline-bounded per slice either +way, so a deeper cap just lets each bounded walk cover the whole holder +interior — already-admitted entries are skipped, so repeated passes march +deeper each time): + +- **Prong 1, thread-locals** (`walkCandidateThreadLocals()`): descend-walks + the qualifying threads' `Thread` objects — registered via + `Profiler::onThreadStart/onThreadEnd` + (`registerThreadObject()`/`unregisterThreadObject()`, a mutex-guarded + tid → global-ref map; a running thread's `Thread` object is reachable via + the VM anyway, so the strong ref does not distort reachability) — through + their `ThreadLocalMap` subgraphs. A `Thread → ThreadLocalMap → table[] → + Entry → value → holder → chunk` chain is 5-6 hops that ordinary BFS may + reach very late, if at all. Capped at `THREAD_WALK_MAX_ANCHORS` (4) per + pass, rotated fairly across the (klass, tid) pairs by a cursor. +- **Prong 2, static anchors** (`walkStaticFieldAnchors()`): descend-walks + root-attached static holders. Anchors are selected by + `collectStaticFieldAnchorsForRotation()` with *tiered* per-tier cursors + (fresh container-interface holders first); classes are classified once by + `reconcileAnchorClassShapes()` (capped at + `ANCHOR_SHAPE_RECONCILE_BUDGET` = 128 distinct classes per pass — + `resolveContainerInterfaceTags()`/`classImplementsContainerOrMap()` decide + whether a class implements a `Collection`/`Map`-like interface); at-risk + holders (frontier entries whose parent died) flow through a FIFO + (`pushAtRiskStaticAnchor()`/`drainStaticAnchorFifo()`, popped + `STATIC_ANCHOR_FIFO_DRAIN` = 16 per pass). Capped at + `STATIC_ANCHOR_ROTATION_BUDGET` (32) anchors per pass. Selection size and + walked size are decoupled: a truncation requeues un-walked anchors + (`requeueStaticAnchorFifoFront()`), so a larger selection never extends the + pause — it can only spend selection-scan time, which holds no safepoint. + +Both prongs resolve their anchor batch with a single `GetObjectsWithTags` +call (the call's O(tag_map) cost is the dominant term on a large tag map — +the same batching rationale as `expandFrontier()`). + +## 3. Frontier expansion and batching + +`expandFrontier()` resolves up to `budget` pending tags in *one* +`GetObjectsWithTags` call, packs the live objects into a single JNI array +(`holder`), and expands the whole batch via exactly one +`FollowReferences(initial_object = holder)` — one BFS level per +VM-safepoint operation rather than one safepoint per entry: + +```mermaid +sequenceDiagram + participant BFS as BFS thread + participant JVMTI as JVMTI + participant JNI as JNI holder array + participant CB as heapReferenceCallback + + BFS->>BFS: pull up to budget tags from front of _pending_expand + BFS->>JVMTI: GetObjectsWithTags, resolve which tags are still live + JVMTI-->>BFS: live jobject references + BFS->>JNI: EnsureLocalCapacity, then NewObjectArray to build holder + alt exception, EnsureLocalCapacity failure, or null holder + BFS->>BFS: ctx.truncated = true, retry this batch next pass + else holder built successfully + BFS->>JNI: SetObjectArrayElement per resolved object + BFS->>JVMTI: FollowReferences, initial_object = holder + JVMTI->>CB: heapReferenceCallback per outgoing edge + CB-->>JVMTI: descend only for batch_tags boundary objects + JVMTI-->>BFS: one BFS hop expanded for the whole batch + BFS->>BFS: markExpanded, admitObject appends children to _pending_expand + end +``` + +The JNI/JVMTI error handling is defensive by construction: a null `holder`, +a pending JNI exception after `NewObjectArray`/`SetObjectArrayElement`, or an +`EnsureLocalCapacity` failure all set `ctx.truncated = true` (retry next +pass) instead of marking the batch permanently `EXPANDED`. A failed +`java/lang/Object` class resolution with pending work also forces +`truncated = true`, so `runPass()` cannot mistake it for +`SearchState::COMPLETED`. + +## 4. The canary candidate chase + +`pollWatchedTargets()` pre-tags each candidate's *specific representative +object* (identity match — matching by class alone could record a chain for an +unrelated, possibly short-lived, instance of the same class) with a distinct +negative marker tag (`MARKER_TAG_BASE - i`; the negative range is disjoint +from the positive frontier-tag counter and the negative class-tag range). +When the walk's `heapReferenceCallback()` later encounters one, it *prunes* +it: records the candidate's referrer link (parent tag, referrer klass, +depth) for `buildCanaryChainEvent()`, and does not expand through it. + +The walk also auto-marks *any* instance of a watched class it discovers (up +to `MAX_DISCOVERED_INSTANCES_PER_CLASS` = 8 per slot): a leaking class's +many live instances each have independently useful chains — different +parents, different retention paths — and the pre-tagged representative is +just one sample. + +The chase's *scheduling* (back-to-back on progress, work-scaled exponential +backoff on no progress, pain-budget refill) is covered in +`ReferenceChains-SignalsExplained.md` §8, including the measured pod incident +(32 minutes at ~88 passes/min) that motivated the backoff. The *termination* +half is here: a chase that has made zero candidate-discovery progress for +`CANARY_NO_PROGRESS_PASS_LIMIT` (30) consecutive passes is abandoned with +`SearchAbandonReason::CANARY_STUCK`. Unlike the TTL check, this fires even +while `isUrgent()` holds: a zero-progress chase at urgency-boosted +budget/cadence only burns STW pause budget the process needs during the same +OOM approach the chase was launched to diagnose. The abandon limit *widens* +with repeated stuck restarts (`_canary_stuck_restart_count` survives +`restartSearch()`), so a permanently un-findable candidate cannot keep +cycling at the base limit. + +## 5. Rotation: rediscovering growth in an already-visited container + +The frontier walk visits each object once. That is insufficient for the leak +shape this feature targets: a `static final` collection field that is +*appended to*, not reassigned. The container is already `EXPANDED` long +before its element klass earns a leak signal. `runPass()`'s completion branch +therefore requires `_watched_leak_klass_count == 0` (no klass under active +leak watch) before moving to `SearchState::COMPLETED`; while a klass is +watched, the search stays `RUNNING` and rotation re-queues the growing +container: + +```mermaid +flowchart TD + LT2["LivenessTracker::topKlassesByGenerationCount"] -->|"refreshed once per tick,
only after hasLeakSignal() fires"| WK["_watched_leak_klass_ids
(max 5)"] + WK -->|"klass_id newly watched"| SEED["seedLeakAccumulationForNewlyWatchedKlass:
one-time scan of already-EXPANDED
frontier entries by class_tag"] + ADM["admitObject: ADMITTED"] -->|"class_tag of new object"| TLA["trackLeakAccumulation"] + SEED --> TLA + WK -->|"class_tag match?"| TLA + TLA --> T1["Tier 1: _leak_signature_totals
(leaf_klass_id, parent_class_id) -> count"] + TLA --> T2["Tier 2: _leak_parent_fanout
parent_tag -> count, within winning signature"] + T1 -->|"delta vs previous pass's snapshot"| RANK["collectLeakAccumulationCandidatesForRotation:
pick winning signature, then its top parent_tag(s)"] + T2 --> RANK + RANK -->|"re-queue for re-expansion,
budget 16/pass"| EF2["expandFrontier"] + EF2 -->|"new elements admitted"| ADM +``` + +Matching a newly-admitted object against a watched klass id uses a stable +JVMTI class tag (`classTagAllocator.h`, shared between +`ReferenceChainTracker` and `LivenessTracker`) rather than the classMap +`StringDictionary` id — that id is not guaranteed stable if the dictionary +is compacted/regenerated mid-search (observed live: the same class resolved +to two different classMap ids from two subsystems). + +The rotation tiers, with their per-pass budgets: + +| Tier | Budget/pass | What it selects | +|---|---|---| +| Leak-accumulation (Tier 1/2 above) | 16 | The containers of currently-flagged klasses — the targeted tier | +| Stale-EXPANDED (own cursor) | 256 | Any long-`EXPANDED` entry — low-priority eventual coverage of the whole table; deliberately *not* scaled with table size (two earlier versions tried; see git history) | +| Root-kind (own cursor) | 16 | Transient-root entries, so attribution converges to a durable root kind | +| Static-anchor FIFO | 16 drained | At-risk holders whose parent died, into the same `walkStaticFieldAnchors()` batch | + +The stale-EXPANDED tier's own cursor exists so a frontier table full of +long-lived infrastructure objects cannot permanently starve higher-tag +entries of ever being re-queued. + +## 6. Termination + +```mermaid +stateDiagram-v2 + direction LR + [*] --> RUNNING + RUNNING --> COMPLETED: frontier drained,
no truncation this pass,
no leak klass still watched + RUNNING --> ABANDONED_TTL: wall-clock TTL exceeded
with work still pending
(suppressed while urgent) + RUNNING --> ABANDONED_CAP: frontier-size cap hit + RUNNING --> ABANDONED_CANARY: zero candidate-discovery progress for
CANARY_NO_PROGRESS_PASS_LIMIT (30) passes + ABANDONED_TTL --> RUNNING: restartSearch + ABANDONED_CAP --> RUNNING: restartSearch + ABANDONED_CANARY --> RUNNING: restartSearch + COMPLETED --> RUNNING: restartSearch +``` + +The abandon reason is recorded (`SearchAbandonReason`) and surfaced as the +`datadog.ReferenceChainAbandoned` JFR event — no silent truncation. +`releaseSearchTags()` clears every live tag the search still owns once it +ends, without discarding the frontier table's own records: chain +reconstruction keeps working from memory after the search ends. + +Restarts are gated on an actual leak indication (a +`selectLeakCandidates()` candidate, or an urgent-latched seconds-to-OOM +projection — one search per latched episode via `_urgent_search_spent`) plus +the safepoint pain budget. This closes a structural gap: a one-shot walk +could finish before population-trend detection accumulated enough GC epochs +to flag a candidate, leaving anything allocated afterward permanently +undiscoverable. See `ReferenceChains-SignalsExplained.md` §5-§6 for the +gate's full behavior. + +## 7. Pacing and budgets (as-built arithmetic) + +- **PID controller on genuine in-safepoint time.** `updatePacing()` is fed + only each pass's in-safepoint ticks — measured across every + `IterateOverReachableObjects`/`FollowReferences` call the pass makes, + explicitly excluding `GetObjectsWithTags` (not a safepoint call) and every + bookkeeping line in between. Non-safepoint bookkeeping must not be + mistaken for pause-time-SLO pressure. The controller's overflow term also + widens/narrows the fallback cadence. +- **Separate CPU pain budget.** Non-safepoint pass cost is charged to + `_cpu_pain_budget`, a second `PainBudget` sharing the same + `painbudget=N` percent knob as the safepoint pain budget — one + operator-facing "acceptable background cost" percentage covers both leaky + buckets. +- **Budget borrowing.** A sustained run of comfortably-under-target passes + (after a warmup) earns extra headroom above `_budget` + (`_borrowed_budget`); any pass that is not comfortably under target + revokes it immediately (`maybeRevokeBorrowForRootEnumPass()` also revokes + on a root-enum pass), so `_budget` itself stays the ceiling the instant + the search stops proving it has room. +- **First-pass budget.** The search's one-shot root-seeded first pass draws + its own much larger edge budget (`firstpassbudget`, auto-scaled 10× from + `budget`, capped) exactly once — a steady-state budget sized for cheap + incremental expansion would truncate a cold full-graph walk long before + it reaches anything interesting — and its duration is excluded from the + pacing signal so it cannot throttle every cheap pass that follows. +- **Urgency ramp.** See `ReferenceChains-SignalsExplained.md` §9: pause + target and cadence ramp exponentially toward their ceilings as + `secondsToOOM()` falls inside `OOM_RAMP_START_S` (30 min); the budget + ceiling is held at 4× for the ramp's entire duration; the ramp owns + `_effective_cadence_ns` outright while active. +- **Auto-tuned defaults.** `autoTuneDefaults()` scales the defaults for + budget (√heap-proportional), first-pass budget, TTL, frontier cap, and + pause target (capped at 50ms) with the resolved max heap / container + limit, but only for sub-options the operator did not set explicitly. + +The urgency *latch* is covered in `ReferenceChains-SignalsExplained.md` §9. +On the `LivenessTracker` side, `secondsToOOM()` itself: accounts for the +container memory limit (not just `-Xmx`) when projecting the exhaustion +point; requires a confirmed rising heap-floor trend; and corroborates the +projection with a recent-half slope check, so a single stale sample cannot +spike or collapse it. + +## 8. `LivenessTracker` admission boost (chase phase) + +While a candidate chase is open, allocations by the candidates' *qualifying +threads* are admitted to liveness tracking at 100% (`admitForTracking()`, +published two-phase — slots first, then the count with RELEASE — by +`noteSelectedCandidates()`), and urgency admits everything. Without this, +the default 10% liveness subsampling thins small per-(klass, tid) +populations enough to make leak-tag correlation intermittently fail on real +pods (observed live as the intermittent zero-tag runs in +`LeakTagCorrelationReferenceChainTest`'s own lottery analysis). The boost +only ever adds admissions on top of the configured ratio — fail-open by +construction: a stale or missed boost cannot drop an allocation the ratio +would have admitted. The draw itself is the process-wide xorshift64 stream +from main's sampling refactor (#794): per-thread TLS state, an integer +threshold compare, no `` machinery. + +## 9. Output path and JFR persistence + +A resolved chain is cached per klass id (`_resolved_chains`, capped at +`MAX_RESOLVED_CHAINS` = 128 entries, drop-not-evict once full — surfaced as +`REFERENCE_CHAIN_EVENTS_DROPPED`) and re-stamped into *every* subsequent +dump the sample survives into (`drainPendingChainEvents()` snapshots without +clearing, mirroring how `LivenessTracker` re-emits live-object samples). A +long-lived leak's chain is therefore present in each JFR chunk, not only in +the chunk active when it was first reconstructed. The write happens on the +dump thread (`Profiler::dump()`), never on the BFS thread — the walk never +blocks on JFR I/O, and JFR writes never trigger a walk. + +## 10. Configuration + +`referencechains=true:hops=N:budget=N:ttl=N:framecap=N:pausetarget=N:painbudget=N:firstpassbudget=N` + +- Negative `hops`/`budget`/`framecap` values are floored (an unfloored + negative `hops` would wrap to ~4e9 as a `u32`, silently disabling the cap + it is meant to enforce); all three are also ceiling-clamped. +- `ttl <= 0` disables the wall-clock TTL cutoff. +- Unset values are auto-tuned (see §7). + +## 11. Why the tuning pass is not a JMH benchmark + +The defaults above are placeholders pending a measurement pass. That pass +cannot be a JMH benchmark, for three reasons: + +1. **The cost is not per-Java-operation.** JMH measures the throughput and + latency of a benchmark method. This subsystem's cost is (a) STW + safepoint pauses from `VM_HeapWalkOperation` — global stalls + attributable to no benchmark method — and (b) CPU on a dedicated native + background thread. Both reach a Java workload only as indirect + throughput degradation mixed with GC noise; JMH can neither observe + nor attribute them directly. +2. **Activation is leak-signal-gated.** No pass runs until the population + rings fill (~10 GC epochs), trend hysteresis clears (5 consecutive + qualifying epochs), and the search gate opens. A seconds-scale JMH + iteration measures the feature idle. Exercising the chase needs minutes + of continuous leaking allocation; run-to-run variance is then dominated + by GC cadence and by *when* hysteresis cleared — exactly the steady-state + assumption JMH's fork/iteration statistics make. +3. **The knobs control quantities JMH cannot see.** `pausetarget`, + `budget`, cadence, backoff, and the pain budgets regulate per-pass + safepoint duration, pass-cost EMA, and refill rates — all directly + observable as `jdk.ExecuteVMOperation[HeapWalkOperation]` JFR durations + and the tracker's own pass telemetry, which is what the shipped `utils/` + repro/sweep/report tooling consumes. + +A coarse JMH A/B (leaking workload, `referencechains` off vs on) remains +possible with the repo's existing `ddprof-stresstest` JMH setup, and would +serve as an end-to-end throughput regression guard. It cannot tune these +defaults. Tuning them requires the JFR-based measurement pass above. diff --git a/doc/architecture/LiveHeapReferenceChains.md b/doc/architecture/LiveHeapReferenceChains.md new file mode 100644 index 0000000000..5276da3f6b --- /dev/null +++ b/doc/architecture/LiveHeapReferenceChains.md @@ -0,0 +1,610 @@ +# Reference Chains for Surviving Live Heap Samples + +**Status:** Implemented (see "Implementation status" below) +**Date:** 2026-07-07 +**Jira:** [PROF-15341](https://datadoghq.atlassian.net/browse/PROF-15341) + +## Implementation status + +The "Chosen design" section below has been implemented following +`LiveHeapReferenceChains-ImplementationPlan.md` (kept locally, not committed) +(Phases 0-7). It is off by default; the shipping switch is the `referencechains` argument +parsed by `Arguments` (`arguments.cpp`'s `CASE("referencechains")`), e.g. +`referencechains=true:hops=64:budget=2000:ttl=60000:framecap=65536`. + +Read this status note alongside the actual code before relying on it, not instead of it: + +- **The BFS engine (frontier table, tag lifecycle, incremental resumption, termination, + JFR event shapes) is implemented and unit-tested** (`ddprof-lib/src/main/cpp/referenceChains.h`/ + `.cpp`, `ddprof-lib/src/test/cpp/referenceChains_ut.cpp`). +- **The lifecycle gap is closed: it now runs inside a live profiling session.** + `Profiler::start()` (`profiler.cpp`) calls `ReferenceChainTracker::instance()->start(args)` + (gated on `args._reference_chains`, independent of the CPU/wall/alloc engine mask, the + same way `malloc_tracer`/`NativeSocketSampler` are gated on their own flags) followed by + the new `ReferenceChainTracker::startThread()`, which spawns the BFS thread + (`threadLoop()`) - safe there because the JVM/JVMTI environment is already fully up by + that point in the lifecycle, unlike inside `start()` itself, which must stay callable + with no live JVM for `referenceChains_ut.cpp`'s tests. `Profiler::stop()` calls the + matching `stopThread()`/`stop()` pair. Because `start()` runs, `SetEventNotificationMode` + for the GC callbacks is now actually invoked, so `onGCStart()`/`onGCFinish()` fire and the + BFS thread's `shouldRunPass()` scheduling loop (GC-epoch signal or the fixed cadence) is + live. A `datadog.ReferenceChainAbandoned` event now reaches a real `Recording`: when a + dump is requested (`Profiler::dump()`, the same call site that already flushes + `LivenessTracker`) and the search's state is `SearchState::ABANDONED`, + `buildAbandonedEvent()`'s output is written via the new + `Profiler::writeReferenceChainAbandoned()` / `FlightRecorder::recordReferenceChainAbandoned()` + wrappers (mirroring `writeHeapUsage()`'s exact shape). +- **`buildChainEvent()` now has a call site: the target-selection feed is closed.** + `ReferenceChainTracker::pollWatchedTargets()` (referenceChains.cpp), called from + `threadLoop()` once per scheduling cycle after `runPass()`, is that feed. It polls + `LivenessTracker::selectLeakCandidates()` (the positive population-slope ranking, Open + Question 3 below) and, for each ranked klass's live representative instance that an + ordinary `runPass()` walk has *already* tagged (`getTag() > 0` - a read, never a `SetTag` + seed; see Open Question 3 for why the design's original seeding proposal was replaced), + calls `buildChainEvent(tag, ...)`, which is cached (`cacheResolvedChain()`) rather than + written immediately. The actual JFR write happens later and on a different thread: enqueued + via `enqueueChainEvent()` and drained by `Profiler::dump()` -> `drainPendingChainEvents()` -> + `Profiler::writeReferenceChain()` / `FlightRecorder::recordReferenceChain()` (`profiler.cpp` + lines ~1960-1974), decoupling the walk from JFR I/O. Deduplication is `_resolved_chains` + (an `unordered_map` keyed by klass_id, `referenceChains.h`), not a + per-search tag set, so a klass flagged across consecutive polls emits its chain only once. + This is gated on `_gc_generations` *and* + liveness tracking both being enabled - `referencechains=...` alone still gets the + whole-graph-only behavior (no target seeding), resolving Open Question 3's "still + undecided" fallback. See + `ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTrackingTest.java`: + `shouldReportAbandonedSearchOnTinyFrontierCap` exercises the abandonment path. + `shouldReconstructReferrerChainToGcRoot` (Phase E's own exit criterion) is no longer + `@Disabled`: it allocates a growing, real population of a fixture class, drives GCs and + `Profiler::dump()` calls until `LivenessTracker::selectLeakCandidates()` trusts the resulting + trend, then asserts on the `datadog.ReferenceChain` event `pollWatchedTargets()` produces - + a real end-to-end exercise of this whole mechanism against a live JVM, not a synthetic + frontier fixture. +- **Phase 5's tuning defaults are provisional, not empirically finalized.** The hop cap, + per-pass budget, TTL, and frontier-size cap (`arguments.h`'s `DEFAULT_REFERENCE_CHAINS_*` + constants) are explicitly-labeled placeholders; no benchmark against this codebase has + run yet (see Open Question 2 below and the implementation plan's Phase 5). + +## Goal + +For a subset of live-heap samples that survive past their allocation window, produce +a **reference chain** — a sequence of referrer *types* (not full field-level paths, not +necessarily to *all* GC roots) connecting the sampled instance back to *a* GC root. This +is diagnostic information ("what kind of object chain is keeping this alive"), not a +heap-dump-grade exact retainer analysis. + +## Constraints + +- Must run cheaply, with as short a safepoint / STW contribution as possible. +- Must work on stock vendor JDKs the agent attaches to — no forked/patched JVM builds. +- Exhaustive (all-roots, full-path) chains are explicitly **not** required; referrer-type-only, + bounded-depth, best-effort chains are acceptable. + +## Approaches considered + +Three approaches were evaluated; two are ruled out as launch requirements for concrete, +evidence-backed reasons. One sub-idea (Approach C's `ParallelObjectIterator` variant) is +explicitly kept open as a conditional future option; see its discussion below. + +| # | Approach | Completeness | Complexity | Feasibility | Status | +|---|---|---|---|---|---| +| A | Full JVMTI `FollowReferences` reverse-graph walk, piggybacked on an already-scheduled major GC | 4/5 | 4/5 | 2/5 | Rejected | +| B | Bounded BFS-from-roots with frontier pruning (JFR "leak profiler" technique, adapted) | 3/5 | 3/5\* | 4/5 | **Chosen** | +| C | Hook GC mark/copy closures (G1, ZGC) to record parent pointers inline during marking | 2/5 | 5/5 | 1/5 | Rejected | + +Scale (1-5 for each column): Completeness — higher is more complete (5 = closest to exhaustive all-roots/full-path); Complexity — higher is more complex to implement/maintain (5 = most complex, lower is better); Feasibility — higher is more feasible to ship on stock vendor JDKs (5 = most feasible). No single column dominates the decision; see the per-approach rationale below for why B was chosen despite not scoring highest on every column. + +\* This 3/5 reflects only the single-pass BFS sketched at selection time. The "Chosen +design" section below replaces that sketch with an incremental, resumable BFS +(JVMTI-tag-based frontier persistence across GC cycles, an agent-thread-driven pass that +gets its safepoint transparently from the JVMTI heap-walk call it makes, GC-callback +signaling, and explicit termination/tag-cleanup bookkeeping), which is materially more +complex than this score suggests — closer to 4/5 in implementation and maintenance +effort. The score is left unchanged above (it documents the state of the comparison at +decision time) rather than retroactively edited. + +### A — Full reverse-reachability walk (rejected) + +Safepoint length scales with live-set size regardless of how the walk is triggered. +Modern regionalized collectors (G1, Shenandoah) rarely perform a true full-heap walk +during ordinary major GCs, so "ride an already-paid pause" is not a reliable amortization +strategy. Cost is fundamentally at odds with the "short safepoint" constraint. + +### B — Bounded BFS-from-roots (chosen) + +Mirrors OpenJDK's own `jdk.OldObjectSample` leak-profiler implementation +(`src/hotspot/share/jfr/leakprofiler/chains/{edgeStore,bfsClosure,dfsClosure}.cpp`): +a `VM_Operation`-driven BFS from GC roots, retaining only edges on the frontier toward a +small, fixed sample set, with a hard hop cap (HotSpot itself caps chains at ~200 hops, +split 100/100 from leaf and from root). We can go cheaper than JFR because only the +**referrer class**, not object identity or field name, is needed — the `EdgeStore` +degenerates to `(referrer_klass, parent_tag, depth)` records, where `parent_tag` links +each record back to the record that discovered it, enabling chain reconstruction. + +Adopting this pattern is a re-scoping of proven, shipping HotSpot code, not a novel +algorithm design. + +### C — GC mark/copy closure piggyback (rejected) + +Investigated specifically for G1 and ZGC on the premise that per-edge referrer +information is already available inside the collector's own marking/evacuation closures +(`G1ParCopyClosure::do_oop_work`, ZGC's `ZMarkConcurrentRootsIteratorClosure` / +load-barrier closures), so recording it would cost nothing beyond what the GC already +pays. + +Rejected because there is no stable, externally reachable hook into these closures: + +- They are internal, template-instantiated C++ classes compiled into `libjvm.so` at + HotSpot build time — not a registrable/pluggable extension point. +- This differs categorically from `VMStructs`-style introspection already used in this + codebase (`ddprof-lib/src/main/cpp/hotspot/vmStructs.cpp`), which reads VM state + passively via an officially exported offset table. Intercepting a GC closure's + *behavior* would require either shipping a patched OpenJDK build (a fork/maintenance + commitment far beyond anything in this codebase) or binary-patching unversioned, + per-build-mangled function addresses — not shippable across JDK point releases. + +A related idea — using HotSpot's internal `ParallelObjectIterator` +(landed via [JDK-8322043](https://www.mail-archive.com/serviceability-dev@openjdk.org/msg12977.html), +used by `VM_HeapDumper` to partition heap regions across GC worker threads for parallel +heap dumping) to shrink Approach B's safepoint by parallelizing the walk — was also +investigated. Same verdict: it is an internal C++ class, not exposed via JVMTI, with no +stable ABI for an attached agent to call. Symbol-sniffing internal HotSpot functions *is* +an established pattern in this codebase (`VMStructs::findHeapUsageFunc`, +`vmStructs.cpp:489-509`), but that precedent covers a single leaf virtual method with a +value/POD-ish return; `ParallelObjectIterator` is a multi-class subsystem that coordinates +the VM's own GC worker threads under safepoint control — an order of magnitude larger +fragility surface, with a much higher blast radius if a layout assumption is wrong (GC +worker-thread coordination corruption vs. a bad JMX stat). Not pursued as a launch +requirement; revisit only if Approach B's single-threaded pause proves to be a measured +bottleneck, and treat it as an isolated, heavily version/flag-gated fast path with +automatic fallback — never a dependency. + +## Chosen design: incremental, resumable bounded BFS + +A single-pass bounded BFS still means one pause sized to whatever budget is configured. +The refinement below spreads that budget across multiple short passes instead of one +contiguous one, trading a possibly-higher *aggregate* STW total for a much better +*latency distribution* — no single long tail pause. + +### Why the frontier can survive across passes: JVMTI object tags + +The obstacle to pausing and resuming a BFS is that the frontier (the worklist of +not-yet-expanded objects) is normally a set of raw addresses, and a moving/compacting GC +between passes can relocate or collect any of them. + +JVMTI object tags solve this: + +- Tags are identity-based and GC-move-transparent — a tagged object can be re-resolved + after a GC regardless of where it moved. +- Tags are **non-retaining** — tagging does not keep an object alive. This is a *new* + subsystem dependency, not a reuse of one: the existing live-object sampler + (`LivenessTracker`, `livenessTracker.cpp`) does not use JVMTI tags at all — it + correlates sampled objects via JNI weak global references (`NewWeakGlobalRef`) held in + its own index table with its own locking and GC-triggered cleanup + (`livenessTracker.cpp:327-357`, `:53-70`). Non-retention is a property both mechanisms + happen to share, not evidence that this reuses proven infrastructure. +- **Investigated replacing tags outright with `LivenessTracker`'s weak-ref + index-table + pattern — resolved as "adopt the table pattern, keep the tags."** The pattern cannot + fully substitute for tags: `FollowReferences`/`IterateThroughHeap` (the calls that + actually discover a frontier object's referrers) can filter/report against a *tagged* + object set natively; a JNI weak-ref table has no hook into that machinery, so the + frontier-discovery step would still need tagged objects regardless of what stores the + metadata. What the investigation *does* carry over: `LivenessTracker`'s proven + `TrackingEntry`-style slot table — CAS-based index allocation, a signal-safe `SpinLock` + (`spinLock.h`), doubling-resize, and GC-epoch-triggered cleanup + (`livenessTracker.cpp:152-176`, `:213-278`, `:369-409`) — is a better-precedented design + for the frontier's *metadata* storage than inventing one from scratch, since a JVMTI tag + is a single `jlong` with no room for `(parent_tag, referrer_klass, depth)` on its own. + Recommendation: use the tag as an index into a `TrackingEntry`-style table (fields: + `parent_tag`, `referrer_klass`, `depth`) rather than encoding all three into the tag + value or a from-scratch hashmap. See "Frontier metadata storage" below and Open + Question 4. +- Non-retention gives incremental resumption a useful side effect for free: if a frontier + object dies between passes, it simply fails to re-resolve on the next pass. That branch + of the search is pruned automatically, with no extra liveness bookkeeping required. + +### Data structures + +- **Frontier**: a set of `(tag, parent_tag, referrer_klass, depth)` records. `tag` is the + JVMTI tag assigned to a not-yet-expanded object; `parent_tag` links back for chain + reconstruction; `depth` supports the hop cap. +- **EdgeStore**: accumulates `(referrer_klass, parent_tag, depth)` per discovered edge for + objects that are on a path toward a target sample. Keyed by tag, not address — + degenerate relative to JFR's `EdgeStore` since object identity/field names are not + required, but it retains the same `parent_tag` linkage field as the Frontier so a chain + can be walked back from a target sample to a root by following `parent_tag` across + EdgeStore records. + +### Frontier metadata storage: reusing `LivenessTracker`'s table pattern + +A JVMTI tag is one `jlong` — it can identify a frontier object and make it visible to +`FollowReferences`/`IterateThroughHeap`, but it cannot itself hold the three fields +(`parent_tag`, `referrer_klass`, `depth`) each Frontier/EdgeStore record needs. Two ways +to close that gap were considered: + +1. Build a bespoke hashmap keyed by tag value, from scratch. +2. Reuse `LivenessTracker`'s existing slot-table design (`livenessTracker.h:21-30` + `TrackingEntry`, `livenessTracker.cpp:152-176` sizing, `:213-278`/`:369-409` CAS slot + allocation and doubling resize, `spinLock.h`'s signal-safe `SpinLock`), using the tag + value as the slot index instead of a `jweak` as the identity handle. + +(2) is the better-precedented choice — it's shipping code, already exercises the exact +"per-slot payload, GC-cycle-driven cleanup, contention-safe locking" shape this needs — +provided the sizing formula is **not** copied as-is. `LivenessTracker` sizes its table +from `max_heap / sampling_interval` (a flat allocation-sample rate, +`livenessTracker.cpp:152-176`, capped at `MAX_TRACKING_TABLE_SIZE = 262144`, +`livenessTracker.h:39`); a BFS frontier's width is driven by per-hop fan-out in the object +graph, not by an allocation rate, and multiple concurrent searches (one per live-heap +sample being chased, see Open Question 3) each need their own capacity — the existing +formula does not transfer and a new one is an open question (folded into Open Question 2). + +### Algorithm + +1. Seed the frontier from GC roots (first pass) or from the persisted frontier + (resumed pass). +2. Resolve currently-live tagged frontier objects. Objects that fail to resolve are + dropped (dead — free pruning). +3. Expand the frontier up to a fixed per-pass budget (edge count or time slice). +4. Newly discovered objects are tagged and added to the frontier for the next pass. +5. Persist the frontier (native memory owned by the agent, not thread-local scratch) and + return control to the VM. +6. Repeat until: a target sample is reached, the hop cap is hit, or a per-search + abandonment limit (see Termination) is exceeded. + +### Triggering passes: resolved — the profiler never schedules its own safepoint + +Investigated whether pass-continuation work could ride the JVMTI +`GarbageCollectionStart`/`GarbageCollectionFinish` callbacks — the same callback this +codebase already uses to call `_heap_usage_func` (`vmStructs.cpp`) — instead of each pass +paying for its own safepoint. + +**Correction to an earlier framing in this doc**: a pass does not run "inside a dedicated +`VM_Operation::doit()`" that the profiler constructs — HotSpot's `VM_Operation`/ +`VMThread::execute()` machinery is internal, unexported C++ with no agent-facing entry +point; nothing outside HotSpot can submit one. What actually happens, confirmed against +`src/hotspot/share/prims/jvmtiTagMap.cpp` and `jvmtiEnv.cpp`: +`SetTag`/`GetTag` need no safepoint at all — they take only a `MutexLocker` over a +JVM-internal "hot lock" on the tag map. `FollowReferences`/`IterateThroughHeap` **do** +bring the VM to a safepoint, but the JVM does this internally and transparently +(`VM_HeapWalkOperation`/`VM_HeapIterateOperation`, dispatched via +`VMThread::execute()` *inside* HotSpot's own implementation of those calls) the moment an +ordinary attached agent thread calls them — the calling thread simply blocks until the +walk finishes. A pass is therefore: an agent-owned, already-attached thread (the same kind +`LivenessTracker` already runs on, see `livenessTracker.cpp:303-409`) calling +`FollowReferences`/`IterateThroughHeap` directly; the safepoint is a side effect of that +call, not something the profiler builds or schedules. + +**Confirmed the VM is genuinely at a safepoint (all mutators stopped) for the full +duration of both callbacks**, on every collector: + +- JVMTI spec: *"This event is sent while the VM is still stopped... the event handler + must not use JNI functions and must not use JVM TI functions except those which + specifically allow such use (see the raw monitor, memory management, and environment + local storage functions)."* +- openjdk/jdk source: delivery is synchronous on the VMThread + (`src/hotspot/share/prims/jvmtiExport.cpp:2752-2790`, comment *"this event is posted + from VM-Thread"*); every call site is inside a safepoint-executing `VM_Operation::doit()`, + backed by explicit asserts — e.g. Parallel GC's + `assert(SafepointSynchronize::is_at_safepoint())` (`gc/parallel/psScavenge.cpp:305-306`), + G1's `assert_at_safepoint_on_vm_thread()` (`gc/g1/g1VMOperations.cpp:141-157`), + Shenandoah and ZGC wrapping the same `SvcGCMarker` only inside their respective + `VM_Operation`/`VM_ZOperation::doit()` paths. Stable JDK 11 → mainline, across + Serial/Parallel/G1/Shenandoah/ZGC. + +**But this does not make the GC-triggered callback itself usable as the execution vehicle +for a pass.** The "functions which specifically allow such use" are exactly two: +`Allocate` and `Deallocate` (the entire **Memory Management** category). `SetTag`, +`GetTag`, `GetObjectsWithTags`, `FollowReferences`, and `IterateThroughHeap` are all in +the **Heap** category, which is *not* on that allowlist — calling any of them from inside +`GarbageCollectionStart`/`Finish` is exactly what the restriction forbids. The spec's own +prescribed escape hatch — notify a raw monitor from the callback, do the real work on a +separate agent thread — doesn't preserve "the pass rides the GC's own pause" property +either: the woken agent thread runs after the GC's collection pause has already ended +(mutators resumed), so calling `FollowReferences`/`IterateThroughHeap` there triggers a +**new**, separate safepoint of its own (per the corrected mechanism above) rather than +reusing the GC's. + +The only way to fold the tag/walk work into the GC's own STW window would be to bypass +the official JVMTI entry points and reach into HotSpot's internal `JvmtiTagMap` directly +via symbol-sniffing — reintroducing exactly the fragility class already rejected for +Approach C (unversioned internal C++ state, no stable ABI). Doing that here would undo +the reason C was rejected. + +**Conclusion: "no new marginal safepoints" is not achievable while staying within +official JVMTI usage.** Each pass still triggers its own safepoint — transparently, via +whichever agent thread calls `FollowReferences`/`IterateThroughHeap` for that pass, not +via anything the profiler schedules itself. The GC callbacks remain useful only as a +low-cost *signal* ("a GC just happened, a pass may be worth running soon") — not as the +execution vehicle for the pass itself. This does not change the core incremental design +(frontier persistence via JVMTI tags, self-pruning of dead branches, per-pass budget) — +it only removes the "zero marginal safepoints" claim from the cost/benefit case. The +design's actual value remains what it was framed as: trading one long pause for several +short, independently-triggered ones — a latency-distribution improvement, not a +total-STW reduction. + +### Termination and abandonment + +Because passes are spread across a mutating heap, a search that never reaches a root or +the hop cap could otherwise persist indefinitely, accumulating abandoned frontier state +across GC cycles. Required cutoffs: + +- Hop cap (as in Approach B's single-pass form). +- A hard cap on passes-per-search or wall-clock TTL from first observation (value TBD — + see Open Question 2). +- Explicit reporting of abandoned searches (no silent truncation) so this shows up as a + measurable "chain not found within budget" outcome rather than being indistinguishable + from "no chain exists." +- Tag release: on abandonment or completion, every JVMTI tag this search assigned to + frontier/`EdgeStore` objects (`SetTag(obj, 0)`) must be cleared before the search's + state is discarded. Without this, an abandoned search leaves its tags in place + indefinitely, directly aggravating the tag-table sizing/contention risk raised in + Open Question 4. +- A hard cap on frontier size (record count or native-memory footprint). The hop cap and + pass/TTL cap bound how *long* a search runs, but not how *wide* the frontier can grow + within that time — a wide fan-out graph could accumulate an unbounded number of + `(tag, parent_tag, referrer_klass, depth)` records before either cutoff is hit. When + the cap is reached, stop admitting new frontier entries for that search and report it + as an abandoned/truncated search (value TBD — see Open Question 2). + +### Correctness note: chains are historical, not a single consistent snapshot + +A chain built across multiple passes stitches together `"A referenced B"` facts observed +at different points in time, not one frozen graph. For the stated purpose — explaining, +by referrer type, what typically retains this class of surviving object — this is +sufficient, and is not meaningfully weaker than a single-pass walk: GC roots (e.g. thread +stack frames) are themselves a live-changing set across a single pause's boundary, so +"one true snapshot" is already an approximation in the single-pass case. Any +documentation or output surface built on this must describe results as an **observed** +retaining path, not a claim about the object's current exact retention state. + +### Cost/benefit summary + +- **Does not reduce total STW time.** Each safepoint/callback entry pays fixed + synchronization overhead; K short increments likely sum to equal or *more* aggregate + pause time than one contiguous walk covering the same work. +- **Improves latency distribution.** No single long tail pause — the thing most likely to + actually affect deployed application health (p99 latency, heartbeat timeouts), even + when total accumulated pause-ms is flat or slightly worse. + +## Non-goals + +- Exhaustive paths to all GC roots. +- Field-level or object-identity-level chains (referrer *type* only). +- Any GC-internal-closure hook (Approach C) or internal parallel-iteration API use as a + launch dependency. + +## Open questions before implementation + +1. ~~Confirm `GarbageCollectionStart`/`GarbageCollectionFinish` callback timing relative to + safepoint release.~~ **Resolved** (see Triggering section): the callback is genuinely + at a safepoint, but the JVMTI Heap-category functions needed to do frontier work + (`SetTag`/`GetTag`/`FollowReferences`/`IterateThroughHeap`) are not in the callback's + allowed function set. Each pass instead triggers its own safepoint transparently, via + whichever agent thread calls `FollowReferences`/`IterateThroughHeap` for that pass — + the profiler never constructs a `VM_Operation` itself (see correction in Triggering + section). The "no new marginal safepoints" framing is dropped; the design's value is + latency distribution, not total-STW reduction. +2. Choose per-pass budget defaults (edge count vs. time slice), hop cap, the + passes-per-search/wall-clock TTL abandonment cutoff, and the frontier-size/memory cap + (see Termination and "Frontier metadata storage") — needs measurement against + representative heap shapes and per-hop fan-out, not a guess; `LivenessTracker`'s + flat-sample-rate sizing formula does not transfer to a graph-search frontier. + **Not resolved — provisional defaults only, no measurement has occurred.** The + implementation currently ships explicitly-labeled "provisional default pending Phase 5 + empirical tuning" constants (`arguments.h`: `DEFAULT_REFERENCE_CHAINS_HOP_CAP = 200`, + citing this doc's own JFR ~200-hop/100-100 precedent; `DEFAULT_REFERENCE_CHAINS_BUDGET + = 1000`; `DEFAULT_REFERENCE_CHAINS_TTL_MS = 60000`; `DEFAULT_REFERENCE_CHAINS_FRONTIER_CAP + = 65536`, sized as a fraction of `LivenessTracker::MAX_TRACKING_TABLE_SIZE` rather than + derived from any BFS-specific measurement; plus `referenceChains.h`'s + `FrontierTable::INITIAL_TABLE_CAPACITY = 1024` and + `ReferenceChainTracker::PASS_CADENCE_NS` = 1 s). These let the subsystem run and be + tested end-to-end, but none are backed by a benchmark against this codebase — do not + describe them as measured. The real resolution path is + `LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed), + which specifies the JMH/async-profiler matrix and decision rule Phase 5 still needs to + execute; this question stays open until that plan is actually run. + + **Pause-time-SLO feedback loop — SHIPPED, reusing the existing `PidController`.** + Implemented in + `LiveHeapReferenceChains-RemainingWorkPlan.md` (kept locally, not committed)'s + Phase D (`ReferenceChainTracker::updatePacing()`, `referenceChains.cpp`). This does not + replace the hop/TTL/frontier-cap constants raised in the first half of this question — only + the per-pass edge-count budget and the pass cadence, per the shipped mechanism below. + - New config sub-option `referencechains=...:pausetarget=` (`arguments.cpp`'s + `CASE("referencechains")` parser, field `Arguments::_reference_chains_pause_target_ms`), + defaulting to `arguments.h`'s `DEFAULT_REFERENCE_CHAINS_PAUSE_TARGET_MS = 5` — explicitly + labeled provisional/un-benchmarked, the same way the other + `DEFAULT_REFERENCE_CHAINS_*` constants are. Choosing its real value is still a Phase-5-style + empirical question, not resolved by this mechanism landing. + - `ReferenceChainTracker::start()` (re)constructs its own `PidController` instance + (`_pause_pid`) targeting `_pause_target_ms`, with its own gain triple — + **not** `ObjectSampler`/`MallocTracer`'s shared, uncited P=31/I=511/D=3/cutoff=15s + (`NativeSocketSampler` uses its own `RateLimiter`, not this `PidController` triple, and + was never part of the shared instance). The caveat this question raised about that triple + being copy-pasted, not independently derived, still stands for those two; it was not + "resolved," just not repeated here. The new instance uses P=10/I=1/D=2/window=1/cutoff=5s + — smaller proportional gain than the shared triple because a pass-duration-ms error is + single/low-double-digit in magnitude, unlike the shared triple's event-count scale + (`referenceChains.cpp`'s `start()`, inline comment on each gain). Gain *convergence* is + verified by gtest (three `ReferenceChainsTest` cases: steady-state at the ceiling, over- + ceiling, under-ceiling — see Phase D's exit criteria below), not by a live benchmark + against representative heap shapes; that remains a + `LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed) item, + not fully closed by this mechanism landing. + - Measurement point: `runPass()` (`referenceChains.cpp`) times its own root + `IterateOverReachableObjects` call (first pass) or `expandFrontier()`'s + `GetObjectsWithTags`+`FollowReferences` pair (resumed pass) — already the thread blocked + inside the safepoint those calls trigger (Triggering section) — and converts to whole + milliseconds before feeding `_pause_pid.compute()` (matching every other `PidController` + caller in this codebase, which all feed integer counts). + - `updatePacing(u64 pass_wall_ns)` folds the budget-scaling and cadence-widening/relaxing + decisions into that one `compute()` call per pass: the signal is added to + `_effective_budget` (the *value* `runPass()` now passes to `expandFrontier()` instead of + the fixed `_budget`) and clamped to `[MIN_EFFECTIVE_BUDGET = 50, _budget + _borrowed_budget]` + — a later addition lets the ceiling temporarily borrow above `_budget` (up to + `BORROW_CEILING_MULTIPLIER`x) rather than capping hard at the config value; see the + rotation-budget-starvation fix in git history for why. Whatever the clamp could not + absorb (`overflow`) drives `_effective_cadence_ns` (see Open + Question 5 below for why cadence is folded into this same output rather than a second + controller): `CADENCE_NS_PER_EDGE_OVERFLOW = 1ms/edge` scales the unabsorbed overflow into + a nanosecond adjustment, widening `_effective_cadence_ns` toward + `MAX_EFFECTIVE_CADENCE_NS = 4s` when still over-ceiling even at the budget floor, relaxing + it toward `MIN_EFFECTIVE_CADENCE_NS = 10ms` when comfortably under-ceiling even at the + budget ceiling. + - The hop cap and the frontier-size hard cap (Termination section) are untouched by this + mechanism — they stay fixed correctness/memory-safety bounds, not controller-tuned, exactly + as this question originally specified. + - One known, deliberate scope limit carried over from the plan: `buildAbandonedEvent()`'s + `datadog.ReferenceChainAbandoned` event still reports the static config ceiling `_budget`, + not the adaptive `_effective_budget` — changing that event's semantics was out of Phase D's + stated scope. +3. Decide the sample-batching policy: one incremental search per live-heap sample, or + batched multi-target BFS sharing a single frontier walk (batching amortizes better but + couples unrelated samples' termination conditions together). + **Shipped, but not in either form this question anticipated.** The implemented + `ReferenceChainTracker::runPass()` (referenceChains.cpp) does not target any sample at + all: it runs a single, singleton-owned search that walks the whole root-reachable graph + (bounded by the hop/budget/frontier caps) with no per-sample seeding. Reconstructing a + chain for a specific tag is a separate, read-only step (`buildChainEvent(target_tag, ...)`) + applied after (or during) that one shared search - closer in spirit to "batched" (one + frontier walk can answer for many targets) than "one search per sample", but arrived at + by omission (the target-sample feed did not exist at that time, see the implementation + plan's Phase 7 report) rather than a deliberate batching design. + Whether this generalizes to true multi-target batching (explicit seeding from multiple + samples, coordinated termination) is still open and deferred, consistent with this + question's original framing. + + **Target-selection policy — SHIPPED (positive population-slope ranking).** Implemented in + `LiveHeapReferenceChains-RemainingWorkPlan.md` (kept locally, not committed)'s + Phases A-C. The missing piece above was *which* tag(s) `buildChainEvent()` should + reconstruct for. As shipped: per klass, `LivenessTracker` tracks a rolling window of its + live tracked-instance population count, sampled once per `LivenessTracker::cleanup_table()` + epoch advance (the same GC-epoch cadence that already recomputes survivor status, + `livenessTracker.cpp` — a *different*, slower cadence than `ReferenceChainTracker`'s own BFS pass cadence, Open + Question 5; "past N passes" below means GC epochs observed by `LivenessTracker`, not BFS + passes). A klass whose population trend over that window is positive — new instances + arriving faster than old ones are dying — is a leak candidate; a bounded cache/pool also + holds old objects but its population stabilizes or shrinks. Of all klasses with a + positive trend, seed only the top 3–5 by trend magnitude for the next BFS pass — this + doubles as the per-pass seeding cap this question's last bullet asks for, so no separate + budget constant is needed. + + Mechanics (as shipped): + - `LivenessTracker` resolves the class lazily at JFR-flush time (`flush_table()`) to keep + the allocation-sampling path free of a `GetObjectClass` call. Per-klass population is + computed by resolving the klass once per *surviving* entry inside the existing + `cleanup_table()` epoch-advance pass instead (`resolveKlassId()`, the same + `GetObjectClass` + `Class.getName()` + `Profiler::lookupClass()` sequence + `flush_table()` uses) — cost scales with live-table size × GC frequency, not with + allocation rate, so it does not touch the hot sampling path. This whole step is gated on + `_gc_generations` (`cleanup_table()`), so a plain liveness session pays none of it. + - A fixed-capacity table (`KlassPopulationEntry _klass_population[]`, + `MAX_KLASS_POPULATION_ENTRIES = 256`) keyed by klass `StringDictionary` id holds a + `KLASS_POPULATION_RING_SIZE = 30`-slot ring of per-epoch population counts plus one + `jweak` of a currently-live representative instance (minted fresh in + `foldKlassCountsLocked()`, deliberately *not* aliasing the source `TrackingEntry::ref`, + which `cleanup_table()` can reap out from under it). When full, the + least-recently-updated entry is evicted (`recordKlassPopulationSampleLocked()`), the + same fixed-capacity/LRU shape every other table in this design uses. Counts are + accumulated into a reused scratch array (`accumulateKlassCount()`) during the survivor + loop, then folded into the ring at the end of the pass. + - Trend/slope: computed by the shared `ringThirdsStats()` helper (`livenessTracker.cpp`) — + average (and minimum) of the earliest third of the filled window vs. the most recent third + (cheap, allocation-free, avoids full least-squares regression). Trend is trusted only once + the ring reaches a minimum fill (`KLASS_POPULATION_MIN_FILL_FOR_TREND = 10` samples) to + avoid noise right after a klass starts being tracked. A klass only qualifies as a leak + candidate once its growth (`hasQualifyingGrowth()`, which also checks a floor-rise bar to + reject oscillations whose peak passes a magnitude test but whose baseline never rises) has + held for `consecutive_positive` consecutive epochs at or above a hysteresis threshold — + `LEAK_TREND_HYSTERESIS_BASE = 5`, lowered to `LEAK_TREND_HYSTERESIS_CORROBORATED = 3` when + the aggregate post-GC heap floor is itself rising (`heapFloorRising()`, fed by a + lock-free, single-writer `_heap_floor_ring`). This closes the false-positive gap a + single-epoch positive-slope test had: a see-saw/oscillating population could trigger a + search without any real longer-term growth. The exact window size (up to 30) and the + "top 3–5" cutoff are starting points, not measured values — a separate tuning pass, not + folded into Open Question 2's pause-time work (this is a leak-detection sensitivity + tradeoff, not a safepoint-cost tradeoff). See + [reference-chains-collection-summary.md](../reference-chains-collection-summary.md) for + the full mechanism. + - `LivenessTracker::selectLeakCandidates(KlassCandidate *out, int max)` returns, on + demand under a shared lock, the positive-slope klasses ranked by magnitude descending, + capped at `min(max, MAX_LEAK_CANDIDATES = 5)` — this top-N cutoff *is* the per-pass + seeding cap, so no separate budget constant is needed. Each `KlassCandidate` carries the + klass id and its representative `jweak`. + - Known limitation, stated rather than solved: if population trends positive across + *many* klasses simultaneously, that's more likely heap-wide growth (warm-up, load + increase) than several independent leaks. The top-3–5 ranking limits how many candidates + get chased, but does not distinguish this case from true multi-leak; a future refinement. + - The leak *judgment* is retrospective (needs a full window of GC-epoch history to see the + trend) but the *reconstruction target* does not need to be the exact instance that built + up the trend — any currently-live tracked instance of the flagged klass is evidence of + the same leak. **The bridging step is a READ, not a `SetTag` write** (a correction to + this doc's original proposal, found while grounding + `LiveHeapReferenceChains-RemainingWorkPlan.md` (kept locally, not committed); + see its "Correction to the design doc's Open Question 3 mechanism"). Pre-`SetTag`ing a + candidate before the forward walk reached it would make `heapReferenceCallback()`'s + `*tag_ptr == 0` branch — the *only* branch that records `parent_tag`/`depth` — skip it, + yielding an empty/root chain. Instead `ReferenceChainTracker::pollWatchedTargets()` + resolves each candidate's `jweak`, and if `runPass()`'s whole-graph walk has already + tagged it (`getTag() > 0`, a pure read), calls `buildChainEvent(tag, ...)` and emits the + chain. A candidate still at tag 0 is retried on a later poll, since the whole-graph walk + eventually visits every root-reachable object (barring the hop/budget/frontier caps). No + new backward-walk primitive is needed; this reuses what the frontier table already + records. + - This couples two independently-flagged, independently-scheduled subsystems + (`referencechains=...` vs. `_record_liveness`/`_gc_generations`) that had no existing + relationship — `LivenessTracker` identifies objects via `jweak` and never calls JVMTI + `SetTag`/`GetTag`, while `ReferenceChainTracker` identifies objects purely via JVMTI tags + it assigns during its own traversal. `pollWatchedTargets()` bridging them is the one new + piece of machinery this adds; everything else reuses existing structures. + - **Resolved:** `referencechains=...` gets this target-seeding behavior only when liveness + tracking *and* `_gc_generations` are both enabled (`LivenessTracker::gcGenerationsEnabled()`, + checked in `pollWatchedTargets()`); otherwise it falls back to the whole-graph-only + behavior (no target seeding), which is the doc's originally-stated fallback. +4. ~~Decide whether the frontier should use JVMTI object tags at all, or adopt + `LivenessTracker`'s weak-ref + index-table pattern instead.~~ **Resolved** (see + "Frontier metadata storage"): tags stay, because `FollowReferences`/ + `IterateThroughHeap` need tagged objects to filter/report frontier membership and a + weak-ref table has no hook into that machinery — but the per-tag *metadata* + (`parent_tag`, `referrer_klass`, `depth`) should be stored in a + `LivenessTracker`-style slot table (tag value as index) rather than a bespoke + structure, reusing its proven `SpinLock`/CAS-allocation/resize code. Remaining open + item: the table-sizing formula, folded into Open Question 2. +5. Decide the actual pass-scheduling policy now that GC callbacks can only be a signal, + not a vehicle: e.g. a background agent thread woken by the GC-callback signal that + then calls `FollowReferences`/`IterateThroughHeap` for the next pass (paying its own + transparent safepoint), vs. a fixed-cadence timer independent of GC activity. Needs a + cost model for how many such safepoints per second are acceptable before this stops + being "more palatable" than one larger pause. + **A decision shipped, but not the cost-modeled one this question asks for.** + `ReferenceChainTracker::shouldRunPass()` (referenceChains.cpp) combines both candidates + rather than choosing between them: it triggers a pass when the GC-finish epoch has + advanced since the last pass, *or* a fixed `PASS_CADENCE_NS` (1 second, explicitly + labeled provisional in `referenceChains.h`) has elapsed, whichever comes first. No + safepoints-per-second/per-pause-duration measurement backs the 1-second cadence value - + it was chosen only so an idle search still makes progress without polling tightly. The + cost model this question actually asks for is still open, deferred to Phase 5's + benchmark plan (`LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed)), + which has not been run. + + **SHIPPED — folded into Open Question 2's pause-time-SLO feedback loop, not solved + separately.** Implemented in the same `ReferenceChainTracker::updatePacing()` + (`referenceChains.cpp`, Phase D) described under Open Question 2: one `PidController` + `compute()` call per pass drives both that question's budget adjustment and this question's + cadence adjustment from the single measured per-pass safepoint duration, rather than two + independently-tuned mechanisms. `shouldRunPass()` and `threadLoop()` now compare against + `_effective_cadence_ns` in place of the fixed `PASS_CADENCE_NS` constant (which survives + only as `_effective_cadence_ns`'s starting value in `start()` and as the unit + `MAX_EFFECTIVE_CADENCE_NS` scales from); `threadLoop()`'s own sleep between iterations uses + `_effective_cadence_ns` too; so a controller-driven relaxed cadence actually shortens how + long an idle, no-GC-event search waits between passes, not just what the comparison in + `shouldRunPass()` reads. The GC-finish-epoch trigger in `shouldRunPass()` remains unconditional + on cadence, exactly as before — cadence only governs the fixed-interval fallback for an idle + search, per this question's original framing. See Open Question 2 above for the concrete + clamp/overflow mechanics (`MIN_EFFECTIVE_CADENCE_NS = 10ms`, `MAX_EFFECTIVE_CADENCE_NS = 4s`, + `CADENCE_NS_PER_EDGE_OVERFLOW`) and the gain-tuning caveat, which applies identically here + since it is the same controller instance. The cost-modeled "how many safepoints/sec is + acceptable" question this Open Question originally asked for is answered structurally (the + controller widens cadence exactly when passes are running long relative to the configured + ceiling) rather than by a specific measured number — that number is still a + `LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed) item. diff --git a/doc/architecture/ReferenceChains-SignalsExplained.md b/doc/architecture/ReferenceChains-SignalsExplained.md new file mode 100644 index 0000000000..d45657e469 --- /dev/null +++ b/doc/architecture/ReferenceChains-SignalsExplained.md @@ -0,0 +1,366 @@ +# How reference-chain hunting decides when to run and when to back off + +*A guided tour of the signals in `ReferenceChainTracker` (`referenceChains.cpp`/`.h`), for +readers with no prior context on this subsystem.* + +## 1. What problem this is solving + +Java heaps leak. When they do, the useful question isn't "how big is the heap" — it's +*"what is holding onto these objects and refusing to let go?"* Answering that means +walking live references backwards from a suspect object to a GC root, i.e. reconstructing +a **reference chain**. + +The obstacle: walking the heap graph (JVMTI's `FollowReferences`/`IterateThroughHeap`) +requires the JVM to stop every thread at a safepoint — a Stop-The-World (STW) pause, the +same kind a GC pause is. A profiler that stops the world to investigate a leak is +trading one problem for another. So the whole design of this subsystem is really an +answer to one question: + +> **How do we get a useful heap walk without stopping the world for longer, or more +> often, than the leak justifies?** + +Everything below is the machinery that answers that question — split into two halves: +*when should the next slice of walking happen* (triggering), and *how big/frequent +should that slice be allowed to get* (pacing/pausing). + +## 2. The core trick: one long pause becomes many short ones + +A single "walk the whole reachable heap and find the chain" pass could take seconds on a +large heap — an unacceptable pause. Instead, this subsystem does **bounded, resumable +BFS**: each *pass* only visits a limited number of objects (a *budget*), remembers where +it left off using JVMTI object tags, and picks the walk back up on the next pass. So a +"search" for a leak's chain is really a sequence of many short passes, each its own small +safepoint, spread out over time. + +This reframes the whole design problem from "avoid the pause" (impossible — see §3) to +"decide, pass by pass, whether *now* is a good time to spend one of these small pauses, +and how big it should be." + +## 3. Why GC callbacks can only ever be a *signal*, never the *work* + +The natural instinct: "GC just ran, the heap just changed — hook the GC callback and do +the walk right there, for free, since the VM is already stopped." + +This doesn't work, and the reason is worth understanding because it shapes the rest of +the design. JVMTI's spec is explicit: inside `GarbageCollectionStart`/ +`GarbageCollectionFinish`, an agent may call only the **Memory Management** category +(`Allocate`/`Deallocate`). Everything a heap walk needs — `SetTag`, `GetTag`, +`GetObjectsWithTags`, `FollowReferences`, `IterateThroughHeap` — is in the **Heap** +category, which is explicitly *not* on that allow-list. Calling any of them from inside +the GC callback is exactly what the restriction forbids (confirmed against +`GCCallbackGuard` in `referenceChains.cpp`, which asserts this in debug builds). + +So the GC callback can do exactly one cheap, legal thing: bump an atomic counter. + +```cpp +void ReferenceChainTracker::onGCFinish() { + GCCallbackGuard guard; // "we are inside the forbidden window" + atomicIncRelaxed(_gc_finish_epoch, (u64)1); +} +``` + +That's it. No walk, no tag calls, nothing heap-related — just "a GC finished, epoch N". +This is the first, and most important, idea to internalize: + +> **A GC callback is a doorbell, not a worker.** It tells a separate thread "something +> happened, go check if it's worth acting on" — it never does the acting itself. + +The actual walk happens later, on a dedicated background thread, deliberately outside +the safepoint the GC callback fired inside. That thread calling `FollowReferences` +triggers its *own*, independent safepoint — the GC's pause and the walk's pause are two +separate STW events, not one shared one. (An earlier version of this design hoped to +"ride" the GC's own pause for free; that turned out to be architecturally impossible +without reaching into unversioned HotSpot internals — see +`doc/architecture/LiveHeapReferenceChains.md`'s Triggering section for the full +investigation.) + +## 4. The scheduling loop: cadence + epoch, not "run on every GC" + +The background thread (`threadLoop()`) wakes roughly once a second and asks +`shouldRunPass()`: "should I spend a pass right now?" Two independent signals feed that +decision: + +1. **The GC-finish epoch changed** since the last pass. A GC just happened — the heap + graph likely moved, so a fresh pass is probably worth its cost. +2. **A fixed cadence has elapsed** since the last pass, even with no new GC. Two distinct + roles, easy to conflate: + - For a **workload with no/rare GCs** it is the fallback the naive reader expects — + without it, the search would stall forever waiting for a signal that never comes. + - For an **in-progress search** it is the *crawl's pacing knob*, not a signal re-check: + passes between GCs are the only thing that drains the search's own frontier backlog + (its "found but not yet expanded" objects). GC time does not advance the crawl — + pass time does — so a RUNNING search legitimately runs cadence-driven passes with + zero new signal, and the pause-time controller (§7) widens/narrows exactly this + cadence. The only genuinely stale re-check is the terminal state's cheap + restart-gate re-evaluation (§5), two atomic loads, deliberately kept unconditional. + +```cpp +u64 gc_finish_epoch = gcFinishEpoch(); +if (gc_finish_epoch != _last_pass_gc_finish_epoch) { + return true; // "a GC just happened, a pass may be worth running soon" +} +... +return cadence_elapsed; +``` + +Notice what's deliberately *not* here: the loop does **not** wake up early on every GC. +`onGCFinish()` only bumps a counter — it never calls `pthread_kill()` to interrupt the +sleeping thread (the only early-wake signal is shutdown's abort, §10). Why swallow the +up-to-~1s latency instead of reacting instantly? Because under a GC-heavy workload, +waking on *every* GC would collapse the loop's cadence down to GC frequency — each wake +is a full iteration of scheduling logic, not free. The design accepts "at most ~1s of +extra latency" in exchange for not turning a GC storm into a scheduling storm. This is +a recurring theme in this subsystem: **every signal is deliberately made cheap to check +and expensive to over-react to.** + +One caveat the epoch trigger carries: it is an *unconditional* bypass — while it is +set, no cadence check applies. Minor young GCs bump the epoch as readily as majors, so +once the adaptive cadence (§7) has shrunk to its floor, a workload that GCs more often +than the wake interval effectively drives one pass per wake, at up to GC frequency. +That is bounded and acceptable for the crawl lanes (each pass is pause-budgeted by the +PID controller); the one lane that needs a rate bound of its own — the canary chase — +gets one explicitly (§8). + +## 5. Not every wake actually walks: the leak-signal gate + +Waking up and checking cheap counters is fine to do often. Actually walking the heap +costs real STW time, so before a **brand-new search** (or a **restart** of one that +finished) is allowed to begin, a second, independent gate applies: +`LivenessTracker`'s population-trend signal — "is there currently a class whose live +object count looks like it's growing without bound?" (`hasLeakSignal()`). + +```cpp +bool ReferenceChainTracker::hasLeakSignal() { + if (!LivenessTracker::instance()->gcGenerationsEnabled()) { + return true; // no trend signal available at all -> don't gate on it + } + ... + int n = LivenessTracker::instance()->selectLeakCandidates(probe, 1); + return n > 0; +} +``` + +This matters because a full-heap BFS is expensive relative to a targeted one, and there's +no point paying that cost speculatively, with nothing to justify it. If the leak-tracking +feature isn't even enabled, this reduces to "always true" — the walk runs unconditionally, +exactly as a simpler standalone version of this feature would. + +Once a *search is already running*, though, this leak-signal gate is deliberately **not** +re-applied per pass — an in-progress search's own frontier (its list of "found but not yet +expanded" objects) is allowed to keep converging pass after pass regardless of whether a +fresh leak candidate happens to be visible right now. Gating an already-running search on +"is there a candidate this instant" would stall its progress for no good reason; the gate's +job is to decide whether *starting* new expensive work is worth it, not to second-guess +work already committed to. + +The same "a potential leak is currently detected" signal also raises *liveness tracking +fidelity* itself: once candidate selection has fired, the qualifying tids' allocations are +admitted to the tracking table at 100% instead of the configured live-samples ratio +(default 10% — a 90% probabilistic drop that thins small per-(klass, tid) populations; +`LivenessTracker::admitForTracking()`). The raise is bounded by the candidate threads' +own allocation rate, not the process's whole allocation rate, so the tracking table's +cost scales with the leak's own threads; and it is refreshed — and cleared — poll by poll +with the candidate selection, so it never outlives the chase. The OOM urgency ramp +(§9) raises admission to 100% for *all* allocations for the same reason it drops every +pacing rule: the process is expected to die soon, and maximizing what the last chapter +captures outweighs the tracking table's transient volume. + +## 6. Paying for it: the "pain budget" leaky bucket + +Even with a real leak signal, restarting a full search back-to-back forever would be +its own kind of runaway cost. The safety valve here is a **leaky bucket over cost**, not +over time — `PainBudget` (`painBudget.h`): + +```cpp +// spend(): "that last search cost N milliseconds of wall-clock work" +// canStartNow(): "has that debt drained back to ~0 yet, at refill_rate?" +``` + +The intuition: if a search finished *cheaply*, it can restart again almost immediately. +If it was *expensive*, the next restart has to wait proportionally longer. This is a much +better model than a fixed cooldown timer, because "how expensive was the last search" +is exactly the thing worth reacting to — a fixed cooldown would either be too +conservative after a cheap search or too permissive after an expensive one. + +`canAffordNewSearch()` combines both gates from §5–6: the leak signal has to say "worth +it" *and* the pain budget has to say "affordable" before a new/restarted search is +allowed to begin. + +## 7. Self-tuning the pause itself: a PID controller on pause time + +So far: *when* to run a pass. Now: *how expensive should that pass be allowed to get?* + +Each pass has a target STW duration (`_pause_target_ms` — an operator-configured ceiling, +e.g. "no single pass should take more than N ms"). After every pass, the tracker measures +how long it actually took and feeds that into a small PID controller +(`_pause_pid.compute()`), which nudges the *per-pass budget* — how many objects the next +pass is allowed to visit — up or down: + +- Pass ran comfortably under target → budget can grow a bit (there's headroom). +- Pass ran over target → budget shrinks, so the *next* pass is smaller and faster. + +This is the same self-correcting idea used elsewhere in this profiler for sampling rates +(`ObjectSampler`, `MallocTracer`) — measure the actual cost, compare to a target, adjust +the knob that controls the next iteration's cost, repeat. It means the operator doesn't +have to hand-tune a "safe" fixed budget for every heap size and object graph shape; the +controller finds it empirically, pass by pass. + +There's a second knob the same controller feeds: if the budget is already at its floor +and *still* over target, that's a sign the real problem isn't "how much work per pass" — +it's "passes are happening too close together." In that case the controller widens the +*cadence* (the sleep interval between passes) instead. Conversely, if a pass finishes +comfortably under target even at the configured budget ceiling, the idle cadence is +shortened — there's slack to use it to converge faster. Either way, the same measured +signal (pass duration vs. target) drives both "how much work per pass" and "how often to +even try." + +There's also a small "savings account" on top of this (budget-borrowing): a *sustained* +run of comfortably-under-target passes slowly raises the ceiling itself, not just the +budget inside it — but a single pass that isn't comfortably under target revokes that +extra headroom immediately. The asymmetry is deliberate: earning slack should take +sustained good behavior; losing it should be instant, so a run of easy passes can never +turn into an excuse for one expensive one. + +## 8. When "wait for the next cadence tick" isn't good enough: canary mode + +The scheduling described in §4–7 assumes a slow, whole-heap background search. But +sometimes there's a much sharper signal available: `LivenessTracker` has already flagged +*specific* suspect objects (a small, pre-tagged "canary" set) worth confirming quickly. In +that mode: + +```cpp +bool canary_active = _candidate_count > 0 && + __builtin_popcountll(_candidate_found_bits) < (u64)_candidate_count; +if (canary_active) { + return true; // run the next pass immediately, no cadence wait +} +``` + +While a canary search is active, cadence is bypassed — passes run back-to-back *while the +chase is fresh or making candidate progress*, because the PID controller (§7) is already +keeping each individual pass's pause small, and a chase that is genuinely close to its +target should resolve in a handful of passes. + +Back-to-back-forever, however, is exactly the wrong promise for the failure mode this +feature actually meets in production: a candidate the crawl cannot reach soon (a leak +holder buried behind a deep frontier backlog — the coverage lottery inherent to an +external JVMTI agent with no reverse-edge primitive). Measured live on a production-like +analyzer pod: one such un-findable candidate held the chase open for 32 minutes at ~88 +passes/min — a full core of engine work — because the bypass applied unconditionally and +the loop skips its sleep whenever a pass will run. "Well under 20ms per 60s recording" +is only ever true for *findable* candidates. + +So the canary lane carries its own rate bound — a **progress-driven, work-scaled +exponential backoff**. The inter-pass spacing is a *multiple of the measured cost of a +pass itself* (an EMA of each pass's wall duration), not a fixed wall-clock constant: +- Every pass that makes candidate progress (a candidate found, or a new candidate + admitted) resets the spacing multiplier to 1 — back-to-back. At multiplier 1 the + next pass starts as soon as the last one ended, which is harmless by construction + for a cheap pass, and exactly the fast-resolution burst a chase that is genuinely + close should get. +- Every pass with *no* candidate progress doubles the multiplier, capped at 16. A + permanently stuck chase therefore settles at one pass per 16 × (its own cost) — + it keeps ticking indefinitely (abandonment is a separate, frontier-aware detector's + job, below), but its steady burn is structurally bounded to ~1/16 of a core on pass + work, *whatever that work is*. +- The work-scaling is the point: a fixed cap only binds when it exceeds the pass's own + duration — the production pod's passes ran 0.7–4s, so a 1s cap would have changed + nothing at all (the loop is work-bound, never sleep-bound, when the pass exceeds the + cap), while the same 1s cap starved a genuinely reachable deep chase whose passes + cost milliseconds. Scaling to the measured cost gives the expensive-pod chase a real + bound and the cheap-but-deep chase its density from the same law. +- The GC-epoch trigger deliberately does **not** bypass the backoff: a GC-heavy workload + bumps the epoch on virtually every wake, so letting GCs override the spacing would make + the backoff unreachable on exactly the deployments that burn the most. +- The OOM urgency ramp (§9) overrides it entirely — imminent OOM remains the one regime + where the chase burns budget back-to-back. + +The pain-budget refill rate is raised 100x while a chase is open, but note what that +means now: **not** a rate control (the backoff is the rate bound) — just a double-throttle +guard, so the conservative base refill rate tuned for the ordinary ~1 pass/s crawl doesn't +starve a chase the backoff has already paced. The earlier covering-vs-emergency refill +distinction existed to feed the unbounded back-to-back mode and is gone with it. + +## 9. The panic button: ramping up as OOM approaches + +All of the above optimizes for "acceptable background cost most of the time." But if +`LivenessTracker` projects the heap is genuinely on a collision course with +`OutOfMemoryError` within the next ~30 minutes (`secondsToOOM()`), "acceptable background +cost" is the wrong objective — the process might not survive long enough for a leisurely +search to finish. So there's a third mode: an **urgency ramp**. + +As projected time-to-OOM shrinks from 30 minutes toward zero, the pause-time target and +the pass cadence both ramp *exponentially* toward much more aggressive ceilings: + +```cpp +double x = 1.0 - seconds_to_oom / OOM_RAMP_START_S; // 0 at 30min out, 1 at OOM +target_ms = pause_target * pow(URGENT_PAUSE_TARGET_MS / pause_target, x); +cadence_ns = pow(URGENT_CADENCE_NS / PASS_CADENCE_NS, x) * PASS_CADENCE_NS; +``` + +The reasoning behind exponential (rather than linear) ramping: at 30 minutes out, the +situation still might resolve itself (a GC frees the suspect objects, the trend reverses) +— stay cheap. In the last seconds before OOM, the process is likely to die anyway, so it's +worth spending far more of the pause-time budget to collect a usable chain *before that +happens* than to protect a latency budget for a process that may not be there to benefit +from it. Held flat-out cheap the whole time, this urgency signal would arrive too late to +matter; held aggressive the whole time, it would waste budget on every one of the many +false alarms a rising trend that later reverses produces. + +Two more details make this practical rather than flappy: + +- **Hysteresis, not a bare threshold.** `isUrgent()` *latches* on when + time-to-OOM drops below a threshold, and only *releases* after several consecutive + observations comfortably clear of a separate (higher) release bar. A single noisy + reading crossing back and forth across one threshold would otherwise thrash the ramp + on and off every second. +- **One search per urgency episode.** Once an urgent episode has spent its one + authorized search, further ticks within the same episode don't keep tearing down and + restarting it from scratch — the per-candidate probe (§5) remains the only trigger + until the episode actually clears. + +## 10. Stopping mid-pass: the abort path + +Everything above is about *starting* passes thoughtfully. There's also a clean way to +*stop* one that's already in flight — needed when the profiler itself is shutting down +(or a test needs to reset state) while a `FollowReferences` call is still blocked inside +the JVM. + +```cpp +_abort_pass_requested.store(true, std::memory_order_relaxed); +pthread_kill(_thread, WAKEUP_SIGNAL); // interrupts a sleeping thread promptly +``` + +The callback JVMTI invokes for each visited object checks this flag and returns an abort +code the moment it sees it set — since nothing outside the JVM can interrupt a call +already inside `FollowReferences`, the flag has to be checked *from inside* the callback +JVMTI itself is driving. `pthread_kill` with a no-op-handler signal only helps the *other* +common case — a thread parked in `OS::sleep()` between passes — wake up promptly instead +of waiting out the rest of its interval. + +## 11. Putting it together + +| Question | Signal | Where | +|---|---|---| +| Did the heap graph just change? | GC-finish epoch bump | `onGCFinish()` → `shouldRunPass()` | +| No GC signal — is it time anyway? | Fixed/adaptive cadence elapsed | `shouldRunPass()` | +| Is a *new* search worth starting at all? | LivenessTracker population trend | `hasLeakSignal()` | +| Can we afford to spend that cost right now? | Pain-budget leaky bucket | `canAffordNewSearch()` | +| How big/frequent should passes be, steady-state? | PID controller on measured pause time | `updatePacing()` | +| Are we chasing specific known suspects? | Canary candidate set + progress-driven backoff | `canary_active` bypass + work-scaled `_canary_backoff_mult` | +| Is OOM close enough to abandon caution? | secondsToOOM() latch/release | urgency ramp in `threadLoop()` | +| Need to stop a pass already in flight? | Abort flag + wakeup signal | `_abort_pass_requested` | + +The unifying idea across all eight mechanisms: **every trigger is a cheap check, and +every response is proportional to real, measured cost** — never a fixed guess. GC +callbacks stay legal by doing nothing but incrementing a counter. Whether to search at +all is gated on an independent leak-trend signal, not "because a GC happened." Whether a +search can *restart* is gated on how expensive it actually was last time, not a flat +cooldown. How big a pass gets is tuned from its own measured pause time, not a static +config value. And the one scenario where none of that caution applies — imminent OOM — is +its own explicitly separate, hysteretic escalation path, not a tweak to the steady-state +knobs. + +That's what makes several short, adaptively-sized pauses a genuinely better trade than +one long one: the *decision* of when to pay each of those small costs is never blind — +it's always backed by a signal that says this particular pause is likely to be worth it. diff --git a/doc/reference-chains-collection-summary.md b/doc/reference-chains-collection-summary.md new file mode 100644 index 0000000000..0354a8cbe7 --- /dev/null +++ b/doc/reference-chains-collection-summary.md @@ -0,0 +1,87 @@ +# Reference Chain Collection: Design Summary + +## Problem + +Given a JVM heap with objects suspected of leaking (e.g., klasses whose live population grows monotonically across GC generations), reconstruct a **referrer chain** from a GC root down to a representative instance of the suspect klass — without pausing the JVM for longer than a small, bounded budget, and without assuming the entire heap graph can be walked in one pass. + +Three constraints drive the design: + +1. **Detecting *which* klasses are worth walking** must be near-free and based on survivorship trend, not raw allocation volume. +2. **The walk itself** (JVMTI `FollowReferences`) can be arbitrarily expensive on a large heap, so it must be interruptible and resumable. +3. **Total STW/JVMTI-callback time per pass** must stay under a small budget so the profiler doesn't visibly perturb the target application. + +--- + +## Component 1: Surviving-Generation Signal (`LivenessTracker`) + +Rather than triggering a heap walk on every allocation or every GC, the tracker maintains a **per-klass population history** and only nominates a klass as a "leak candidate" once it shows a **sustained positive trend across GC generations** — i.e., its live (surviving) instance count keeps growing generation over generation, not just spiking transiently. + +**Mechanics:** + +- Population sampling is driven off the existing allocation-sampling hot path (`track()`), but the actual **per-klass counts are only folded into history at `cleanup_table()`'s GC-epoch-advance pass** — i.e., once per GC, not once per allocation. This keeps the hot path allocation-free and cheap. +- Each klass gets a small **ring buffer of recent per-epoch surviving counts** (`KLASS_POPULATION_RING_SIZE = 30` samples). A ring, not an unbounded history, because we only care about recent trend, not lifetime totals. +- A klass's trend is only trusted once its ring has a **minimum fill (`KLASS_POPULATION_MIN_FILL_FOR_TREND = 10` samples)** — avoids false-positive trend detection on a klass that's simply new to being tracked (too few points to fit a slope to). +- `selectLeakCandidates()` computes a slope over each ring and returns the **top-N klasses by slope magnitude** (`MAX_LEAK_CANDIDATES = 5`), each paired with a live representative instance (a `jweak`) discovered during sampling — this weak reference is what seeds the walk in Component 2. +- A klass only qualifies once its growth (`hasQualifyingGrowth()`) has held for `consecutive_positive` epochs at or above a **hysteresis threshold** — `LEAK_TREND_HYSTERESIS_BASE = 5` by default, lowered to `LEAK_TREND_HYSTERESIS_CORROBORATED = 3` when the aggregate post-GC heap floor is itself rising (`heapFloorRising()`, fed by a lock-free, single-writer `_heap_floor_ring` populated from `onGC()`). Because the aggregate heap-floor signal can't attribute growth to any one klass, it only ever raises or lowers the bar uniformly for the whole scan — it never reorders or singles out individual candidates. +- The whole table (`_klass_population`, up to `MAX_KLASS_POPULATION_ENTRIES = 256` entries) is a flat array scanned linearly — deliberately no index structure, since 256 entries is cheap to scan and this stays off the allocation hot path. +- Everything under this table (population array, size counter) is guarded by a single `SpinLock` (`_table_lock`) — the *same* lock `cleanup_table()` already holds for its epoch-advance pass, rather than adding a second lock. **Any code path that mutates this table (including test-only reset seams) must take that lock — mutating `_klass_population_size` or the array unguarded is a data race against the epoch-advance pass**, discovered in practice while hardening test seams. + +**Why this design:** it decouples "is this klass suspicious" (cheap, GC-cadence, statistical) from "reconstruct why it's suspicious" (expensive, JVMTI, on-demand) — the expensive walk only ever runs against klasses that have already earned a positive trend signal, not against every allocation site. + +--- + +## Component 2: Resumable Frontier Walk (`ReferenceChainTracker`) + +Once a klass is nominated, a **persistent background BFS thread** reconstructs a path from a GC root to a tagged instance of that klass, using JVMTI's `FollowReferences`/heap-tag mechanism — but broken into many small, budgeted passes rather than one unbounded walk. + +**Mechanics:** + +- The tracker is a **process-wide singleton** with its own thread (`threadLoop()`), woken on a fixed cadence (`effectiveCadenceNs`) rather than synchronously from allocation or GC callbacks — decouples walk progress from the rate of GC/allocation events. +- `runPass()` dispatches on `_search_started`: + - **First pass for a search**: enumerates heap roots via `IterateOverReachableObjects()` (`heapRootCallback()`/`stackRefCallback()`), tagging root-referenced objects as it goes. + - **Every subsequent pass**: calls `expandFrontier()`, which resumes from a **persisted frontier** (the previous pass's boundary tags) instead of re-walking from roots. This is the resumability mechanism: each pass advances the frontier outward by one bounded increment and stops. +- Each pass is capped by an **edge-admission budget** (`effectiveBudget`, e.g. `edges_admitted` capped at a configured value like 4000/200000/500 depending on test config) — `expandFrontier()`'s nested loops (`while (!ctx.truncated && progress)` outer, `for (jlong tag : candidate_tags)` inner) both check a truncation flag and bail out the moment the budget is exhausted, so a single pass's JVMTI-callback time is bounded regardless of heap size. +- **Cooperative abort**: an `std::atomic _abort_pass_requested` flag, checked inside `heapReferenceCallback()` (the JVMTI callback invoked per edge), lets `stopThread()` interrupt an **in-flight** walk promptly — set before `pthread_kill(WAKEUP_SIGNAL)`/`pthread_join()`, cleared by `startThread()`. Without this, a `FollowReferences` call already in progress at JVM shutdown or profiler restart can't be interrupted, and `pthread_join()` blocks indefinitely (a real, previously-diagnosed shutdown hang). +- Search state is a small state machine: `RUNNING → {ABANDONED | COMPLETED}`. `RUNNING` can **restart itself** (fresh root walk) once a candidate's chain is found and its tags released, gated by `canAffordNewSearch()`'s **pacing budget** — self-throttling, not unconditional: a search won't restart back-to-back if it would blow the perturbation budget. A terminal state (`ABANDONED`/`COMPLETED`) is not final: `shouldRunPass()` restarts the search from it via `restartSearch()` once the pain budget has drained and a leak indication is (still) present. +- `runPass()` only moves to `COMPLETED` once the frontier is fully drained **and** `_watched_leak_klass_count == 0` (no klass currently under active leak watch, Component 4). A fully-drained frontier while a klass is still watched leaves `_search_state` at `RUNNING` instead: the walk has visited every reachable object once, but a leak-shaped klass keeps growing by **mutating an already-visited container** (e.g. appending to a `static final` collection field long after the walk first admitted it), which a one-time visit can never observe again. Rotation (Component 4) is what re-observes those already-`EXPANDED` entries on later passes. +- A search is marked `ABANDONED` (with a reason code) if it runs out of frontier budget without completing — e.g. hitting a frontier-cap under a tiny configured budget. This is a deliberate, observable outcome, not a silent failure — surfaced so operators can distinguish "the walk gave up" from "the walk is still in progress." + +**Why this design:** treating the walk as a resumable state machine (persisted frontier + tags) rather than one atomic call means a heap graph of unbounded size never forces an unbounded pause — cost is amortized across many cheap passes, each individually bounded and individually abortable. + +--- + +## Component 3: Latency Budget Enforcement + +The system enforces its "don't perturb the app" guarantee at **three independent layers**, not just one: + +1. **Per-pass edge budget** (`effectiveBudget`) — caps JVMTI callback invocations per pass (Component 2). +2. **Pain budget** (`_pain_budget`/`_search_pain_ms`, spent via `_pain_budget.spend(...)`) — tracks cumulative walk cost against a wall-clock ceiling; used by `canAffordNewSearch()` to decide whether a new search/restart is affordable right now, not just whether the current pass fit its edge budget. This is what prevents "many cheap passes" from silently adding up to an expensive aggregate cost. +3. **Pass cadence** (`effectiveCadenceNs`) — the background thread only wakes and attempts a pass on a fixed cadence (plus GC-epoch-triggered wakeups), rather than continuously spinning, bounding CPU overhead between passes. + +Together these mean: a single pass is bounded (edge budget), a sequence of passes is bounded (pain budget), and idle overhead between passes is bounded (cadence) — the three layers target three different ways an unbounded-cost walk could otherwise leak into the target application's latency. + +--- + +## Component 4: Rediscovering Growth in an Already-Visited Container + +Once a klass is leak-flagged, `pollWatchedTargets()` refreshes `_watched_leak_klass_ids` (up to `MAX_WATCHED_LEAK_KLASSES = 5`, matching `LivenessTracker::MAX_LEAK_CANDIDATES`) from `LivenessTracker::topKlassesByGenerationCount()` — a faster, un-hysteresis-gated ranking than the `selectLeakCandidates()` canary set, but only consulted once `hasLeakSignal()` has already fired via that slower path. + +**Why a matching mechanism is needed at all:** the real leak shape this targets is a `static final` collection field that gets *appended to*, not reassigned — the container itself was already admitted and `EXPANDED` in an early pass, long before `selectLeakCandidates()`'s hysteresis authorized watching its element klass. New elements can only be rediscovered by re-expanding that already-visited container, not by discovering a brand-new root. + +**Mechanics:** + +- Matching a newly-admitted object against `_watched_leak_klass_ids` must use a class identity that stays valid for the object's whole lifetime, not `referrer_klass` — a classMap `StringDictionary` id that can differ for the same class at different times if that dictionary is compacted/regenerated. Both `ReferenceChainTracker` and `LivenessTracker` mint from a single shared, process-wide `ClassTagAllocator` (`classTagAllocator.h`) and store the resulting stable `class_tag` (`FrontierEntry::class_tag`, `KlassPopulationEntry::stable_class_tag`) instead. +- `trackLeakAccumulation()` runs on every successful admission (`admitObject()`'s `ADMITTED` result) and aggregates, per `(leaf_klass_id, parent_class_id)` signature, how many admitted children of a watched leaf klass were observed under a parent of that class (`_leak_signature_totals`, ranked by delta against the previous pass's snapshot — Tier 1), and per parent *object* tag, how many such children that specific parent holds (`_leak_parent_fanout` — Tier 2, ranked within the winning Tier-1 signature). +- `seedLeakAccumulationForNewlyWatchedKlass()` runs once, the moment a klass_id first enters `_watched_leak_klass_ids`: it scans the whole frontier table for already-`EXPANDED` entries whose `class_tag` matches, since a container that was fully admitted before its element klass started being watched would otherwise never get its first Tier-1/Tier-2 data point. +- `collectLeakAccumulationCandidatesForRotation()` re-queues the Tier-2 winner(s) for re-expansion (`LEAK_ACCUMULATION_ROTATION_BUDGET = 16` per pass) — this is what actually re-visits the growing container and picks up elements appended since its first expansion. + +**Two additional robustness fixes surfaced only under a real growing-collection repro, not by the unit suite alone:** + +- **Urgent-signal latch.** `isUrgent()` used to be a bare `secondsToOOM() < OOM_URGENT_THRESHOLD_S` comparison; that projection is derived from a short ring of heap deltas and can swing by orders of magnitude between consecutive observations of the same steadily-growing heap. A bare comparison flapped, and each flap back to "urgent" bypassed the per-klass hysteresis gate in `hasLeakSignal()` and restarted the search — which discards the frontier table and the Tier-1/Tier-2 accumulators above, so they never got the several passes they need to converge. `isUrgent()` now latches on first crossing and only releases after `URGENT_RELEASE_CONSECUTIVE` (5) consecutive observations at or above `OOM_URGENT_RELEASE_S` (2× the threshold); `_urgent_search_spent` limits each latched episode to authorizing one restart. +- **Classmap-generation sync at startup.** `LivenessTracker::initialize()` now seeds `_last_class_map_generation` from the real classMap generation instead of leaving it at the default `0`. Previously, the first `cleanup_table()` call after any profiler start saw `current_generation != 0`, treated it as a classMap reset, and wiped `_klass_population` — discarding any population history folded in between `initialize()` and that first `cleanup_table()` call. + +--- + +## Output Path + +Once a candidate's chain is fully reconstructed, `pollWatchedTargets()` builds a chain event (`buildChainEvent()`), which is enqueued (`enqueueChainEvent()`) and later drained (`drainPendingChainEvents()`, called from `Profiler::dump()`, not from the BFS scheduling thread) into `Profiler::writeReferenceChain()` — ultimately surfaced as a `datadog.ReferenceChain` JFR event on the next `Profiler::dump()`. This keeps the expensive walk and the (comparatively cheap, already-existing) JFR-write path decoupled — the walk never blocks on JFR I/O, and JFR writes never trigger a walk. diff --git a/doc/reference-chains-design.md b/doc/reference-chains-design.md new file mode 100644 index 0000000000..059a181189 --- /dev/null +++ b/doc/reference-chains-design.md @@ -0,0 +1,259 @@ +# Reference Chains for Surviving Live Heap Samples + +**Status:** Implemented (see `doc/reference-chains-collection-summary.md` for the as-built design) +**Date:** 2026-07-07 +**Jira:** TBD + +## Goal + +For a subset of live-heap samples that survive past their allocation window, produce +a **reference chain** — a sequence of referrer *types* (not full field-level paths, not +necessarily to *all* GC roots) connecting the sampled instance back to *a* GC root. This +is diagnostic information ("what kind of object chain is keeping this alive"), not a +heap-dump-grade exact retainer analysis. + +## Constraints + +- Must run cheaply, with as short a safepoint / STW contribution as possible. +- Must work on stock vendor JDKs the agent attaches to — no forked/patched JVM builds. +- Exhaustive (all-roots, full-path) chains are explicitly **not** required; referrer-type-only, + bounded-depth, best-effort chains are acceptable. + +## Approaches considered + +Three approaches were evaluated; two are ruled out as launch requirements for concrete, +evidence-backed reasons. One sub-idea (Approach C's `ParallelObjectIterator` variant) is +explicitly kept open as a conditional future option; see its discussion below. + +| # | Approach | Completeness | Complexity | Feasibility | Status | +|---|---|---|---|---|---| +| A | Full JVMTI `FollowReferences` reverse-graph walk, piggybacked on an already-scheduled major GC | 4/5 | 4/5 | 2/5 | Rejected | +| B | Bounded BFS-from-roots with frontier pruning (JFR "leak profiler" technique, adapted) | 3/5 | 3/5\* | 4/5 | **Chosen** | +| C | Hook GC mark/copy closures (G1, ZGC) to record parent pointers inline during marking | 2/5 | 5/5 | 1/5 | Rejected | + +\* This 3/5 reflects only the single-pass BFS sketched at selection time. The "Chosen +design" section below replaces that sketch with an incremental, resumable BFS +(JVMTI-tag-based frontier persistence across GC cycles, a dedicated `VM_Operation` per +pass, GC-callback signaling, and explicit termination/tag-cleanup bookkeeping), which is +materially more complex than this score suggests — closer to 4/5 in implementation and +maintenance effort. The score is left unchanged above (it documents the state of the +comparison at decision time) rather than retroactively edited. + +### A — Full reverse-reachability walk (rejected) + +Safepoint length scales with live-set size regardless of how the walk is triggered. +Modern regionalized collectors (G1, Shenandoah) rarely perform a true full-heap walk +during ordinary major GCs, so "ride an already-paid pause" is not a reliable amortization +strategy. Cost is fundamentally at odds with the "short safepoint" constraint. + +### B — Bounded BFS-from-roots (chosen) + +Mirrors OpenJDK's own `jdk.OldObjectSample` leak-profiler implementation +(`src/hotspot/share/jfr/leakprofiler/chains/{edgeStore,bfsClosure,dfsClosure}.cpp`): +a `VM_Operation`-driven BFS from GC roots, retaining only edges on the frontier toward a +small, fixed sample set, with a hard hop cap (HotSpot itself caps chains at ~200 hops, +split 100/100 from leaf and from root). We can go cheaper than JFR because only the +**referrer class**, not object identity or field name, is needed — the `EdgeStore` +degenerates to `(referrer_klass, parent_ref, depth)` records, where `parent_ref` links +each record back to the record that discovered it, enabling chain reconstruction. + +Adopting this pattern is a re-scoping of proven, shipping HotSpot code, not a novel +algorithm design. + +### C — GC mark/copy closure piggyback (rejected) + +Investigated specifically for G1 and ZGC on the premise that per-edge referrer +information is already available inside the collector's own marking/evacuation closures +(`G1ParCopyClosure::do_oop_work`, ZGC's `ZMarkConcurrentRootsIteratorClosure` / +load-barrier closures), so recording it would cost nothing beyond what the GC already +pays. + +Rejected because there is no stable, externally reachable hook into these closures: + +- They are internal, template-instantiated C++ classes compiled into `libjvm.so` at + HotSpot build time — not a registrable/pluggable extension point. +- This differs categorically from `VMStructs`-style introspection already used in this + codebase (`ddprof-lib/src/main/cpp/hotspot/vmStructs.cpp`), which reads VM state + passively via an officially exported offset table. Intercepting a GC closure's + *behavior* would require either shipping a patched OpenJDK build (a fork/maintenance + commitment far beyond anything in this codebase) or binary-patching unversioned, + per-build-mangled function addresses — not shippable across JDK point releases. + +A related idea — using HotSpot's internal `ParallelObjectIterator` +(landed via [JDK-8322043](https://www.mail-archive.com/serviceability-dev@openjdk.org/msg12977.html), +used by `VM_HeapDumper` to partition heap regions across GC worker threads for parallel +heap dumping) to shrink Approach B's safepoint by parallelizing the walk — was also +investigated. Same verdict: it is an internal C++ class, not exposed via JVMTI, with no +stable ABI for an attached agent to call. Symbol-sniffing internal HotSpot functions *is* +an established pattern in this codebase (`VMStructs::findHeapUsageFunc`, +`vmStructs.cpp:489-509`), but that precedent covers a single leaf virtual method with a +value/POD-ish return; `ParallelObjectIterator` is a multi-class subsystem that coordinates +the VM's own GC worker threads under safepoint control — an order of magnitude larger +fragility surface, with a much higher blast radius if a layout assumption is wrong (GC +worker-thread coordination corruption vs. a bad JMX stat). Not pursued as a launch +requirement; revisit only if Approach B's single-threaded pause proves to be a measured +bottleneck, and treat it as an isolated, heavily version/flag-gated fast path with +automatic fallback — never a dependency. + +## Chosen design: incremental, resumable bounded BFS + +A single-pass bounded BFS still means one pause sized to whatever budget is configured. +The refinement below spreads that budget across multiple short passes instead of one +contiguous one, trading a possibly-higher *aggregate* STW total for a much better +*latency distribution* — no single long tail pause. + +### Why the frontier can survive across passes: JVMTI object tags + +The obstacle to pausing and resuming a BFS is that the frontier (the worklist of +not-yet-expanded objects) is normally a set of raw addresses, and a moving/compacting GC +between passes can relocate or collect any of them. + +JVMTI object tags solve this: + +- Tags are identity-based and GC-move-transparent — a tagged object can be re-resolved + after a GC regardless of where it moved. +- Tags are **non-retaining** — tagging does not keep an object alive. This is the same + property the existing live-object sampler in this codebase already relies on, so this + is a new *use* of an existing mechanism, not new risk surface. +- Non-retention gives incremental resumption a useful side effect for free: if a frontier + object dies between passes, it simply fails to re-resolve on the next pass. That branch + of the search is pruned automatically, with no extra liveness bookkeeping required. + +### Data structures + +- **Frontier**: a set of `(tag, parent_tag, referrer_klass, depth)` records. `tag` is the + JVMTI tag assigned to a not-yet-expanded object; `parent_tag` links back for chain + reconstruction; `depth` supports the hop cap. +- **EdgeStore**: accumulates `(referrer_klass, parent_tag, depth)` per discovered edge for + objects that are on a path toward a target sample. Keyed by tag, not address — + degenerate relative to JFR's `EdgeStore` since object identity/field names are not + required, but it retains the same `parent_tag` linkage field as the Frontier so a chain + can be walked back from a target sample to a root by following `parent_tag` across + EdgeStore records. + +### Algorithm + +1. Seed the frontier from GC roots (first pass) or from the persisted frontier + (resumed pass). +2. Resolve currently-live tagged frontier objects. Objects that fail to resolve are + dropped (dead — free pruning). +3. Expand the frontier up to a fixed per-pass budget (edge count or time slice). +4. Newly discovered objects are tagged and added to the frontier for the next pass. +5. Persist the frontier (native memory owned by the agent, not thread-local scratch) and + return control to the VM. +6. Repeat until: a target sample is reached, the hop cap is hit, or a per-search + abandonment limit (see Termination) is exceeded. + +### Triggering passes: resolved — cannot avoid dedicated safepoints + +Investigated whether pass-continuation work could ride the JVMTI +`GarbageCollectionStart`/`GarbageCollectionFinish` callbacks — the same callback this +codebase already uses to call `_heap_usage_func` (`vmStructs.cpp`) — instead of +scheduling a dedicated `VM_Operation` per pass. + +**Confirmed the VM is genuinely at a safepoint (all mutators stopped) for the full +duration of both callbacks**, on every collector: + +- JVMTI spec: *"This event is sent while the VM is still stopped... the event handler + must not use JNI functions and must not use JVM TI functions except those which + specifically allow such use (see the raw monitor, memory management, and environment + local storage functions)."* +- openjdk/jdk source: delivery is synchronous on the VMThread + (`src/hotspot/share/prims/jvmtiExport.cpp:2752-2790`, comment *"this event is posted + from VM-Thread"*); every call site is inside a safepoint-executing `VM_Operation::doit()`, + backed by explicit asserts — e.g. Parallel GC's + `assert(SafepointSynchronize::is_at_safepoint())` (`gc/parallel/psScavenge.cpp:305-306`), + G1's `assert_at_safepoint_on_vm_thread()` (`gc/g1/g1VMOperations.cpp:141-157`), + Shenandoah and ZGC wrapping the same `SvcGCMarker` only inside their respective + `VM_Operation`/`VM_ZOperation::doit()` paths. Stable JDK 11 → mainline, across + Serial/Parallel/G1/Shenandoah/ZGC. + +**But this does not make the callback usable as the execution vehicle for a pass.** The +"functions which specifically allow such use" are exactly two: `Allocate` and +`Deallocate` (the entire **Memory Management** category). `SetTag`, `GetTag`, +`GetObjectsWithTags`, `FollowReferences`, and `IterateThroughHeap` are all in the +**Heap** category, which is *not* on that allowlist — calling any of them from inside +`GarbageCollectionStart`/`Finish` is exactly what the restriction forbids. The spec's own +prescribed escape hatch — notify a raw monitor from the callback, do the real work on a +separate agent thread — doesn't preserve the "rides the pause" property either: by the +time the woken agent thread runs, `VM_Operation::doit()` has already returned and the +safepoint has been released, so the tagging/walk work ends up running concurrently with +resumed mutators, not during the STW window. + +The only way to do the tag/walk work *while actually inside* the callback's STW window +would be to bypass the official JVMTI entry points and reach into HotSpot's internal +`JvmtiTagMap` directly via symbol-sniffing — reintroducing exactly the fragility class +already rejected for Approach C (unversioned internal C++ state, no stable ABI). Doing +that here would undo the reason C was rejected. + +**Conclusion: "no new marginal safepoints" is not achievable while staying within +official JVMTI usage.** Each pass needs its own dedicated, budget-capped `VM_Operation`. +The GC callbacks remain useful only as a low-cost *signal* ("a GC just happened, a pass +may be worth scheduling soon") — not as the execution vehicle for the pass itself. This +does not change the core incremental design (frontier persistence via JVMTI tags, +self-pruning of dead branches, per-pass budget) — it only removes the "zero marginal +safepoints" claim from the cost/benefit case. The design's actual value remains what it +was framed as: trading one long pause for several short, independently-scheduled ones — +a latency-distribution improvement, not a total-STW reduction. + +### Termination and abandonment + +Because passes are spread across a mutating heap, a search that never reaches a root or +the hop cap could otherwise persist indefinitely, accumulating abandoned frontier state +across GC cycles. Required cutoffs: + +- Hop cap (as in Approach B's single-pass form). +- A hard cap on passes-per-search or wall-clock TTL from first observation. +- Explicit reporting of abandoned searches (no silent truncation) so this shows up as a + measurable "chain not found within budget" outcome rather than being indistinguishable + from "no chain exists." + +### Correctness note: chains are historical, not a single consistent snapshot + +A chain built across multiple passes stitches together `"A referenced B"` facts observed +at different points in time, not one frozen graph. For the stated purpose — explaining, +by referrer type, what typically retains this class of surviving object — this is +sufficient, and is not meaningfully weaker than a single-pass walk: GC roots (e.g. thread +stack frames) are themselves a live-changing set across a single pause's boundary, so +"one true snapshot" is already an approximation in the single-pass case. Any +documentation or output surface built on this must describe results as an **observed** +retaining path, not a claim about the object's current exact retention state. + +### Cost/benefit summary + +- **Does not reduce total STW time.** Each safepoint/callback entry pays fixed + synchronization overhead; K short increments likely sum to equal or *more* aggregate + pause time than one contiguous walk covering the same work. +- **Improves latency distribution.** No single long tail pause — the thing most likely to + actually affect deployed application health (p99 latency, heartbeat timeouts), even + when total accumulated pause-ms is flat or slightly worse. + +## Non-goals + +- Exhaustive paths to all GC roots. +- Field-level or object-identity-level chains (referrer *type* only). +- Any GC-internal-closure hook (Approach C) or internal parallel-iteration API use as a + launch dependency. + +## Open questions before implementation + +1. ~~Confirm `GarbageCollectionStart`/`GarbageCollectionFinish` callback timing relative to + safepoint release.~~ **Resolved** (see Triggering section): the callback is genuinely + at a safepoint, but the JVMTI Heap-category functions needed to do frontier work + (`SetTag`/`GetTag`/`FollowReferences`/`IterateThroughHeap`) are not in the callback's + allowed function set, so each pass still needs its own dedicated `VM_Operation`. The + "no new marginal safepoints" framing is dropped; the design's value is latency + distribution, not total-STW reduction. +2. Choose per-pass budget defaults (edge count vs. time slice) and hop cap — needs + measurement against representative heap shapes, not a guess. +3. Decide the sample-batching policy: one incremental search per live-heap sample, or + batched multi-target BFS sharing a single frontier walk (batching amortizes better but + couples unrelated samples' termination conditions together). +4. Decide behavior when JVMTI tagging is already saturated by the existing live-object + sampler (tag-table sizing/contention) — this reuses infrastructure that has other + consumers in this codebase. +5. Decide the actual pass-scheduling policy now that GC callbacks can only be a signal, + not a vehicle: e.g. a background thread woken by the GC-callback signal that then + requests its own bounded `VM_Operation`, vs. a fixed-cadence timer independent of GC + activity. Needs a cost model for how many dedicated small safepoints per second are + acceptable before this stops being "more palatable" than one larger pause. From 6c5ba816454bfad64f66c0972a1987d66a7d7b87 Mon Sep 17 00:00:00 2001 From: Jaroslav Bachorik Date: Fri, 18 Sep 2026 14:56:27 +0200 Subject: [PATCH 1150/1150] Drop references to uncommitted plan docs and the superseded branch --- doc/ReferenceChains-PatentMemo.md | 187 +++ .../LiveHeapReferenceChains-Algorithm.html | 1450 +++++++++++++++++ .../LiveHeapReferenceChains-Implementation.md | 2 +- doc/architecture/LiveHeapReferenceChains.md | 58 +- ...ofiler-is-interacting-badly-with-wasmti.md | 15 + ...7751-causing-jvm-crash-on-ibm-was-90516.md | 95 ++ ...tthreads-skips-every-other-thread-in-th.md | 75 + ...-14549-sigsegv-in-recordingwriteclasses.md | 52 + ...548-sigsegv-in-profilerupdatethreadname.md | 39 + ...-14582-crash-sigsegv-in-dictionaryclear.md | 54 + ...btreeincrement-during-recordingwritecpo.md | 54 + ...5-crash-sigsegv-in-recordingswitchchunk.md | 57 + .../2026-03-30-continuation-enter-unwind.md | 598 +++++++ .../2026-04-01-remove-jmethodid-design.md | 190 +++ .../2026-04-01-remove-jmethodid-execution.md | 649 ++++++++ .../2026-04-21-signal-origin-validation.md | 197 +++ doc/review-pr-488-comparison.md | 122 ++ doc/review-pr-488-muse.json | 386 +++++ doc/review-pr-488-simple.json | 118 ++ doc/sframe-feasibility.md | 163 ++ doc/specs/sframe-transparent-loading.md | 411 +++++ 21 files changed, 4937 insertions(+), 35 deletions(-) create mode 100644 doc/ReferenceChains-PatentMemo.md create mode 100644 doc/architecture/LiveHeapReferenceChains-Algorithm.html create mode 100644 doc/libretti/2026-04-15-free-text-it-seems-like-java-profiler-is-interacting-badly-with-wasmti.md create mode 100644 doc/libretti/2026-04-20-scp-1154-crash-datadog-agent-7751-causing-jvm-crash-on-ibm-was-90516.md create mode 100644 doc/libretti/2026-04-22-prof-14332-wallclockasgct-collectthreads-skips-every-other-thread-in-th.md create mode 100644 doc/libretti/2026-05-07-prof-14549-sigsegv-in-recordingwriteclasses.md create mode 100644 doc/libretti/2026-05-08-prof-14548-sigsegv-in-profilerupdatethreadname.md create mode 100644 doc/libretti/2026-05-11-prof-14582-crash-sigsegv-in-dictionaryclear.md create mode 100644 doc/libretti/2026-05-11-prof-14583-crash-sigsegv-in-stdrbtreeincrement-during-recordingwritecpo.md create mode 100644 doc/libretti/2026-05-12-prof-14585-crash-sigsegv-in-recordingswitchchunk.md create mode 100644 doc/plans/2026-03-30-continuation-enter-unwind.md create mode 100644 doc/plans/2026-04-01-remove-jmethodid-design.md create mode 100644 doc/plans/2026-04-01-remove-jmethodid-execution.md create mode 100644 doc/plans/2026-04-21-signal-origin-validation.md create mode 100644 doc/review-pr-488-comparison.md create mode 100644 doc/review-pr-488-muse.json create mode 100644 doc/review-pr-488-simple.json create mode 100644 doc/sframe-feasibility.md create mode 100644 doc/specs/sframe-transparent-loading.md diff --git a/doc/ReferenceChains-PatentMemo.md b/doc/ReferenceChains-PatentMemo.md new file mode 100644 index 0000000000..b4c7aef828 --- /dev/null +++ b/doc/ReferenceChains-PatentMemo.md @@ -0,0 +1,187 @@ +# Patentability Memo: The Interruptible Heap Walker + +**Prepared for:** Datadog IP / patent counsel +**Prepared by:** Engineering (profiler team) +**Subject:** Technical novelty / prior-art assessment of the "interruptible heap walker" in the Datadog Java Profiler's reference-chains feature +**Status:** Internal working draft for counsel review. This is an engineer's technical read, **not** legal advice. Counsel must run a formal prior-art search (PatentScope, Espacenet, USPTO PPUBS, non-patent literature) and do claim construction before any filing decision. + +--- + +## Part 1 — The problem space + +You know the shape of this already, so this is a short orientation, not a tutorial. I include it only so the rest of the memo has shared vocabulary. + +### 1.1 What we are building + +A production, in-process Java profiler that, for objects suspected of leaking, reconstructs a **reference chain** (a path of references from a GC root down to the suspect object) — and does so continuously, in a live customer JVM, without imposing an unacceptable stop-the-world (STW) pause. + +### 1.2 Why this is hard + +Building a reference chain requires a **heap walk**: starting at the GC roots, following references outward, searching for the target object. On a real production heap (tens of millions of objects), a full walk inside a single STW pause can take hundreds of milliseconds to seconds. That is unacceptable for a continuous production profiler — it causes latency spikes, failed health checks, and visible disruption. + +The naive single-freeze approach (what a heap dump does) is a non-starter in production. Even the closest existing system — OpenJDK's built-in JFR leak profiler (`jdk.OldObjectSample`) — runs its chain search as **one single walk inside one STW pause**, bounded by a wall-clock timer. When the timer fires, it **stops and discards the partial work**. It does not resume. This is the single most important piece of prior art; we return to it in Part 3. + +### 1.3 The specific obstacle to pausing and resuming + +The obstacle is the **moving/compacting GC**. A breadth-first walk keeps a **frontier**: the set of objects reached but not yet expanded. Between two passes, the GC can **relocate** objects (compacting collectors move them to new addresses) or **collect** them (if they became unreachable). If the frontier were a set of raw addresses, a GC between passes would invalidate it — addresses would point at moved or dead objects. You could not resume. + +That is the problem our invention solves. + +--- + +## Part 2 — The invention + +One sentence: + +> **Instead of one long STW walk, we break the walk into many short, independently-bounded passes, and we persist the frontier across passes using JVMTI object tags — stable identities that survive a moving GC — so we can resume exactly where we left off, with dead branches pruned automatically.** + +Six mechanisms make it work. Only the first is a strong novelty candidate; the rest are supporting machinery, and I say so plainly. + +### 2.1 Mechanism 1 (the heart): a tag-persisted frontier across GC cycles + +We keep the frontier as a table keyed by **JVMTI object tags**, not raw addresses. A tag is a 64-bit identity attached to an object through the JVMTI interface. Two properties make this work: + +1. **Tags move with the object.** A tag is an identity, not an address. If the GC relocates the object, the tag still resolves to it. So a tag-keyed frontier stays valid across a moving GC. +2. **Tags are non-retaining.** Tagging an object does not keep it alive — the GC may still collect it. (We must not cause leaks ourselves.) + +The frontier table — `(tag, parent_tag, referrer_class, depth)` records — lives in our own native memory between passes. On the next pass, we resolve the tags to live objects. Any tag whose object was collected simply fails to resolve; that branch of the search prunes itself for free, with no separate liveness tracking. + +**This is the key novelty candidate.** We did not find prior art that persists a graph-walk frontier as JVMTI object tags across multiple GC cycles to resume a heap walk on a moving heap. The closest art (JFR) stores raw addresses and discards the frontier on timeout. + +### 2.2 Mechanism 2: cooperative abort from another thread + +We sometimes must stop an in-flight walk immediately (profiler shutdown/reconfig). A started heap walk cannot be interrupted by an external signal; the JVM keeps invoking our per-object callback until the walk finishes or the callback says stop. So we use an atomic flag set by the shutdown thread and checked by the callback on every object; on seeing it set, the callback returns the JVMTI "abort" value, ending the walk within one callback. + +Standard cooperative cancellation. Supporting machinery, not a novelty hook — the abort return value is JVMTI-spec-taught. + +### 2.3 Mechanism 3: in-callback wall-clock deadline + +The callback also checks a per-pass deadline periodically (every 4096 invocations) and aborts when over budget, bounding each pass's STW time regardless of heap size. + +**Prior-art note:** this exact pattern (periodic counter + time check inside a heap-walk callback, abort when over) already exists in the OpenJDK JFR leak profiler, in `granularTimer.cpp`, copyright **2014**. We read the source. This mechanism is **anticipated by shipping code** — not novel. + +### 2.4 Mechanism 4: array-holder batching (one walk per BFS level) + +Naively, expanding the frontier one step means one heap walk per frontier object — each a separate STW operation with fixed overhead. Instead, we build a Java `Object[]`, place a batch of frontier objects in it, and issue one `FollowReferences` over the array as the starting point. One STW walk expands a whole BFS level. The callback is gated to descend only one hop past the batch. + +Engineering optimization, "obvious to try" risk. Not a standalone novelty hook. + +### 2.5 Mechanism 5: reverse-fill + rolling-resume cursor + +When a pass is cut off mid-batch, we want to resume at the exact object we were inside, not redo the batch. We exploit a HotSpot implementation detail: `FollowReferences` visits an array's elements in LIFO order. We fill the holder in **reverse**, so the LIFO visit yields objects in **ascending** original order, giving the resume cursor a clean meaning: "got through 0..k; resume at k." We track the batch entry being visited at abort time (`_last_visited_batch_tag`) and resume there; already-processed entries are marked done, the partial entry is redone idempotently. + +Narrowest, cleverest piece. Not found in art as a specific combination. But the generic "checkpoint and resume at last reachable point" is known (US10365963, below), which weakens obviousness. Best as a dependent limitation. + +### 2.6 Mechanism 6: dead-branch self-pruning + +A consequence of tag-keying: frontier objects collected between passes fail to resolve and are dropped, with no separate liveness bookkeeping. Natural consequence of the JVMTI tag spec (tags are non-retaining, cleared on collection). Spec-taught, likely obvious. Supporting machinery. + +--- + +## Part 3 — Prior art (read, not just searched) + +This is a starting point, not an exhaustive search. We used web search plus reading of source code we have locally. A formal search in PatentScope / Espacenet / USPTO PPUBS / non-patent literature is still required. + +### 3.1 OpenJDK JFR leak profiler — closest art, shipping + +Source read locally: `src/hotspot/share/jfr/leakprofiler/chains/` (`bfsClosure.cpp`, `edgeStore.cpp`, `edgeQueue.hpp`, `granularTimer.{hpp,cpp}`, `pathToGcRootsOperation.cpp`). + +What it does: +- BFS from GC roots, hop cap, parent links in an `EdgeStore` for chain reconstruction. This is the structural template our own design doc cites as the starting point. +- Wall-clock bounded: `GranularTimer` (copyright **2014**) decrements a counter per iteration and checks the time; `BFSClosure::closure_impl` checks `if (GranularTimer::is_finished()) return;` per edge. **This is the same pattern as our mechanism 3.** → Mechanism 3 is anticipated by shipping code from 2014. + +What it does **not** do (the gaps we fill): +- Its `EdgeQueue` stores raw `UnifiedOopRef` addresses and raw `const Edge*` parent pointers — **not** JVMTI tags. Raw addresses do not survive a moving GC. +- The entire BFS runs inside one `PathToGcRootsOperation::doit()` asserting `SafepointSynchronize::is_at_safepoint()`. When the timer fires, `process_queue()` returns and the `EdgeQueue` (a local) is destroyed. **Frontier discarded. No resume, no cross-GC persistence, no multi-pass walk.** +- No cooperative abort from another thread (it is single-pass, timer-bounded). +- No array-holder batching. + +### 3.2 Oracle US10635570B2 / US20190102278A1 — "Memory leak profiling events" + +Inventors: Erik Gahlin, Marcus Hirt (the JFR authors). Priority 2017-09-29, granted 2020-04-28, assigned to Oracle. Covers the JFR leak profiler's *sampling* side. + +Claims: allocation sampling, a threshold-constrained priority queue, GC-triggered queue updates (removing dead samples, redistributing allocation span), and **a single BFS at dump time** to find the shortest path to a GC root. + +Does **not** claim or describe: a resumable multi-pass walk, a frontier persisted across GC cycles, mid-walk interruption from another thread, or batched holder traversal. + +→ Anticipates the *sampling + single dump-time BFS* concept (overlaps our candidate-selection side), but leaves the entire *interruptible/resumable walk* delta untouched. Most likely to be cited against our sampling parts; does not cover walk interruption. + +### 3.3 US10365963B2 — "Accessing damaged heaps using combined linear heap slot based and object graph walking" + +Post-mortem walk of a **crashed/dead** heap with corrupted regions; resumes at the nearest reachable slot after damage. Teaches generic "store last reachable point and resume." + +Off-point for our core novelty: the heap is static (dead process, no moving GC, no live collection between passes); it is about corruption robustness, not live STW bounding. Relevant only as generic checkpoint-resume prior art, which weakens obviousness for our cursor pieces (4, 5) but does not teach the tag-persisted frontier across live GCs (1). + +### 3.4 Other patents checked, ruled out + +- **US7971010B2** "Loitering trace" — sampling of GC survivors, not a resumable walk. +- **US20100223433A1**, **US11221947B2** — traversal *inside* the GC itself (collector internals, not an external attached agent). +- **US6286016B1** — incremental heap *expansion* (sizing), not traversal. +- **US8543987B2** — GC + profiler allocation callbacks, not a resumable walk. + +### 3.5 Non-patent art checked + +- **async-profiler "live mode" (2.9+)** — allocation sampling + LRU live-object tracking. No resumable `FollowReferences`, no tag-persisted frontier. +- **.NET gcdump** — single induced GC walks live objects, emits events inline. Not resumable across passes. +- **Eclipse MAT / IBM HeapAnalyzer** — offline heap-dump analysis, not live. +- **JVMTI specification** — documents the abort return value and that tags are identity-based, GC-move-transparent, non-retaining (cleared on collection). These spec-taught facts feed obviousness against mechanisms 2 and 6. + +--- + +## Part 4 — Per-mechanism novelty verdict + +| # | Mechanism | Prior art | Novelty | +|---|---|---|---| +| 1 | Resumable BFS via tag-persisted frontier across GCs | JFR discards frontier; Oracle patent single-pass; US10365963 dead heap | **Not found — strongest candidate** | +| 5 | Reverse-fill + rolling-resume cursor (exploit HotSpot LIFO visit order) | Not found; US10365963 teaches generic checkpoint-resume | Narrow, cleverest; some obviousness exposure | +| 4 | Array-holder batching (one walk per BFS level) | Not found | Engineering opt; "obvious to try" risk | +| 3 | In-callback wall-clock deadline | **JFR `GranularTimer` (2014)** — same pattern | **Anticipated** | +| 2 | Cooperative abort (atomic flag + callback + abort return) | Abort value is spec-taught; cooperative cancellation textbook | Weak, likely obvious | +| 6 | Dead-branch self-pruning (tags vanish on GC) | JVMTI spec (tags cleared on collection) | Spec-taught, likely obvious | + +--- + +## Part 5 — Assessment and recommendation + +### What is most likely patentable + +The one piece that survives the closest art is **mechanism 1: persisting the BFS frontier as JVMTI object tags so the walk resumes across GC cycles on a moving heap.** Neither the shipping JFR leak profiler (raw addresses, single STW, frontier destroyed on timeout) nor the Oracle JFR patent (single dump-time BFS) nor US10365963 (post-mortem dead heap) teach this. + +Mechanisms 4 and 5 (batched holder + reverse-fill resumable cursor) are also not found in the art, but are engineering optimizations with "obvious to try" exposure, and US10365963's generic checkpoint-resume weakens them. Best as **dependent limitations** narrowing an independent claim built around mechanism 1, not as standalone inventions. + +### What is not patentable / weak + +- Mechanism 3 (in-callback deadline) is anticipated by JFR's `GranularTimer` (2014). +- Mechanisms 2 and 6 are spec-taught or textbook. + +### Risks counsel should weigh + +1. **§101 / Alice.** Software on a generic JVM doing "incremental graph traversal with checkpoints." Post-Alice, survival is best when tied to a *specific technical improvement to the computer itself*, not "do the known thing in smaller chunks." Our design doc's own framing — "trading one long pause for several short ones … a latency-distribution improvement, not a total-STW reduction" — is a *performance optimization*, the weakest category post-Alice / *Enfish* / *Berkheimer*. Consider framing the tag-persisted frontier as a *specific technical mechanism* (identity-based, GC-move-surviving tags solving a concrete moving-heap problem) rather than an abstract "resume a search in pieces." + +2. **Obviousness.** The primitives (tags, abort, BFS, batching, checkpoints) are each known. Novelty is in the *specific combination* for moving-heap resumption. Assess "obvious to try" under KSR, and whether the tag-persisted-across-GC frontier is a non-obvious specific solution. + +3. **Disclosure cost.** Filing publishes the reverse-fill trick and the tag-frontier design. If the patent is later rejected or narrowed, we disclose engineering cleverness for limited protection. Weigh trade-secret on the batching/resume trick against a likely-narrowed patent. + +### Suggested next step for counsel + +- Run a focused search on the strongest claim: **"persisting a graph-traversal frontier as object identity tags that survive a moving/compacting GC, to resume an incremental heap walk across GC cycles on a live JVM."** Search terms: "resumable heap walk", "tag-persisted frontier", "incremental reachability across GC", "object tag resume", "multi-pass heap traversal moving collector". Targets: PatentScope, Espacenet, USPTO PPUBS, Google Patents full text, non-patent literature (incremental/graph-traversal profiler papers, IBM/OpenJ9 tooling docs, OpenJDK JFR history). +- If it clears, consider an independent claim around mechanism 1, with mechanisms 4 and 5 as dependent limitations; **avoid** claiming mechanism 3 (anticipated) or 2/6 (spec-taught) as standalone. +- Do **not** rely on the broad framing "interruptible/resumable heap walking with bounded pauses" — JFR + the Oracle patent likely render that obvious, and §101 will bite. + +--- + +## Appendix A — Code locators + +- Implementation: `ddprof-lib/src/main/cpp/referenceChains.cpp` and `referenceChains.h`. The interruptible heap walker is the `ReferenceChainTracker` class; key methods `runPass`, `expandFrontier`, `admitStaticFieldRoots`, `heapReferenceCallback`. +- Design docs: `doc/reference-chains-design.md`, `doc/reference-chains-collection-summary.md`. +- Closest prior art (OpenJDK JFR leak profiler): `src/hotspot/share/jfr/leakprofiler/chains/` in the OpenJDK repo — `bfsClosure.cpp`, `edgeStore.cpp`, `edgeQueue.hpp`, `granularTimer.{hpp,cpp}`, `pathToGcRootsOperation.cpp`. + +## Appendix B — Prior-art references + +| Reference | Number / location | Relevance | +|---|---|---| +| OpenJDK JFR leak profiler (shipping) | `src/hotspot/share/jfr/leakprofiler/chains/` | Closest art; anticipates mechanism 3 (`GranularTimer`, 2014); does not teach resumable tag-frontier | +| Oracle "Memory leak profiling events" | US10635570B2 / US20190102278A1 (priority 2017-09-29, granted 2020-04-28; Gahlin & Hirt) | Anticipates sampling + single dump-time BFS; leaves resumable walk untouched | +| "Accessing damaged heaps …" | US10365963B2 | Generic checkpoint-resume on a dead heap; off-point for live tag-frontier | +| "Loitering trace" | US7971010B2 | Sampling of GC survivors; off-point | +| JVMTI specification | Oracle JVMTI 1.2.3 / 11 | Teaches the abort return value; tag identity / move-transparency / non-retention | diff --git a/doc/architecture/LiveHeapReferenceChains-Algorithm.html b/doc/architecture/LiveHeapReferenceChains-Algorithm.html new file mode 100644 index 0000000000..e6412ffea4 --- /dev/null +++ b/doc/architecture/LiveHeapReferenceChains-Algorithm.html @@ -0,0 +1,1450 @@ + + + + + +Reference Chains — Leak Triggers & Resumable Heap Walk + + + + +
+

Reference Chains: leak-detection triggers & the resumable heap walk

+

Extracted from ddprof-lib/src/main/cpp/referenceChains.{h,cpp}, + livenessTracker.{h,cpp}, painBudget.h — branch jb/reference-chains-pi. + Every claim carries a file:line anchor.

+
+ + + +
+ + +
+

1 · End-to-end pipeline

+

Two independent subsystems cooperate. LivenessTracker decides that + something is leaking and which class. ReferenceChainTracker then answers why it is + retained, by running a budgeted, resumable, tag-based breadth-first walk of the reachable object graph on + its own JVMTI-attached thread.

+ +
+flowchart LR + subgraph LT["LivenessTracker (leak detection)"] + direction TB + GC["GC epoch advance"] --> POP["cleanup_table(): fold survivors
into per-klass population rings"] + POP --> TREND["hasQualifyingGrowth():
least-squares slope + hysteresis"] + POP --> FLOOR["heapFloorRising() /
secondsToOOM() projection"] + TREND --> CAND["selectLeakCandidates()
top-5 by slope"] + FLOOR --> CAND + end + + subgraph RC["ReferenceChainTracker (retention explanation)"] + direction TB + SIG["hasLeakSignal()"] --> AFF["canAffordNewSearch()
pain budget + signal"] + AFF --> SCHED["shouldRunPass()"] + SCHED --> PASS["runPass() — one budgeted pass"] + PASS --> FRONT[("FrontierTable
tag -> parent_tag")] + FRONT --> PASS + PASS --> TERM["Termination check"] + TERM -->|"terminal"| REL["releaseSearchTags() -> restartSearch()"] + REL --> SCHED + end + + CAND --> SIG + CAND --> POLL["pollWatchedTargets():
tag candidate instances"] + POLL --> PASS + FRONT --> RECON["reconstructChain()
leaf -> root"] + RECON --> JFR["JFR: ReferenceChain /
ReferenceChainAbandoned"] +
+ +
+
+

One thread, one lock

+

A single agent-owned pthread (threadLoop(), referenceChains.cpp:690) + wakes on an adaptive cadence, runs at most one pass, then polls targets. GC callbacks only bump an atomic epoch.

+
+
+

Tags, not handles

+

Frontier identity is a JVMTI object tag — non-retaining, so the walk never keeps a dying object alive. + Dead objects vanish for free at the next GetObjectsWithTags.

+
+
+

Resumable by construction

+

Every pass is bounded by an edge budget and a wall-clock deadline. Unfinished work stays in a FIFO + (_pending_expand) and is picked up by the next pass — including a partially-visited batch.

+
+
+

Cost is paid for, not assumed

+

Two leaky buckets (PainBudget): one gates starting a search on accumulated safepoint cost, + one gates every pass on non-safepoint CPU cost.

+
+
+ +
+ Three places where the in-source narrative has drifted from the code — verified, worth knowing before reading the headers: +
    +
  1. There is no root-seeded FollowReferences first pass any more. runPass() always + drives runPassManualWalk(); the only first-pass difference is that root enumeration is forced + (referenceChains.cpp:3665-3699). The header comment at + referenceChains.h:47-53 still describes the old shape.
    2. +
  2. SearchAbandonReason::TTL is not driven by _ttl_ms. That field is assigned in + start() and only reported in the abandoned event; the branch that stores TTL is the + no-progress detector (referenceChains.cpp:3807-3818).
    3. +
  3. CANARY_NO_PROGRESS_PASS_LIMIT = 3 carries an in-source // TEMP: was 30, lowered for testing + marker (referenceChains.h:2376), while the comment below it still reasons about a base of 30.
    4. +
+
+
+ + +
+

2 · Leak signal: how a class becomes a candidate

+

A klass is only trusted as leaking after it clears a fill gate, a slope gate, a class-level hysteresis + gate and a per-thread hysteresis gate. The bar itself moves depending on whether the aggregate heap floor + corroborates the story.

+ +
+flowchart TB + A["GC epoch: cleanup_table() folds surviving
tracked objects per klass"] --> B["count_ring[30] push
(sample = distinct GC ages, not raw count)"] + B --> C{"ring_fill >= 10
KLASS_POPULATION_MIN_FILL_FOR_TREND"} + C -->|"no"| X1["no trend yet"] + C -->|"yes"| D["least-squares regression over the ring
cached_slope = recent_mean - earliest_mean"] + D --> E{"cached_slope >= max(0.15 * earliest_mean, 1)
LEAK_GROWTH_REL_MIN / ABS_MIN"} + E -->|"no"| F["consecutive_positive = 0 (hard reset)"] + E -->|"yes"| G["consecutive_positive++ (saturating)"] + + G --> H{"heapFloorRising()?"} + H -->|"yes, corroborated"| I["required_hysteresis = 3"] + H -->|"no"| J["required_hysteresis = 5"] + I --> K + J --> K{"consecutive_positive >= required_hysteresis
AND cached_slope > 0"} + K -->|"no"| X2["skip klass"] + K --> L{"any tid_trend with
consecutive_positive >= required_hysteresis"} + L -->|"no"| X3["skip klass — no single thread
owns the growth"] + L -->|"yes"| M["insert into top-5 by slope descending"] + M --> N["selectLeakCandidates() returns candidates"] +
+ +
+ Latency to first candidate ≈ 10 ring pushes to fill the trend window, plus 3–5 further qualifying + epochs of hysteresis — roughly 13–15 GC epochs. This is exactly why the search must be restartable: a search that + completes the whole reachable graph before the trend detector has spoken would otherwise never look again. +
+ +

Heap-floor corroboration

+

heapFloorRising() (livenessTracker.cpp:1220-1259) demands + both a rising mean and a rising minimum, so a sawtooth workload whose troughs stay flat does not lower the bar.

+ + + + + + + + +
GateConstantValue
Mean rise, relativeHEAP_FLOOR_GROWTH_REL_MIN0.02
Mean rise, absoluteHEAP_FLOOR_GROWTH_ABS_MIN1 MiB
Floor rise, relativeHEAP_FLOOR_FLOOR_REL_MIN0.01
Floor rise, absoluteHEAP_FLOOR_FLOOR_ABS_MIN512 KiB
+ +
+ Two rankings, deliberately different. selectLeakCandidates() is slow and gated + (livenessTracker.cpp:1370). topKlassesByGenerationCount() is fast and + ungated — ranked on the single latest sample with no hysteresis at all + (livenessTracker.cpp:1480). The fast one is only ever consulted after the slow one + has already fired, so it never needs to wait out the same hysteresis twice + (referenceChains.cpp:4175-4201). +
+ +
+

Qualifying growth bar: max(0.15 · earliest_mean, 1)

+

Below an earliest-window mean of ~6.7 tracked instances, the absolute floor + (LEAK_GROWTH_ABS_MIN = 1) sets the bar, not the 15% relative term + (LEAK_GROWTH_REL_MIN) — a klass with only a handful of instances still needs to grow by a whole + instance to qualify, it cannot coast in on a tiny relative slope.

+
+
+
+ + +
+

3 · Urgency: the OOM projection and its latch

+

secondsToOOM() projects when the live-heap floor will hit the tighter of the JVM max heap + and the container limit. The raw value swings by orders of magnitude between observations, so it is never compared + bare — it is read through a latch with separate arm and release bars.

+ +
+flowchart TB + A["_heap_floor_ring / _time_ring
30 post-GC samples"] --> B{"_gc_generations on
AND max_heap > 0"} + B -->|"no"| N1["return -1 (no projection)"] + B -->|"yes"| C["limit = min(JVM max heap, container limit)"] + C --> D{"ring_fill >= 10"} + D -->|"no"| N2["return -1 (INSUFFICIENT_FILL)"] + D -->|"yes"| E["regress bytes and time"] + E --> F{"bytes_delta > 0 AND time_delta > 0"} + F -->|"no"| N3["return -1 (NOT_RISING)"] + F -->|"yes"| G["re-regress the most recent half
(min fill 5)"] + G --> H{"recent half also rising?"} + H -->|"no"| N4["return -1 (RECENT_HALF_FLAT)
rejects a plateaued step change"] + H -->|"yes"| I["seconds = (limit - recent_mean) / rate"] +
+ +
+stateDiagram-v2 + direction LR + [*] --> Calm + Calm --> Latched: "secondsToOOM() in [0, 300) — OOM_URGENT_THRESHOLD_S
also clears _urgent_search_spent, so a new episode earns a fresh entitlement" + note right of Latched: "a reading between 300 and 600 stays Latched
and resets _urgent_release_ticks to 0" + Latched --> Releasing: "reading at or over 600 (OOM_URGENT_RELEASE_S), or negative — ++_urgent_release_ticks" + Releasing --> Latched: "any reading back under the release bar" + Releasing --> Calm: "URGENT_RELEASE_CONSECUTIVE = 5 clear readings in a row" +
+ +
+ Two distinct notions of "urgent" coexist. + isUrgent() uses the latch above with a 300 s arm bar and is what suppresses the no-progress abandon and + authorises one out-of-band search per episode. + threadLoop()'s ramp uses a raw, unlatched, much wider window — + seconds_to_oom < OOM_RAMP_START_S = 1800 s — recomputed on every wake + (referenceChains.cpp:735-736). They arm at different distances and flap differently. +
+ +

The OOM ramp (referenceChains.cpp:739-772)

+

Inside the 30-minute window, with x = 1 - secondsToOOM()/1800 (0 at the far edge, 1 at OOM), + both the pause target and the cadence ramp exponentially toward their urgent ceilings:

+ + + + + + + +
QuantityBaselineUrgent ceilingRamp
Per-pass pause target_pause_target_msURGENT_PAUSE_TARGET_MS = 100 msbase * (100/base)^x
Pass cadencePASS_CADENCE_NS = 1 sURGENT_CADENCE_NS = 10 ms1s * (10ms/1s)^x
Edge budget ceiling_budgetmin(_budget * 4, MAX_REFERENCE_CHAINS_BUDGET)step, on target change
+

The cadence ramps from the fixed baseline, not from the live _effective_cadence_ns — + anchoring on the moving value would compound the exponent across iterations. While urgent, the ramp owns + _effective_cadence_ns outright; updatePacing() silently resumes ownership the moment urgency clears.

+ +
+

Cadence ramp: 1000ms · (10ms/1000ms)^x

+

Fully determined by PASS_CADENCE_NS and URGENT_CADENCE_NS — no + assumed baseline. The dashed marker is where isUrgent()'s own, separately-latched threshold + (OOM_URGENT_THRESHOLD_S = 300 s) falls on this same x-axis, for scale.

+
+
+
+ + +
+

4 · Pass scheduling: shouldRunPass()

+

Called once per BFS-thread wake. Three regimes — no search yet, search terminal, search running — + evaluated strictly in this order (referenceChains.cpp:870-1006).

+ +
+flowchart TB + S(["shouldRunPass(now_ns)"]) --> B1{"_search_started?"} + + B1 -->|"no"| A1{"canAffordNewSearch(now)
= safepoint pain budget drained
AND hasLeakSignal()"} + A1 -->|"no"| F1["FALSE"] + A1 -->|"yes"| T1["_urgent_search_spent = _urgent_latched
TRUE — take the very first pass"] + + B1 -->|"yes"| B2{"_search_state == RUNNING?"} + + B2 -->|"no (terminal)"| C1{"_tags_released?"} + C1 -->|"no"| T2["TRUE — force runPass() to retry
releaseSearchTags(); restart is forbidden
until every tag is confirmed cleared"] + C1 -->|"yes"| C2{"canAffordNewSearch(now)"} + C2 -->|"no"| F2["FALSE — terminal, waiting"] + C2 -->|"yes"| T3["restartSearch()
TRUE"] + + B2 -->|"yes"| D0["canary_active = candidates outstanding
all_covered = leak tags all resolved
emergency = canary_active AND no candidate
progress for 3 passes"] + D0 --> D1["CPU pain refill multiplier:
all_covered -> 1x
emergency -> 100x
canary_active -> 15x
else -> 1x"] + D1 --> D2{"_cpu_pain_budget.canStartNow()"} + D2 -->|"no"| F3["FALSE — throttled"] + D2 -->|"yes"| D3{"gcFinishEpoch() changed
since last pass?"} + D3 -->|"yes"| T4["TRUE — GC trigger"] + D3 -->|"no"| D4{"canary_active?"} + D4 -->|"yes"| T5["TRUE — run back-to-back,
cadence bypassed"] + D4 -->|"no"| D5{"now - _last_pass_ns >=
_effective_cadence_ns"} + D5 -->|"yes"| T6["TRUE — cadence trigger"] + D5 -->|"no"| F4["FALSE — idle"] +
+ +
+
+

Order matters in the multiplier

+

all_covered is tested before emergency, so a search whose leak tags are all resolved + drops back to 1× even if the canary is nominally stuck.

+
+
+

Two budgets, two scopes

+

_safepoint_pain_budget is never consulted for a RUNNING search — only through + canAffordNewSearch() when starting or restarting one. _cpu_pain_budget gates each pass.

+
+
+

No early wake on GC

+

onGCFinish() bumps an atomic epoch and nothing else. Waking the thread per GC would buy ≤1 s of + latency while collapsing the loop cadence to GC frequency (referenceChains.cpp:858-866).

+
+
+

Sleep only when idle

+

threadLoop() skips its sleep entirely when a pass is about to run, so canary passes execute + back-to-back and the PID controller alone regulates cost (referenceChains.cpp:805-811).

+
+
+ +
+ Why a RUNNING search is not gated on hasLeakSignal() + (referenceChains.cpp:785-798): that signal answers "is there a leak candidate right now", + which is unrelated to whether an in-flight search still has pending frontier work. Gating every pass on it would + stall a search's own convergence whenever no candidate happens to be visible. +
+
+ + +
+

5 · One search, pass by pass

+

A search is a sequence of passes over a persistent frontier. Each pass performs up to four sub-phases, + each of which can truncate independently and hand its remainder to the next pass.

+ +
+flowchart TB + P(["runPass(jvmti, jni)"]) --> G0{"enabled AND frontier exists?"} + G0 -->|"no"| Z0["bail"] + G0 -->|"yes"| G1{"_search_state == RUNNING?"} + G1 -->|"no"| Z1["retry releaseSearchTags() if needed;
return as a no-op"] + G1 -->|"yes"| R0["resolveLoadedClasses()
class_tag -> class-name dictionary id
(per-class scan skipped when the count is unchanged)"] + + R0 --> R1{"first pass?
(_search_started == false)"} + R1 -->|"yes"| R2["_search_started = true
_search_start_ns = now
force root enumeration"] + R1 -->|"no"| R3{"last root enum truncated
OR >= 2s since last
(ROOT_ENUM_MIN_INTERVAL_NS)"} + R3 -->|"yes"| R2 + R3 -->|"no"| R4["skip root enumeration this pass"] + + R2 --> W["runPassManualWalk()"] + R4 --> W + + subgraph WALK["runPassManualWalk — four sub-phases under one deadline"] + direction TB + W1["A · Root enumeration
IterateOverReachableObjects
budget = _first_pass_budget · no deadline"] + W2["B · Static-field sweep
admitStaticFieldRoots(), 512-class chunk
only when the loaded-class count changed"] + W3["C · Ordinary expansion
expandFrontier(_pending_expand / _priority_expand)"] + W4["D · Rotation
collect stale roots / leak accumulators / stale expanded
then a second expandFrontier() on the reserved budget"] + W1 --> W2 --> W3 --> W4 + end + + W --> WALK + WALK --> M0["_passes_run++ · _last_pass_gc_finish_epoch · _last_pass_ns"] + M0 --> M1{"root-enum pass?"} + M1 -->|"no"| M2["updatePacing(safepoint_ticks)"] + M1 -->|"yes"| M3["maybeRevokeBorrowForRootEnumPass()
(excluded from the PID signal)"] + M2 --> M4 + M3 --> M4["_search_pain_ms += safepoint ms
_cpu_pain_budget.spend(non-safepoint ms)"] + M4 --> M5["Termination decision chain — see tab 8"] +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Sub-phaseBudgetDeadlineWhy it exists
A · Root enumeration_first_pass_budget (auto = min(_budget*50, 200000))none, by designEnumerates heap roots and stack refs. Re-run at most every 2 s; already-admitted roots short-circuit + cheaply, so a root missed this pass is picked up later, never lost.
B · Static-field sweepchunk of 512 classes_pass_deadline_nsJVMTI has no jvmtiHeapRootKind for static fields, and expansion never descends from class + objects — without this, SomeClass.staticField → obj is structurally undiscoverable.
C · Ordinary expansion_effective_budget − rotation reserve_pass_deadline_nsThe actual BFS: resolve a batch of frontier tags, walk exactly one hop from each.
D · Rotationmin(expand_budget/2, 288) reserved up front_pass_deadline_nsRe-observes entries whose recorded story may have gone stale: transient root kinds, leak-accumulating + signatures, long-expanded entries whose fields have since been mutated.
+
+ + +
+

6 · expandFrontier(): the resumable batch loop

+

The heart of the resumability. Each iteration resolves a batch of frontier tags back to live objects, + runs one stop-the-world FollowReferences for the whole batch, and descends exactly one hop — + enforced by a membership gate on the batch's own tag set (referenceChains.cpp:2943-3303).

+ +
+flowchart TB + E(["expandFrontier(hop_cap, budget)"]) --> L0{"deadline reached?
OS::nanotime() >= _pass_deadline_ns"} + L0 -->|"yes"| TR["truncated = true — break"] + L0 -->|"no"| L1["pick a lane:
_priority_expand vs _pending_expand,
alternating via _expand_lane_prefer_priority"] + L1 --> L2{"both lanes empty?"} + L2 -->|"yes"| DONE["no pending work — break"] + L2 -->|"no"| L3["batch_size = min(queue, budget, _gotw_batch_size)"] + L3 --> L4["GetObjectsWithTags(batch)
dead tags simply do not come back = free pruning"] + L4 --> L5["self-calibrate _gotw_batch_size:
per-call EMA scaled by the remaining
deadline window, clamped to [8, 512]"] + L5 --> L6["build jobjectArray holder
+ fill batch_tags membership set"] + L6 --> L7["FollowReferences(holder) — one STW HeapWalkOperation
heapReferenceCallback descends only into tags in batch_tags"] + L7 --> L8{"truncated?"} + + L8 -->|"no"| S1["for each tag in batch:
resolved -> markExpanded()
unresolved -> clear() = ABANDONED
pop_front() · progress = true"] + L8 -->|"yes, some batch entry visited"| S2["rolling resume:
settle only entries BEFORE
_last_visited_batch_tag;
leave it and the rest at the queue front"] + L8 -->|"yes, nothing visited"| S3["leave the whole batch queued"] + + S1 --> L0 + S2 --> TR + S3 --> TR +
+ +
+
+

One hop, guaranteed

+

heapReferenceCallback returns JVMTI_VISIT_OBJECTS only when the visited object's tag is + in batch_tags. Everything else is admitted but not descended into + (referenceChains.cpp:2001-2017).

+
+
+

Rolling resume

+

_last_visited_batch_tag is the cursor inside a truncated batch. Entries before it are settled; + the partially-visited one stays at the queue head and is re-walked next pass.

+
+
+

Deadline sampled cheaply

+

Inside the callback the wall clock is only read every 4096th invocation + ((++counter & 0xFFF) == 0) — the loop top checks it per batch + (referenceChains.cpp:1647-1656).

+
+
+

Adaptive batch size

+

_gotw_batch_size is tuned per call from measured GetObjectsWithTags cost against the + remaining deadline window, so a slow JVM naturally shrinks its batches rather than blowing the pause target.

+
+
+ +

Admission, in callback order

+
+flowchart TB + C(["heapReferenceCallback(kind, tag_ptr, referrer_tag_ptr)"]) --> A0{"_abort_pass_requested"} + A0 -->|"yes"| Z1["truncated · VISIT_ABORT"] + A0 -->|"no"| A1{"every 4096th call:
past _pass_deadline_ns?"} + A1 -->|"yes"| Z1 + A1 -->|"no"| A2{"tag at or below MARKER_TAG_BASE
(canary marker)"} + A2 -->|"yes"| Z2["record chain link, set found bit,
do NOT descend"] + A2 -->|"no"| A3{"tag is negative (class object)"} + A3 -->|"yes"| Z3["descend only for the static-field
seed holder -> class edge"] + A3 -->|"no"| A4["compute parent_tag and depth
from the referrer's tag"] + A4 --> A5{"depth >= hop_cap"} + A5 -->|"yes"| Z4["drop — no admission, no descent"] + A5 -->|"no"| A6{"isLeakTag(tag)?"} + A6 -->|"yes"| Z5["allocate frontier tag, insert,
setLeakTag(), record instance, descend"] + A6 -->|"no"| A7{"tag == 0 (unseen)"} + A7 -->|"yes"| A8["admitObject()"] + A7 -->|"no"| A9["already admitted ->
improveChain() / reparentToDurableRoot()
/ maybeUpgradeRootAttachedRootKind()"] + A8 --> A10{"result"} + A10 -->|"BUDGET_EXHAUSTED"| Z6["truncated · abort"] + A10 -->|"FRONTIER_CAP_HIT"| Z7["frontier_cap_hit · abort
-> whole search abandoned"] + A10 -->|"ADMITTED"| A11["push onto priority or pending queue"] + A9 --> A12 + A11 --> A12{"tag in batch_tags?"} + A12 -->|"yes"| Z8["JVMTI_VISIT_OBJECTS — descend one hop"] + A12 -->|"no"| Z9["return 0 — do not descend"] +
+ +
+ ALREADY_ADMITTED is the idempotency that makes re-enumeration cheap. + admitObject() short-circuits on any non-zero tag + (referenceChains.cpp:2022-2057), which is precisely why root enumeration can be re-run on + every pass without re-paying for the graph it already discovered. +
+
+ + +
+

7 · Root discovery and the chunked static-field sweep

+

Two disjoint sources of root-attached entries. IterateOverReachableObjects reports stack + locals, JNI handles, monitors and thread roots — but never static fields, which need their own sweep.

+ +
+flowchart TB + subgraph RE["A · IterateOverReachableObjects"] + direction TB + R1["heapRootCallback / stackRefCallback"] --> R2["translateHeapRootKind():
jvmtiHeapRootKind -> jvmtiHeapReferenceKind"] + R2 --> R3["admitObject(parent_tag = 0, depth = 0, root_kind)"] + R3 --> R4{"ALREADY_ADMITTED?"} + R4 -->|"yes"| R5["maybeUpgradeRootAttachedRootKind()
durability tie-break"] + R4 -->|"no"| R6["new root-attached frontier entry"] + end + + subgraph SF["B · admitStaticFieldRoots — chunked, cursor-resumed"] + direction TB + S1["GetLoadedClasses()"] --> S2["partition app classes
(non-null loader) to the front"] + S2 --> S3["chunk = [cursor, cursor + 512)"] + S3 --> S4["holder array filled in REVERSE chunk order
so HotSpot's LIFO descent visits ascending"] + S4 --> S5["one FollowReferences with static_field_seed = true
empty batch_tags -> exactly one hop past each class"] + S5 --> S6{"truncated?"} + S6 -->|"yes"| S7["mark lap truncated;
cursor = chunk_start + classes_visited - 1
(redo the partial class)"] + S6 -->|"no"| S8["cursor = chunk_end"] + S7 --> S9 + S8 --> S9{"cursor reached class_count?"} + S9 -->|"yes"| S10["lap wrap: cursor = 0;
cycle_complete only if no chunk truncated"] + S9 -->|"no"| S11["resume here next pass"] + end +
+ +
+ Why chunking is not an optimisation but a correctness fix. On a JVM with ~34k loaded classes a single + FollowReferences over every class cannot finish inside a 5–50 ms deadline — the observed behaviour was + truncated = 1 on 275 of 275 passes with 0–1 edges admitted. Because a single-call sweep restarts at + class 0 every time, every class past the deadline point was permanently unreachable + (referenceChains.h:1936-1960). +
+ +

Root-kind durability

+

A root-attached entry records why it is reachable. When a more durable root is later observed + admitting the same object, the recorded kind is upgraded rather than keeping whichever root happened to be enumerated + first (referenceChains.h:249-278).

+ + + + + + + +
TierKindsMeaning
3 — most durableSTATIC_FIELD, SYSTEM_CLASSGenuine retention evidence.
2JNI_GLOBALDurable, but owned outside the JVM heap.
1 — transientMONITOR, STACK_LOCAL, JNI_LOCAL, THREAD, OTHER"First observed via", not "rooted by". THREAD and OTHER have no documented tier and are conservatively bucketed here.
+

Two repair operations exist because a depth comparison alone cannot express both cases: + improveChain() replaces a shallow root-attached entry when a strictly deeper path reaches it, and + reparentToDurableRoot() handles the equal-depth case — a depth-1 entry parented to a transient root is + re-parented to a durable one. Without the latter, the real hotdog shape (a static singleton collection at depth 0, its + elements at depth 1) would keep a stack-local parent forever, because the depths tie.

+
+ + +
+

8 · State machines

+

Two independent state machines: one per frontier entry, one per search.

+ +

Per-entry: FrontierEntryState

+
+stateDiagram-v2 + direction LR + [*] --> FRONTIER: "insert() from admitObject(), leak-tag interception or canary pruning" + FRONTIER --> EXPANDED: "markExpanded() — its batch's FollowReferences completed" + FRONTIER --> ABANDONED: "clear() — GetObjectsWithTags did not resolve the tag (object died)" + note right of EXPANDED: "rotation re-queues the tag, and
the re-walk marks it EXPANDED again" + FRONTIER --> EDGE: "markEdge() — reconstructChain() walked through this hop" + EXPANDED --> EDGE: "markEdge()" + EDGE --> EXPANDED: "rotation re-expansion overwrites the state" + EXPANDED --> ABANDONED: "releaseSearchTags() at search end" + EDGE --> ABANDONED: "releaseSearchTags() at search end" + ABANDONED --> [*]: "resetForRestart() zeroes _table_size — every slot becomes un-inserted" +
+
+ ABANDONED means "the JVMTI tag was released", not "the record was discarded". + releaseSearchTags() leaves parent_tag, referrer_klass, depth and + root_kind intact, so reconstructChain() keeps working from memory after the search ends + (referenceChains.h:1966-1975). +
+ +

Per-search: SearchState

+
+stateDiagram-v2 + direction TB + [*] --> RUNNING: "first shouldRunPass() that clears canAffordNewSearch()" + + RUNNING --> ABANDONED_FC: "frontier_cap_hit — checked first, wins over everything" + RUNNING --> COMPLETED_G: "not truncated AND no watched leak klass" + note right of RUNNING: "not truncated BUT a watch is active —
stays RUNNING, rotation keeps re-observing" + RUNNING --> ABANDONED_TTL: "30 passes with no frontier growth AND not isUrgent()" + RUNNING --> COMPLETED_C: "all canary candidates found" + RUNNING --> ABANDONED_CS: "candidates outstanding AND 30 passes without frontier growth AND canary stall past canaryStuckPassLimit()" + + ABANDONED_FC --> RELEASE + ABANDONED_TTL --> RELEASE + ABANDONED_CS --> RELEASE + COMPLETED_G --> RELEASE + COMPLETED_C --> RELEASE + + note right of RELEASE: "releaseSearchTags() failed — shouldRunPass()
returns true to retry; restart stays blocked" + RELEASE --> RUNNING: "_tags_released AND canAffordNewSearch() -> restartSearch()" + + state "ABANDONED / FRONTIER_CAP" as ABANDONED_FC + state "ABANDONED / TTL (no-progress)" as ABANDONED_TTL + state "ABANDONED / CANARY_STUCK" as ABANDONED_CS + state "COMPLETED (graph exhausted)" as COMPLETED_G + state "COMPLETED (candidates found)" as COMPLETED_C + state "tag release + canary reset" as RELEASE +
+ + + + + + + + + +
ReasonValueSuppressed by isUrgent()?Rationale
NONE0—Not abandoned.
FRONTIER_CAP1NoThe metadata table is full; nothing further can be admitted at all.
TTL2YesA search still making real progress must not be killed just because the process is close to OOM.
CANARY_STUCK3NoA candidate chase with zero discovery progress is provably not converging; continuing at urgency-boosted budget only burns pause budget the dying process needs.
+
+ + +
+

9 · Termination, tag release and restart

+

Six checks, evaluated in a fixed priority order at the end of every pass + (referenceChains.cpp:3765-3856). Only the first match applies.

+ +
+flowchart TB + T(["end of runPass()"]) --> C1{"frontier_cap_hit"} + C1 -->|"yes"| A1["ABANDONED / FRONTIER_CAP
enqueue abandoned event"] + C1 -->|"no"| C2{"no pending frontier
AND _watched_leak_klass_count == 0"} + C2 -->|"yes"| A2["COMPLETED — graph exhausted
within the caps"] + C2 -->|"no"| C3{"no pending frontier
but a watch is active"} + C3 -->|"yes"| A3["stay RUNNING — rotation keeps
re-observing mutated fields"] + C3 -->|"no"| C4{"_passes_since_last_progress >= 30
AND NOT isUrgent()"} + C4 -->|"yes"| A4["ABANDONED / TTL"] + C4 -->|"no"| C5{"all candidates found"} + C5 -->|"yes"| A5["COMPLETED + counter
REFERENCE_CHAIN_CANDIDATES_FOUND"] + C5 -->|"no"| C6{"candidates outstanding
AND 30 passes no frontier growth
AND canary stall >= canaryStuckPassLimit()"} + C6 -->|"yes"| A6["ABANDONED / CANARY_STUCK
_canary_stuck_restart_count++"] + C6 -->|"no"| A7["stay RUNNING"] + + A1 --> R + A2 --> R + A4 --> R + A5 --> R + A6 --> R + R["releaseSearchTags():
GetObjectsWithTags over every non-ABANDONED tag,
SetTag(obj, 0), then mark all ABANDONED"] --> R2{"batch call succeeded?"} + R2 -->|"no"| R3["_tags_released = false — mark NOTHING;
a failed batch says nothing about liveness,
and rewinding _next_tag while a live object
still carries a tag would break tag uniqueness"] + R2 -->|"yes"| R4["_tags_released = true
clear canary marker tags · zero candidate state
reset _canary_stuck_restart_count unless CANARY_STUCK"] +
+ +

Escalating patience

+

canaryStuckPassLimit() = CANARY_NO_PROGRESS_PASS_LIMIT << min(_canary_stuck_restart_count, 8) + (referenceChains.h:2389-2393). Repeated CANARY_STUCK abandons double the stall + tolerance, up to 256×. _canary_stuck_restart_count is deliberately not reset by + restartSearch() — it must survive restarts or the escalation could never widen.

+ +
+

Escalation: canaryStuckPassLimit(n) = 3 << min(n, 8)

+

Each consecutive CANARY_STUCK abandon doubles the stall tolerance for the + next attempt at the same candidate chase, up to the MAX_CANARY_STUCK_BACKOFF_SHIFT = 8 cap + (restart count 8 and 9 tolerate the same 768 passes — the shift has saturated).

+
+
+ +

What a restart resets, and what it does not

+
+
+

Reset (restartSearch())

+

_frontier->resetForRestart() (slot occupancy only, the allocation survives) · _next_tag = 1 · + _search_started = false · state back to RUNNING · both expand queues and the membership index · + leak-signature maps · _leak_tags_assigned/_resolved · _last_pass_* · _passes_run · + the resolved/static-field class counts.

+
+
+

Persists across a restart

+

The class-tag table and its allocator · _resolved_chains · _watched_leak_klass_ids · + _canary_stuck_restart_count · the rotation cursors and the static-field sweep cursor · + _pause_pid, _effective_budget, _effective_cadence_ns · the cached + java/lang/Object global ref · the FrontierTable allocation and capacity.

+
+
+

restartSearch() also spends the finished search's accumulated + _search_pain_ms into _safepoint_pain_budget before zeroing it + (referenceChains.cpp:1123) — a cheap search may restart again soon, an expensive one must wait + proportionally longer. It asserts on _tags_released.

+
+ + +
+

10 · Pacing: PID controller, budget borrowing, pain budgets

+

The configured constants are ceilings and baselines, not literal per-pass values. What each pass + actually spends is decided by a feedback loop over the previous pass's measured in-safepoint time.

+ +
+flowchart LR + M["measure the pass:
pass_wall_ticks total,
safepoint_ticks inside
IterateOverReachableObjects / FollowReferences"] --> SPLIT{"split"} + SPLIT -->|"safepoint ms"| PID["_pause_pid.compute(pass_ms)
target = _effective_pause_target_ms
positive signal = came in UNDER target"] + SPLIT -->|"safepoint ms"| PAIN1["_search_pain_ms +=
-> gates the NEXT search"] + SPLIT -->|"non-safepoint ms"| PAIN2["_cpu_pain_budget.spend()
-> gates the NEXT pass"] + + PID --> BORROW{"pass_ms at or under 50% of target
for 5 consecutive passes?"} + BORROW -->|"yes"| B1["_borrowed_budget += _budget * 0.25
capped at 3 * _budget"] + BORROW -->|"no"| B2["_borrowed_budget = 0 immediately"] + B1 --> CL + B2 --> CL["ceiling = _budget + _borrowed_budget
floor = min(2000, ceiling)
_effective_budget = clamp(prev + signal)"] + CL --> OV{"overflow = desired - clamped"} + OV -->|"negative — still over target at the floor"| W["widen _effective_cadence_ns
by 1ms per overflow edge, up to 4s"] + OV -->|"positive"| S["shorten _effective_cadence_ns,
down to 10ms"] + OV -->|"== 0"| K["leave the cadence alone"] + W --> NEXT + S --> NEXT + K --> NEXT["next pass: expand_budget = _effective_budget,
deadline = _effective_pause_target_ms,
sleep/gate = _effective_cadence_ns"] +
+ +
+

Clamp function: effective_budget = clamp(desired, floor, ceiling)

+

Illustrative baseline _budget = 800 edges/pass (below + MIN_EFFECTIVE_BUDGET = 2000). With no borrowed headroom, floor = min(2000, ceiling) = ceiling + — the clamp range collapses to a single point and _effective_budget is pinned regardless of the PID + signal. Once _borrowed_budget pushes the ceiling past 2000 (shown at 3× budget, the borrowing + ceiling), a real band opens up and the floor becomes MIN_EFFECTIVE_BUDGET itself.

+
+
+ +
+
+

Inverted sign convention

+

Unlike ObjectSampler / MallocTracer / RateLimiter, which subtract the PID signal from an interval, this + controller adds it to a budget: a positive signal means the pass came in under target, so the next one may + do more work (referenceChains.h:2002-2012).

+
+
+

One compute() is one pass

+

sampling_window = 1 and time_delta_coefficient = 1.0, because a pass is not a fixed + real-time window like the other three usages assume. Gains are 10/1/2 — deliberately not copied from the shared triple.

+
+
+

Root-enum passes are excluded

+

They spend the deliberately oversized _first_pass_budget; feeding that into the PID would throttle + every cheap expansion pass that follows (referenceChains.cpp:3713-3728).

+
+
+

Borrowing is lost instantly

+

A single pass that is not comfortably under target zeroes both the warm-up streak and the whole accumulated + borrow. Earning headroom takes 5 passes; losing it takes one.

+
+
+ +

The two leaky buckets

+ + + + + + + + +
_safepoint_pain_budget_cpu_pain_budget
GatesStarting or restarting a searchRunning an individual pass
ChargedOnce per finished search, in restartSearch(), with _search_pain_msEvery pass, with the non-safepoint remainder
Refill ratepain_budget_percent / 100Same base, times 1× / 15× (covering) / 100× (emergency)
Read atcanAffordNewSearch()shouldRunPass(), RUNNING branch
+
+ Degenerate configuration: a refill rate of exactly 0.0 never drains + (painBudget.h:46-49), so once anything has been spent the bucket blocks permanently. +
+ +
+

Leaky bucket: _safepoint_pain_budget over time

+

Illustrative simulation, not a captured log — refill rate 2% (an example + pain_budget_percent), two searches finishing at t=0 and t=1600ms and spending their accumulated + _search_pain_ms (60ms, then 45ms) into the bucket. canAffordNewSearch() only returns + true where the curve touches the zero baseline — everywhere the shaded area is above zero, a new search or + restart is blocked.

+
+
+ +

Auto-tuning at start()

+

autoTuneDefaults() (referenceChains.cpp:360-462) fills in only the + knobs the operator did not set explicitly, from max heap size and processor count:

+ + + + + + + + + + +
KnobFormula
Edge budgetDEFAULT * sqrt(heap_mib / 512), clamped to [default, max] — keeps the pause proportional to sqrt(heap)
First-pass budgetbudget * 10, capped
TTLDEFAULT * (heap_mib / 512), clamped to [default, 30 min]
Frontier capscaled by the budget ratio in floating point (integer division here would undershoot by ~20%)
Pause targetDEFAULT * (1 + (nprocs-1)/3), capped at 50 ms
Pain budget %DEFAULT * (1 + (nprocs-1)/4), capped at 5%
+
+ + +
+

11 · From frontier to emitted chain

+

The frontier table doubles as a degenerate edge store: a chain is just the transitive closure of + parent_tag links from a target back to a root-attached entry.

+ +
+flowchart TB + T["target tag"] --> L{"lookup(tag) succeeds?"} + L -->|"no"| F1["return false — never fabricate a partial chain"] + L -->|"yes"| P["push entry.referrer_klass onto the chain
markEdge(tag)
root_kind = entry.root_kind
tag = entry.parent_tag"] + P --> C{"tag == 0?"} + C -->|"no, and hops <= maxCapacity()"| L + C -->|"no, bound exceeded"| F2["corrupt or cyclic -> return false"] + C -->|"yes"| OUT["chain in leaf -> root order
out_root_kind = the root-attached entry's kind"] + OUT --> EV["buildChainEvent():
targetTag = leak_tag if set, else the frontier tag
depth · root_kind · chain"] + EV --> JFR["JFR ReferenceChain event"] + JFR --> JOIN["backend joins ReferenceChain.targetTag
to HeapLiveObject.leakTag"] +
+ +

FrontierEntry fields

+ + + + + + + + + + + +
FieldPurpose
parent_tagTag of the entry that discovered this one; 0 means root-attached. The only link the reconstruction walks.
referrer_klassStringDictionary id of this object's class name, resolved ahead of time — GetClassSignature is illegal inside a heap callback.
depthHop count from the root. Drives the hop cap and improveChain()'s "deeper wins" comparison.
stateFRONTIER / EXPANDED / EDGE / ABANDONED.
leak_tagLivenessTracker's per-instance tag, copied in at admission. Becomes ReferenceChain.targetTag, the backend's join key. 0 = ordinary BFS admission.
root_kindjvmtiHeapReferenceKind of the admitting edge, meaningful only when parent_tag == 0.
class_tagRaw negative JVMTI class tag — stable across StringDictionary regeneration, unlike referrer_klass. Needed by the retroactive leak-accumulation seed scan, which runs long after the live callback is gone.
+
+ No jobject is ever retained. Holding a live handle would defeat the entire point of using + non-retaining JVMTI tags for frontier identity — the walk would keep the leak alive. +
+ +

The canary path

+

Candidate instances are pre-tagged with distinct negative marker tags + (MARKER_TAG_BASE = -(1<<62)) applied to a specific representative object — matching by class alone + would record a chain for an unrelated, possibly short-lived instance of the same class. When the walk hits a marker it + records the chain link and the found bit, but does not descend. + buildCanaryChainEvent() therefore starts from the recorded parent tag (the negative marker itself is + rejected by lookup()), then prepends the candidate's own class and reverses to leaf→root + (referenceChains.h:2647-2712).

+

Beyond the representative, any object of a watched class discovered by the walk is auto-recorded, up to + MAX_DISCOVERED_INSTANCES_PER_CLASS = 8 per slot — each instance's chain is independently useful, since + different instances may be retained by different paths.

+
+ + +
+

12 · Constants reference

+

Values as they stand on this branch. Several are explicitly marked provisional / unbenchmarked in source.

+ +

Leak detection — livenessTracker.h

+ + + + + + + + + + + + + + +
ConstantValueRole
MAX_KLASS_POPULATION_ENTRIES256Tracked klasses.
KLASS_POPULATION_RING_SIZE30Trend window, in GC epochs.
KLASS_POPULATION_MIN_FILL_FOR_TREND10Minimum fill before any slope is trusted.
LEAK_GROWTH_REL_MIN / _ABS_MIN0.15 / 1Qualifying growth bar.
LEAK_TREND_HYSTERESIS_BASE5Consecutive qualifying epochs required.
LEAK_TREND_HYSTERESIS_CORROBORATED3Lowered bar when the heap floor is rising.
TID_TREND_RING_SIZE / MIN_FILL16 / 6Per-thread trend window.
MAX_TID_TRENDS8Threads tracked per klass.
MAX_LEAK_CANDIDATES5Top-k candidate slots.
HEAP_FLOOR_RECENT_HALF_MIN_FILL5Corroboration window for the OOM projection.
+ +

Scheduling and urgency — referenceChains.h

+ + + + + + + + + + + + + +
ConstantValueRole
PASS_CADENCE_NS1 sBaseline cadence and ramp anchor.
OOM_URGENT_THRESHOLD_S300 sLatch arm bar for isUrgent().
OOM_URGENT_RELEASE_S600 s (2×)Latch release bar.
URGENT_RELEASE_CONSECUTIVE5Clear readings needed to unlatch.
OOM_RAMP_START_S1800 sStart of the unlatched exponential ramp in threadLoop().
URGENT_PAUSE_TARGET_MS100 msRamp ceiling for the pause target.
URGENT_CADENCE_NS10 msRamp floor for the cadence.
CANARY_PAIN_BUDGET_COVERING_MULTIPLIER15×CPU refill while chasing candidates.
CANARY_PAIN_BUDGET_REFILL_MULTIPLIER100×CPU refill in the emergency case.
+ +

Walk, budgets and termination — referenceChains.h

+ + + + + + + + + + + + + + + + + + + + +
ConstantValueRole
AUTO_FIRST_PASS_BUDGET_MULTIPLIER / _CAP50 / 200000Auto-scaled first-pass edge budget.
ROOT_ENUM_MIN_INTERVAL_NS2 sMinimum spacing between root enumerations.
STATIC_FIELD_SWEEP_CHUNK_CLASSES512Classes per static-field chunk.
STATIC_FIELD_SWEEP_NON_STATIC_CAP_PER_CLASS32Non-static edges admitted per class per lap.
MIN_EFFECTIVE_BUDGET2000Budget floor under PID control.
MIN/MAX_EFFECTIVE_CADENCE_NS10 ms / 4 sCadence clamp.
CADENCE_NS_PER_EDGE_OVERFLOW1 ms / edgeHow budget overflow converts into cadence widening.
BORROW_WARMUP_PASSES / ceiling5 / 3× _budgetBudget borrowing.
PRIORITY_EXPAND_CAP1024Fast-lane queue cap (2048-slot membership index).
NO_PROGRESS_PASS_LIMIT30Passes without frontier growth before the TTL abandon.
CANARY_NO_PROGRESS_PASS_LIMIT3 — marked TEMP: was 30Base of the canary stall limit, and the emergency threshold.
MAX_CANARY_STUCK_BACKOFF_SHIFT8Escalation cap — up to 256× the base.
MARKER_TAG_BASE-(1<<62)Canary marker tag namespace.
LEAK_TAG_BASE / POOL_SIZE0x40000000 / 256LivenessTracker leak-tag pool.
MAX_DISCOVERED_INSTANCES_PER_CLASS8Auto-recorded instances per watched class.
INITIAL_TABLE_CAPACITY1024Frontier table start size; doubles to the configured cap.
+
+ +
+ +
+ Generated from source on branch jb/reference-chains-pi. Line references were accurate at extraction time; + re-verify against the working tree before quoting them. +
+ + + + + diff --git a/doc/architecture/LiveHeapReferenceChains-Implementation.md b/doc/architecture/LiveHeapReferenceChains-Implementation.md index b016cd10d3..73658f8ae3 100644 --- a/doc/architecture/LiveHeapReferenceChains-Implementation.md +++ b/doc/architecture/LiveHeapReferenceChains-Implementation.md @@ -1,6 +1,6 @@ # Live Heap Reference Chains — As-Built Implementation Reference -**Status:** matches `jb/reference-chains` as of 2026-09 +**Status:** Implemented as of 2026-09 **Jira:** [PROF-15341](https://datadoghq.atlassian.net/browse/PROF-15341) This document is the detailed, as-built reference for the reference-chain walk diff --git a/doc/architecture/LiveHeapReferenceChains.md b/doc/architecture/LiveHeapReferenceChains.md index 5276da3f6b..dea8f10bb8 100644 --- a/doc/architecture/LiveHeapReferenceChains.md +++ b/doc/architecture/LiveHeapReferenceChains.md @@ -6,9 +6,8 @@ ## Implementation status -The "Chosen design" section below has been implemented following -`LiveHeapReferenceChains-ImplementationPlan.md` (kept locally, not committed) -(Phases 0-7). It is off by default; the shipping switch is the `referencechains` argument +The "Chosen design" section below has been implemented. It is off by default; the +shipping switch is the `referencechains` argument parsed by `Arguments` (`arguments.cpp`'s `CASE("referencechains")`), e.g. `referencechains=true:hops=64:budget=2000:ttl=60000:framecap=65536`. @@ -60,10 +59,10 @@ Read this status note alongside the actual code before relying on it, not instea trend, then asserts on the `datadog.ReferenceChain` event `pollWatchedTargets()` produces - a real end-to-end exercise of this whole mechanism against a live JVM, not a synthetic frontier fixture. -- **Phase 5's tuning defaults are provisional, not empirically finalized.** The hop cap, +- **The tuning defaults are provisional, not empirically finalized.** The hop cap, per-pass budget, TTL, and frontier-size cap (`arguments.h`'s `DEFAULT_REFERENCE_CHAINS_*` constants) are explicitly-labeled placeholders; no benchmark against this codebase has - run yet (see Open Question 2 below and the implementation plan's Phase 5). + run yet (see Open Question 2 below). ## Goal @@ -390,8 +389,8 @@ retaining path, not a claim about the object's current exact retention state. representative heap shapes and per-hop fan-out, not a guess; `LivenessTracker`'s flat-sample-rate sizing formula does not transfer to a graph-search frontier. **Not resolved — provisional defaults only, no measurement has occurred.** The - implementation currently ships explicitly-labeled "provisional default pending Phase 5 - empirical tuning" constants (`arguments.h`: `DEFAULT_REFERENCE_CHAINS_HOP_CAP = 200`, + implementation currently ships explicitly-labeled "provisional default pending empirical + tuning" constants (`arguments.h`: `DEFAULT_REFERENCE_CHAINS_HOP_CAP = 200`, citing this doc's own JFR ~200-hop/100-100 precedent; `DEFAULT_REFERENCE_CHAINS_BUDGET = 1000`; `DEFAULT_REFERENCE_CHAINS_TTL_MS = 60000`; `DEFAULT_REFERENCE_CHAINS_FRONTIER_CAP = 65536`, sized as a fraction of `LivenessTracker::MAX_TRACKING_TABLE_SIZE` rather than @@ -399,15 +398,12 @@ retaining path, not a claim about the object's current exact retention state. `FrontierTable::INITIAL_TABLE_CAPACITY = 1024` and `ReferenceChainTracker::PASS_CADENCE_NS` = 1 s). These let the subsystem run and be tested end-to-end, but none are backed by a benchmark against this codebase — do not - describe them as measured. The real resolution path is - `LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed), - which specifies the JMH/async-profiler matrix and decision rule Phase 5 still needs to - execute; this question stays open until that plan is actually run. + describe them as measured. The real resolution path is a JMH/async-profiler benchmark + matrix with an explicit decision rule, still to be executed; this question stays open + until that benchmark work is actually run. **Pause-time-SLO feedback loop — SHIPPED, reusing the existing `PidController`.** - Implemented in - `LiveHeapReferenceChains-RemainingWorkPlan.md` (kept locally, not committed)'s - Phase D (`ReferenceChainTracker::updatePacing()`, `referenceChains.cpp`). This does not + Implemented as `ReferenceChainTracker::updatePacing()` (`referenceChains.cpp`). This does not replace the hop/TTL/frontier-cap constants raised in the first half of this question — only the per-pass edge-count budget and the pass cadence, per the shipped mechanism below. - New config sub-option `referencechains=...:pausetarget=` (`arguments.cpp`'s @@ -427,9 +423,8 @@ retaining path, not a claim about the object's current exact retention state. single/low-double-digit in magnitude, unlike the shared triple's event-count scale (`referenceChains.cpp`'s `start()`, inline comment on each gain). Gain *convergence* is verified by gtest (three `ReferenceChainsTest` cases: steady-state at the ceiling, over- - ceiling, under-ceiling — see Phase D's exit criteria below), not by a live benchmark - against representative heap shapes; that remains a - `LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed) item, + ceiling, under-ceiling), not by a live benchmark + against representative heap shapes; that benchmark work remains open, not fully closed by this mechanism landing. - Measurement point: `runPass()` (`referenceChains.cpp`) times its own root `IterateOverReachableObjects` call (first pass) or `expandFrontier()`'s @@ -454,10 +449,10 @@ retaining path, not a claim about the object's current exact retention state. - The hop cap and the frontier-size hard cap (Termination section) are untouched by this mechanism — they stay fixed correctness/memory-safety bounds, not controller-tuned, exactly as this question originally specified. - - One known, deliberate scope limit carried over from the plan: `buildAbandonedEvent()`'s + - One known, deliberate scope limit: `buildAbandonedEvent()`'s `datadog.ReferenceChainAbandoned` event still reports the static config ceiling `_budget`, - not the adaptive `_effective_budget` — changing that event's semantics was out of Phase D's - stated scope. + not the adaptive `_effective_budget` — making that event track the adaptive value was + deliberately left out of scope. 3. Decide the sample-batching policy: one incremental search per live-heap sample, or batched multi-target BFS sharing a single frontier walk (batching amortizes better but couples unrelated samples' termination conditions together). @@ -468,15 +463,13 @@ retaining path, not a claim about the object's current exact retention state. chain for a specific tag is a separate, read-only step (`buildChainEvent(target_tag, ...)`) applied after (or during) that one shared search - closer in spirit to "batched" (one frontier walk can answer for many targets) than "one search per sample", but arrived at - by omission (the target-sample feed did not exist at that time, see the implementation - plan's Phase 7 report) rather than a deliberate batching design. + by omission (no target-sample feed existed) rather than a deliberate batching design. Whether this generalizes to true multi-target batching (explicit seeding from multiple samples, coordinated termination) is still open and deferred, consistent with this question's original framing. - **Target-selection policy — SHIPPED (positive population-slope ranking).** Implemented in - `LiveHeapReferenceChains-RemainingWorkPlan.md` (kept locally, not committed)'s - Phases A-C. The missing piece above was *which* tag(s) `buildChainEvent()` should + **Target-selection policy — SHIPPED (positive population-slope ranking).** + The missing piece above was *which* tag(s) `buildChainEvent()` should reconstruct for. As shipped: per klass, `LivenessTracker` tracks a rolling window of its live tracked-instance population count, sampled once per `LivenessTracker::cleanup_table()` epoch advance (the same GC-epoch cadence that already recomputes survivor status, @@ -539,9 +532,8 @@ retaining path, not a claim about the object's current exact retention state. trend) but the *reconstruction target* does not need to be the exact instance that built up the trend — any currently-live tracked instance of the flagged klass is evidence of the same leak. **The bridging step is a READ, not a `SetTag` write** (a correction to - this doc's original proposal, found while grounding - `LiveHeapReferenceChains-RemainingWorkPlan.md` (kept locally, not committed); - see its "Correction to the design doc's Open Question 3 mechanism"). Pre-`SetTag`ing a + this doc's original proposal, found while grounding the implementation): + Pre-`SetTag`ing a candidate before the forward walk reached it would make `heapReferenceCallback()`'s `*tag_ptr == 0` branch — the *only* branch that records `parent_tag`/`depth` — skip it, yielding an empty/root chain. Instead `ReferenceChainTracker::pollWatchedTargets()` @@ -583,13 +575,12 @@ retaining path, not a claim about the object's current exact retention state. labeled provisional in `referenceChains.h`) has elapsed, whichever comes first. No safepoints-per-second/per-pause-duration measurement backs the 1-second cadence value - it was chosen only so an idle search still makes progress without polling tightly. The - cost model this question actually asks for is still open, deferred to Phase 5's - benchmark plan (`LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed)), - which has not been run. + cost model this question actually asks for is still open, deferred to a benchmark + run that has not happened yet. **SHIPPED — folded into Open Question 2's pause-time-SLO feedback loop, not solved separately.** Implemented in the same `ReferenceChainTracker::updatePacing()` - (`referenceChains.cpp`, Phase D) described under Open Question 2: one `PidController` + (`referenceChains.cpp`) described under Open Question 2: one `PidController` `compute()` call per pass drives both that question's budget adjustment and this question's cadence adjustment from the single measured per-pass safepoint duration, rather than two independently-tuned mechanisms. `shouldRunPass()` and `threadLoop()` now compare against @@ -606,5 +597,4 @@ retaining path, not a claim about the object's current exact retention state. since it is the same controller instance. The cost-modeled "how many safepoints/sec is acceptable" question this Open Question originally asked for is answered structurally (the controller widens cadence exactly when passes are running long relative to the configured - ceiling) rather than by a specific measured number — that number is still a - `LiveHeapReferenceChains-BenchmarkPlan.md` (kept locally, not committed) item. + ceiling) rather than by a specific measured number — that benchmark work is still open. diff --git a/doc/libretti/2026-04-15-free-text-it-seems-like-java-profiler-is-interacting-badly-with-wasmti.md b/doc/libretti/2026-04-15-free-text-it-seems-like-java-profiler-is-interacting-badly-with-wasmti.md new file mode 100644 index 0000000000..20e2ff91d4 --- /dev/null +++ b/doc/libretti/2026-04-15-free-text-it-seems-like-java-profiler-is-interacting-badly-with-wasmti.md @@ -0,0 +1,15 @@ +--- +spec_id: REQ-text-13982 +source: text +source_ref: (free text) +title: "It seems like java-profiler is interacting badly with wasmtime, when the later is loaded in the profiled JVM" +status: draft +clarity_score: null +created: 2026-04-15 +implementing_session: null +implemented_pr: null +--- + +# It seems like java-profiler is interacting badly with wasmtime, when the later is loaded in the profiled JVM + +It seems like java-profiler is interacting badly with wasmtime, when the later is loaded in the profiled JVM. It is either crashing or deadlocking, crashing much more often. When the app is running with just wasmtime library, it seems fine. I want to create a reproducer app where this crash can be debugged properly. diff --git a/doc/libretti/2026-04-20-scp-1154-crash-datadog-agent-7751-causing-jvm-crash-on-ibm-was-90516.md b/doc/libretti/2026-04-20-scp-1154-crash-datadog-agent-7751-causing-jvm-crash-on-ibm-was-90516.md new file mode 100644 index 0000000000..6ee1ad36f8 --- /dev/null +++ b/doc/libretti/2026-04-20-scp-1154-crash-datadog-agent-7751-causing-jvm-crash-on-ibm-was-90516.md @@ -0,0 +1,95 @@ +--- +spec_id: REQ-SCP-1154 +source: jira +source_ref: SCP-1154 +title: "[Crash] Datadog Agent 7.75.1 causing JVM crash on IBM WAS 9.0.5.16 after installation" +status: implemented +clarity_score: null +created: 2026-04-20 +implementing_session: impl-20260420-123402 +implemented_pr: https://github.com/DataDog/java-profiler/pull/492 +--- + +# [Crash] Datadog Agent 7.75.1 causing JVM crash on IBM WAS 9.0.5.16 after installation + +## **Customer Issue Summary** + +* **Environment** + + * Platform: IBM WebSphere Application Server (WAS) + * WAS Version: 9.0.5.16 + * OS: RHEL 9.6 + * Java: OpenJDK 1.8.0_462 + * JVMs per server: 5 + * Agent versions involved: + + * Working: **7.75.1** + * Failing (latest upgrade): **7.77.3** + + * Java Tracer version: **1.61.0** + +* **Problem Description** + + * JVM crashes and restart failures occur across all WAS nodes + * Issue started **after upgrading Datadog Agent** + * No CPU or memory spikes observed + * No application code changes + * Removing Datadog Agent resolves the issue + * Impact: **Release blocking (MyAMP application)** + + +## **Key Findings / Investigation (TL;DR)** + +* Issue **reproduced and isolated** +* Root trigger identified as: `-Ddd.profiling.enabled=true` +* Behavior comparison: + + * OK : **Agent 7.75.1** → Profiling enabled → **Works fine** + * KO: **Agent 7.77.3** → Profiling enabled → **JVM crash** + * OK: **Agent 7.77.3** → Profiling disabled → **Stable** + +* Additional validation: + + * Agent alone (no tracer/profiling) → stable + * Confirms issue is **not core agent**, but likely: + + * Java tracer / profiler interaction + * Profiler regression or incompatibility with WAS/JVM + + +* Scope: + + * All WAS nodes affected consistently + * Reproducible on demand + + +## **Configuration Used** + +* JVM arguments: + + * `-javaagent:/opt/datadog-agent/dd-java-agent.jar` + * `-Ddd.service={}` + * `-Ddd.version=9.0.5.16` + * `-Ddd.logs.injection=true` + * `-Ddd.profiling.enabled=true` + * `-Ddd.env=` + + +## **Input Provided from the CX** + +* Agent flare after reproduction : +* Host where issue was reproduced:`i-06765a9af5f75256d` \[[partlow](https://support-admin.us1.prod.dog/admin/switch_handle_get/org_id/380139?next_url=%2Finfrastructure%3Fhost%3Di-06765a9af5f75256d%26tags%3Dapplicationname%253Aidam_tam_-_non-production&reason=zd-2818967)\] +* Note: + + * Application was previously removed and not fully re-instrumented + * Service visibility in APM currently limited + +* **Not available:** + + * JVM crash dumps / hs_err logs + * Full application logs at crash time + + +**Notes: this is very urgent as this is blocking their production release** + +**(** @Reshmi Anand hey I’m tagging you here in case you took this one - it’s the case Agash saw with you in OH - thanks) diff --git a/doc/libretti/2026-04-22-prof-14332-wallclockasgct-collectthreads-skips-every-other-thread-in-th.md b/doc/libretti/2026-04-22-prof-14332-wallclockasgct-collectthreads-skips-every-other-thread-in-th.md new file mode 100644 index 0000000000..6b739936b0 --- /dev/null +++ b/doc/libretti/2026-04-22-prof-14332-wallclockasgct-collectthreads-skips-every-other-thread-in-th.md @@ -0,0 +1,75 @@ +--- +spec_id: REQ-PROF-14332 +source: jira +source_ref: PROF-14332 +title: WallClockASGCT collectThreads skips every other thread in the no-filter fallback +status: draft +clarity_score: null +created: 2026-04-22 +implementing_session: null +implemented_pr: null +--- + +# WallClockASGCT collectThreads skips every other thread in the no-filter fallback + +## Summary + +In `WallClockASGCT::timerLoop`, the `collectThreads` lambda's no-filter +fallback advances the `ThreadList` iterator twice per loop body, silently +skipping every other thread when the wallclock thread filter is disabled. + +## Location + +`ddprof-lib/src/main/cpp/wallClock.cpp:180-190` + +```cpp +ThreadList *thread_list = OS::listThreads(); +while (thread_list->hasNext()) { + int tid = thread_list->next(); // <-- read #1 + // Don't include the current thread + if (tid != OS::threadId()) { + tids.push_back(tid); + } + tid = thread_list->next(); // <-- read #2 (unguarded!) +} +delete thread_list; +``` + +The second `thread_list->next()` runs unconditionally at the tail of each +iteration; the next `hasNext()` check at the top of the loop sees the +cursor already advanced past the last-handled tid. Net effect: only the +1st, 3rd, 5th, ... threads in `/proc/self/task` are pushed into `tids`; +the rest are dropped silently. + +Additionally, when the list has an odd count, the final `next()` call +reads past the end of the array — `next()` in the Linux implementation +returns `_thread_array[_index++]` without a bounds check, so a caller +running against a short list can read garbage. + +## Impact + +When `threadFilter->enabled()` is false (no context-based wallclock +filtering), every wallclock tick samples only half the eligible threads, +producing under-sampled wallclock profiles that aren't diagnosable from +the output. + +The typical `enabled()` path (context filter active) is unaffected; this +only bites on the fallback. + +## Fix + +Remove the spurious extra `tid = thread_list->next();` at line 187. +The `while (thread_list->hasNext()) { tid = thread_list->next(); ... }` +pattern is correct — one `next()` per iteration. + +## Discovery + +Flagged by a muse chorus review of PR-to-be [jb/signal-origin-validation +branch](https://github.com/DataDog/java-profiler/tree/jb/signal-origin-validation) +(hypnos-augur finding #15) as an unrelated out-of-scope pre-existing bug. + +## Scope + +Pre-existing — not introduced by the signal-origin-validation work. +Filed separately so it can land on its own small PR without blocking that +work or being lost in the review. diff --git a/doc/libretti/2026-05-07-prof-14549-sigsegv-in-recordingwriteclasses.md b/doc/libretti/2026-05-07-prof-14549-sigsegv-in-recordingwriteclasses.md new file mode 100644 index 0000000000..07abd1a8a6 --- /dev/null +++ b/doc/libretti/2026-05-07-prof-14549-sigsegv-in-recordingwriteclasses.md @@ -0,0 +1,52 @@ +--- +spec_id: REQ-PROF-14549 +source: jira +source_ref: PROF-14549 +title: "SIGSEGV in Recording::writeClasses" +status: draft +clarity_score: null +created: 2026-05-07 +implementing_session: null +implemented_pr: null +--- + +# SIGSEGV in Recording::writeClasses + +``` +Dictionary::collect(std::map, std::allocator > >&, DictTable*)+0x2b +Dictionary::collect(std::map, std::allocator > >&, DictTable*)+0x144 +Dictionary::collect(std::map, std::allocator > >&, DictTable*)+0x144 +Recording::writeClasses(Buffer*, Lookup*)+0x4f +Recording::writeCpool(Buffer*)+0x2c4 +Recording::finishChunk(bool)+0x267 +Recording::switchChunk(int)+0x17 +FlightRecorder::dump(char const*, int)+0x6f +Profiler::dump(char const*, int)+0x188 +Java_com_datadoghq_profiler_JavaProfiler_dump0+0x47 +0x00007c9d64d85811 com.datadoghq.profiler.JavaProfiler.dump0(Ljava/lang/String;)V +0x00007c9d6504c114 com.datadog.profiling.ddprof.DatadogProfilerRecording.snapshot(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;)Ldatadog/trace/api/profiling/RecordingData; +0x00007c9d6504ba14 com.datadog.profiling.controller.ddprof.DatadogProfilerOngoingRecording.snapshot(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;)Ldatadog/trace/api/profiling/RecordingData; +0x00007c9d64e2722c com.datadog.profiling.agent.CompositeController$CompositeOngoingRecording$$Lambda+0x00000000c4b8a000.apply(Ljava/lang/Object;)Ljava/lang/Object; +0x00007c9d636c955c java.util.stream.ReferencePipeline$3$1.accept(Ljava/lang/Object;)V +0x00007c9d63d0395c java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Ljava/util/function/Consumer;)V +0x00007c9d6399dddc java.util.stream.AbstractPipeline.copyInto(Ljava/util/stream/Sink;Ljava/util/Spliterator;)V +0x00007c9d6399da5c java.util.stream.AbstractPipeline.wrapAndCopyInto(Ljava/util/stream/Sink;Ljava/util/Spliterator;)Ljava/util/stream/Sink; +0x00007c9d63c3a1fc java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object; +0x00007c9d6387f1a4 java.util.stream.AbstractPipeline.evaluate(Ljava/util/stream/TerminalOp;)Ljava/lang/Object; +0x00007c9d63ccb494 java.util.stream.ReferencePipeline.collect(Ljava/util/stream/Collector;)Ljava/lang/Object; +0x00007c9d64fae5c4 com.datadog.profiling.agent.CompositeController$CompositeOngoingRecording.snapshot(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;)Ldatadog/trace/api/profiling/RecordingData; +0x00007c9d64ec10fc com.datadog.profiling.controller.ProfilingSystem$SnapshotRecording.snapshot(Z)V +0x00007c9d64ec0a04 com.datadog.profiling.controller.ProfilingSystem$$Lambda+0x00000000c31c0800.run(Ljava/lang/Object;)V +0x00007c9d63bf9c8c datadog.trace.util.AgentTaskScheduler$PeriodicTask.run()V +datadog.trace.util.AgentTaskScheduler$Worker.run()V (27) +java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V +java.lang.Thread.run()V +~StubRoutines::call_stub +JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x2ce +JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1d6 +thread_entry(JavaThread*, JavaThread*)+0x8b +JavaThread::thread_main_inner()+0x1d8 +Thread::call_run()+0xa8 +thread_native_entry(Thread*)+0xdb +start_routine_wrapper(void*)+0x9f +``` diff --git a/doc/libretti/2026-05-08-prof-14548-sigsegv-in-profilerupdatethreadname.md b/doc/libretti/2026-05-08-prof-14548-sigsegv-in-profilerupdatethreadname.md new file mode 100644 index 0000000000..450d72bee1 --- /dev/null +++ b/doc/libretti/2026-05-08-prof-14548-sigsegv-in-profilerupdatethreadname.md @@ -0,0 +1,39 @@ +--- +spec_id: REQ-PROF-14548 +source: jira +source_ref: PROF-14548 +title: "SIGSEGV in Profiler::updateThreadName" +status: implemented +clarity_score: 85 +created: 2026-05-08 +implementing_session: impl-20260508-184031 +implemented_pr: "DataDog/java-profiler#518" +--- + +# SIGSEGV in Profiler::updateThreadName + +``` +Profiler::updateThreadName(_jvmtiEnv*, JNIEnv_*, _jobject*, bool)+0x9d +Profiler::dump(char const*, int)+0x109 +Java_com_datadoghq_profiler_JavaProfiler_dump0+0x45 +com.datadoghq.profiler.JavaProfiler.dump0(Ljava/lang/String;)V +com.datadoghq.profiler.JavaProfiler.dump(Ljava/nio/file/Path;)V (11) +com.datadog.profiling.ddprof.DatadogProfiler.dump(Ljava/nio/file/Path;)V (5) +com.datadog.profiling.ddprof.DatadogProfilerRecording.snapshot(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;)Ldatadog/trace/api/profiling/RecordingData; (17) +com.datadog.profiling.controller.ddprof.DatadogProfilerOngoingRecording.snapshot(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;)Ldatadog/trace/api/profiling/RecordingData; (10) +com.datadog.profiling.agent.CompositeController$CompositeOngoingRecording.lambda$snapshot$0(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;Lcom/datadog/profiling/controller/OngoingRecording;)Ldatadog/trace/api/profiling/RecordingData; (3) +com.datadog.profiling.agent.CompositeController$CompositeOngoingRecording$$Lambda +0x000070376284537c java.util.stream.ReferencePipeline$3$1.accept(Ljava/lang/Object;)V +0x0000703763652620 java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Ljava/util/function/Consumer;)V +0x0000703762e73938 java.util.stream.AbstractPipeline.copyInto(Ljava/util/stream/Sink;Ljava/util/Spliterator;)V +0x00007037634bb1cc java.util.stream.ReferencePipeline.collect(Ljava/util/stream/Collector;)Ljava/lang/Object; +com.datadog.profiling.agent.CompositeController$CompositeOngoingRecording.compose(Ljava/util/function/Function;)Ldatadog/trace/api/profiling/RecordingData; (22) +com.datadog.profiling.agent.CompositeController$CompositeOngoingRecording.snapshot(Ljava/time/Instant;Ldatadog/trace/api/profiling/ProfilingSnapshot$Kind;)Ldatadog/trace/api/profiling/RecordingData; (8) +com.datadog.profiling.controller.ProfilingSystem$SnapshotRecording.snapshot(Z)V (38) +com.datadog.profiling.controller.ProfilingSystem$SnapshotRecording.snapshot()V (2) +com.datadog.profiling.controller.ProfilingSystem$$Lambda +0x0000703762e9604c datadog.trace.util.AgentTaskScheduler$PeriodicTask.run()V +datadog.trace.util.AgentTaskScheduler$Worker.run()V (27) +java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V +java.lang.Thread.run()V +``` diff --git a/doc/libretti/2026-05-11-prof-14582-crash-sigsegv-in-dictionaryclear.md b/doc/libretti/2026-05-11-prof-14582-crash-sigsegv-in-dictionaryclear.md new file mode 100644 index 0000000000..0164ee4bd2 --- /dev/null +++ b/doc/libretti/2026-05-11-prof-14582-crash-sigsegv-in-dictionaryclear.md @@ -0,0 +1,54 @@ +--- +spec_id: REQ-PROF-14582 +source: jira +source_ref: PROF-14582 +jira_comment_id: "3244603" +title: "[Crash] SIGSEGV in Dictionary::clear" +status: implemented +clarity_score: null +created: 2026-05-11 +implementing_session: impl-20260511-152444 +implemented_pr: https://github.com/DataDog/java-profiler/pull/522 +--- + +# [Crash] SIGSEGV in Dictionary::clear + +## Crash signature + +* Stack-similarity hash: `53fbf099969b056b` (label `dd-crashsig-53fbf099969b056b`) +* Datadog fingerprints observed: `v10.9BAF07DBF7DA7F0D31E7C17BDAF9753B` _(note: this fingerprint is shared with the_ `Profiler::onThreadEnd` cluster — Datadog's fingerprint is coarser than the top library frame in this case; rely on the stack-similarity hash for dedup) + +## Summary + +5 crash events observed in the last 2 days (2026-05-06 → 2026-05-08) where the topmost library frame is `Dictionary::clear()` inside the dd-trace-java profiler. Distinct from the broader `[Crash] SIGSEGV in Profiler::dump` umbrella (PROF-14550) by the specific top frame. Tracer: `dd-trace-java` `1.62.0_16c6a5fd1f`. Signal: SIGSEGV. + +## Top stack frames + +``` +#0 Dictionary::clear() +... (deeper frames not captured in this triage; relates to the Profiler::dump → JFR write path) +``` + +## Representative event + +* Datadog logs explorer: https://app.datadoghq.com/logs?query=service%3Ainstrumentation-telemetry-data%20%40lib_language%3Ajvm%20%40tags.crash_datadog%3Atrue%20%40tracer_version%3A1.62.0\*%20%40error.is_crash%3Atrue%20%40error.stack.frames.function%3A%22Dictionary\*%22 +* Sample organization: org_id 1000043007 (`robaws`), service `robaws`, env `prod`, language `21.0.10`, OS Linux x86_64 +* Sample fingerprint id: `fa6915ac-08a6-48d4-8cfd-1d52a7532634` + +## Distribution (5 events) + +* Top frames split: `Dictionary::clear()` x3, `Dictionary::clear(DictTable*, int)` x2 (variant signatures of the same call site) +* Most events come from EU prod orgs + +## Datadog fingerprints in this cluster + +* `v10.9BAF07DBF7DA7F0D31E7C17BDAF9753B` + +--- + +_Auto-filed by the_ `crash-to-jira` triage command. Relates to PROF-14550 (Profiler::dump umbrella). + +## Audit Trail + +- `2026-05-11` — status: `draft` → `implementing` (session `impl-20260511-152444`) +- `2026-05-11` — status: `implementing` → `implemented` (PR https://github.com/DataDog/java-profiler/pull/522) diff --git a/doc/libretti/2026-05-11-prof-14583-crash-sigsegv-in-stdrbtreeincrement-during-recordingwritecpo.md b/doc/libretti/2026-05-11-prof-14583-crash-sigsegv-in-stdrbtreeincrement-during-recordingwritecpo.md new file mode 100644 index 0000000000..e7155cd76a --- /dev/null +++ b/doc/libretti/2026-05-11-prof-14583-crash-sigsegv-in-stdrbtreeincrement-during-recordingwritecpo.md @@ -0,0 +1,54 @@ +--- +spec_id: REQ-PROF-14583 +source: jira +source_ref: PROF-14583 +jira_key: PROF-14583 +jira_comment_id: "3244785" +title: "[Crash] SIGSEGV in std::_Rb_tree_increment during Recording::writeCpool" +status: implemented +clarity_score: null +created: 2026-05-11 +implementing_session: impl-20260511-155555 +implemented_pr: https://github.com/DataDog/java-profiler/pull/523 +--- + +# [Crash] SIGSEGV in std::_Rb_tree_increment during Recording::writeCpool + +## Crash signature + +* Stack-similarity hash: `4edf2d97decd942f` (label `dd-crashsig-4edf2d97decd942f`) +* Datadog fingerprints observed: `v10.DAECC680F0728EAB44F26DB0B91B703F` _(distinct from the_ `Profiler::onThreadEnd` and `Profiler::processCallTraces` clusters) + +## Summary + +3 crash events observed in the last 2 days (2026-05-06 → 2026-05-08) where the topmost frame is `std::_Rb_tree_increment` reached from `Recording::writeCpool` during JFR chunk finalization. Distinct from the `Profiler::processCallTraces` cluster (PROF-14547) — the SEGV happens in the rbtree iterator within `writeCpool` rather than inside the call-trace processing pass. Sibling of the broader `Profiler::dump` umbrella (PROF-14550). + +Tracer: `dd-trace-java` `1.62.0_16c6a5fd1f`. Signal: SIGSEGV. + +## Top stack frames + +``` +#0 std::_Rb_tree_increment(std::_Rb_tree_node_base*) +#1 Recording::writeCpool(Buffer*) +#2 Recording::finishChunk(bool) +#3 Recording::switchChunk(int) +#4 FlightRecorder::dump(char const*, int) +#5 Profiler::dump(char const*, int) +#6 Java_com_datadoghq_profiler_JavaProfiler_dump0 +``` + +## Representative event + +* Datadog logs explorer: https://app.datadoghq.com/logs?query=service%3Ainstrumentation-telemetry-data%20%40lib_language%3Ajvm%20%40tags.crash_datadog%3Atrue%20%40tracer_version%3A1.62.0\*%20%40error.is_crash%3Atrue +* Sample timestamp: 2026-05-07T21:15:35Z +* Sample organization: org_id 1000000744 (`betclic` / mangas gambling), service `casino.game-launcher.worker`, env `prod`, language `21.0.8`, OS Linux x86_64 (Amazon Corretto 21) +* Sample fingerprint id: `9dca29ee-08a2-451c-b48a-c2278d72ee87` +* TRAPNO 0xd, RAX/R8 = `0xa5f355500467267c` (looks like a poisoned/uninit pointer fed into rbtree increment) + +## Datadog fingerprints in this cluster + +* `v10.DAECC680F0728EAB44F26DB0B91B703F` + +--- + +_Auto-filed by the_ `crash-to-jira` triage command. Relates to PROF-14550 (Profiler::dump umbrella) and PROF-14547 (writeCpool / writeStackTraces family). diff --git a/doc/libretti/2026-05-12-prof-14585-crash-sigsegv-in-recordingswitchchunk.md b/doc/libretti/2026-05-12-prof-14585-crash-sigsegv-in-recordingswitchchunk.md new file mode 100644 index 0000000000..e863fb8098 --- /dev/null +++ b/doc/libretti/2026-05-12-prof-14585-crash-sigsegv-in-recordingswitchchunk.md @@ -0,0 +1,57 @@ +--- +spec_id: REQ-PROF-14585 +source: jira +source_ref: PROF-14585 +title: "[Crash] SIGSEGV in Recording::switchChunk" +status: draft +clarity_score: null +created: 2026-05-12 +implementing_session: null +implemented_pr: null +--- + +# [Crash] SIGSEGV in Recording::switchChunk + +## Crash signature + +* Full-stack hash: `07b2657be5f9313a` (label `dd-crashsig-07b2657be5f9313a`) +* Top-frame hash: `e09d60cedafef867` (label `dd-crashsig-top-e09d60cedafef867`) — fallback for partial-stack matches +* Datadog fingerprints observed: `v10.B1A8D6128A089A3347E6EADF45BE9B1A` _(hint only — fingerprint is not 1:1 with this stack; e.g. it may overlap unrelated crashes)_ + +## Summary + +`1` crash event from `1` org / `1` service in the last `1` day, all matching this stack pattern. Tracer: `DataDog/dd-trace-java` `@tracer_version:1.62.0*`. Signal: `SIGSEGV`. + +## Top stack frames (normalized) + +``` +#0 Recording::switchChunk +#1 FlightRecorder::dump +#2 Profiler::dump +#3 Java_com_datadoghq_profiler_JavaProfiler_dump0 +``` + +## Representative event + +* [Datadog logs](https://app.datadoghq.com/logs?from_ts=1778149950956&live=false&query=service%3Ainstrumentation-telemetry-data+%40lib_language%3Ajvm+%40tracer_version%3A1.62.0%2A+%40error.is_crash%3Atrue&stream_sort=desc&to_ts=1778236350956) for one example +* Service: `land-guidanceplan` +* Env: `cert` +* OS / arch: `Linux` / `amd64` +* Lang version: `25.0.3` +* Tracer build id: `1.62.0~16c6a5fd1f` +* JVM args: `-javaagent:dd-java-agent.jar -XX:MaxRAMPercentage=75 -XX:+UseZGC -XX:+UseCompactObjectHeaders` + +## Distribution + +* Envs: `cert:1` +* OS: `Linux:1` +* Arch: `amd64:1` +* Language versions: `{25.0.3}` + +## Datadog fingerprints in this cluster + +`v10.B1A8D6128A089A3347E6EADF45BE9B1A` + +--- + +_Auto-filed by the_ `crash-to-jira` triage command. diff --git a/doc/plans/2026-03-30-continuation-enter-unwind.md b/doc/plans/2026-03-30-continuation-enter-unwind.md new file mode 100644 index 0000000000..b118d366ff --- /dev/null +++ b/doc/plans/2026-03-30-continuation-enter-unwind.md @@ -0,0 +1,598 @@ +# Virtual Thread Continuation Unwind Fix Implementation Plan + +> **Status: IMPLEMENTED** — All tasks complete as of 2026-03-30. See "Implementation Divergences" section below for how the final code differs from the original design. + +**Goal:** Enable `walkVM` to traverse through `jdk.internal.vm.Continuation.enterSpecial` frames when profiling mounted virtual threads, covering both VTs that have never yielded (all frames thawed) and VTs that have yielded and remounted with frozen frames remaining. + +**Architecture:** Two complementary paths, both navigating via `ContinuationEntry`: + +1. **CPU-bound VT (never yielded, all frames thawed):** Profiler walks up through continuation body frames → `Continuation.enter` (normal JIT nmethod) → reaches `enterSpecial`. On JDK 27+ `enterSpecial` is an nmethod detected by identity (`nm == VMStructs::enterSpecialNMethod()`). On JDK 21-26 `enterSpecial` is a `RuntimeBlob`, so `findNMethod()` returns `NULL`; this is detected in the `nm == NULL` handler by combining three guards: `VMContinuationEntry::type_size() == 0` (JDK 21-26 marker), `VMStructs::hasContReturnBarrier()` (JDK 21+ proxy), and `vm_thread->hasActiveContinuation()`. In both cases the walker jumps to the carrier thread via `walkThroughContinuation(/*path_a=*/true)`. + +2. **Yielded VT (partial thaw, frozen frames remain):** The bottommost thawed compiled frame has its return PC patched to `StubRoutines::_cont_returnBarrier`. Detected by PC comparison before `findNMethod`. Same `walkThroughContinuation(/*path_a=*/false)` navigation to the carrier. + +Both paths use the same `walkThroughContinuation` lambda which computes `{carrier_fp, carrier_pc, carrier_sp}` from `entry_fp`. On JDK 21-26 (no `type_size()`) `entry_fp` is derived from the current `fp` register via fp-based fallback. The `isEntryFrame` / `JavaCallWrapper` path is never reached for either case. + +### Implementation Divergences from Original Plan + +| Aspect | Original plan | Actual implementation | +|--------|--------------|----------------------| +| Path A detection | `nm == enterSpecialNMethod()` inside `isNMethod()` block | JDK 27+: same. JDK 21-26: `nm == NULL` handler + `type_size()==0` + `hasContReturnBarrier()` + `hasActiveContinuation()` | +| Carrier navigation | `entry->entryFP()` via `contEntry()` | `walkThroughContinuation` lambda; `entry_fp` derived from `fp` on JDK 21-26 | +| `CONT_UNWIND_DISABLED` | Not modelled in plan | Handled: when disabled at continuation boundary, emit `BCI_NATIVE_FRAME, "JVM Continuation"` and break cleanly | +| New vmStructs helpers | `contEntry()` only | Also added `hasContReturnBarrier()` and `hasActiveContinuation()` (bypasses `type_size()` guard) | +| Test assertions | No walk-error sentinels | Carrier frames (`ForkJoinWorkerThread`) visible in stack traces | +| Test modes | `vm`, `fp` | `vm`, `vmx` only — fp/dwarf use ASGCT which cannot cross continuation boundary | +| Test profiler args | `wall=1ms` | `wall=1ms,filter=,wextend=vt_carrier` | + +**Tech Stack:** C++ (C++14), JVM vmStructs introspection, GTest (C++ unit tests), JUnit 5 + JMC (Java integration tests), JDK 21+. + +--- + +## File Map + +| File | Change | +|---|---| +| `ddprof-lib/src/main/cpp/vmStructs.h` | Add `VMContinuationEntry` type; add `_cont_return_barrier`, `_cont_entry_return_pc`, `_enter_special_nm` static fields; `contEntry()` on VMThread; `isContReturnBarrier()` and `enterSpecialNMethod()` helpers | +| `ddprof-lib/src/main/cpp/vmStructs.cpp` | Initialise the three new static fields; find `_enter_special_nm` from CodeHeap after `_cont_entry_return_pc` is set | +| `ddprof-lib/src/main/cpp/counters.h` | Add `WALKVM_CONT_BARRIER_HIT`, `WALKVM_ENTER_SPECIAL_HIT`, `WALKVM_CONT_ENTRY_NULL` | +| `ddprof-lib/src/main/cpp/stackWalker.cpp` | (a) Add `enterSpecial` detection inside the `isNMethod()` block; (b) Add `cont_returnBarrier` branch before `isEntryFrame`; both branches share the same `contEntry()` navigation logic | +| `ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java` | New integration test covering both paths: CPU-bound VT (never-yield path) and I/O-blocking VT (cont_returnBarrier path) | + +--- + +## Task 1: Add counters + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/counters.h:95-98` + +- [x] **Step 1: Add three new counters before `NATIVE_LIBS_DROPPED`** + + In `counters.h`, the relevant lines are: + ```cpp + X(WALKVM_ANCHOR_NOT_IN_JAVA, "walkvm_anchor_not_in_java") \ + X(NATIVE_LIBS_DROPPED, "native_libs_dropped") \ + ``` + Change to: + ```cpp + X(WALKVM_ANCHOR_NOT_IN_JAVA, "walkvm_anchor_not_in_java") \ + X(WALKVM_CONT_BARRIER_HIT, "walkvm_cont_barrier_hit") \ + X(WALKVM_ENTER_SPECIAL_HIT, "walkvm_enter_special_hit") \ + X(WALKVM_CONT_ENTRY_NULL, "walkvm_cont_entry_null") \ + X(NATIVE_LIBS_DROPPED, "native_libs_dropped") \ + ``` + +- [x] **Step 2: Build** + + ```bash + cd /Users/jaroslav.bachorik/go/src/github.com/DataDog/java-profiler + ./gradlew :ddprof-lib:assemble -Pbuild_profile=debug 2>&1 | tail -20 + ``` + Expected: `BUILD SUCCESSFUL` + +- [x] **Step 3: Commit** + + ```bash + git add ddprof-lib/src/main/cpp/counters.h + git commit -m "feat(walkvm): add continuation walk counters" + ``` + +--- + +## Task 2: Expose `ContinuationEntry` and `enterSpecial` nmethod in vmStructs + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/vmStructs.h` +- Modify: `ddprof-lib/src/main/cpp/vmStructs.cpp` + +### Background + +`ContinuationEntry` is embedded on the carrier thread's stack immediately below the `enterSpecial` frame's saved-fp slot. `JavaThread::_cont_entry` points to the innermost active entry (a linked list for nested virtual threads). + +On-stack layout (identical on x86-64 and aarch64 — the JVM standardises FP to point at the `{saved_fp, lr}` pair): + +``` +[high addresses] + return_addr_to_carrier <- caller of enterSpecial (in Continuation.run()) + saved_fp <- entryFP() == (uintptr_t)entry + ContinuationEntry::size() + ContinuationEntry { + _parent <- enclosing ContinuationEntry* or null + ... <- other fields not needed by profiler + } <- entry == vm_thread->_cont_entry + [argsize padding] + bottommost thawed frame <- return PC = cont_returnBarrier (if frozen frames exist) + OR real caller PC (if fully thawed) +[low addresses] +``` + +Given `entry_fp = entry->entryFP()`: +- `*(uintptr_t*)entry_fp` = carrier FP +- `((const void**)entry_fp)[1]` = carrier PC (return addr back to Continuation.run()) +- `entry_fp + 2*sizeof(void*)` = carrier SP + +`ContinuationEntry::_return_pc` is a static address baked into the `enterSpecial` nmethod's code at the point after the thaw call. The profiler caches it as `_cont_entry_return_pc` and then uses it to locate the `enterSpecial` nmethod itself via `CodeHeap::findNMethod(_cont_entry_return_pc)`, stored as `_enter_special_nm`. This is the same pattern used to find `_interpreter_nm` from `_interpreter_start`. + +### Steps + +- [x] **Step 1: Add `VMContinuationEntry` to `DECLARE_TYPES_DO`** + + In `vmStructs.h`, `DECLARE_TYPES_DO` currently is: + ```cpp + #define DECLARE_TYPES_DO(f) \ + f(VMClassLoaderData, MATCH_SYMBOLS("ClassLoaderData")) \ + f(VMConstantPool, MATCH_SYMBOLS("ConstantPool")) \ + f(VMConstMethod, MATCH_SYMBOLS("ConstMethod")) \ + f(VMFlag, MATCH_SYMBOLS("JVMFlag", "Flag")) \ + f(VMJavaFrameAnchor, MATCH_SYMBOLS("JavaFrameAnchor")) \ + f(VMKlass, MATCH_SYMBOLS("Klass")) \ + f(VMMethod, MATCH_SYMBOLS("Method")) \ + f(VMNMethod, MATCH_SYMBOLS("nmethod")) \ + f(VMSymbol, MATCH_SYMBOLS("Symbol")) \ + f(VMThread, MATCH_SYMBOLS("Thread")) + ``` + Change to: + ```cpp + #define DECLARE_TYPES_DO(f) \ + f(VMClassLoaderData, MATCH_SYMBOLS("ClassLoaderData")) \ + f(VMConstantPool, MATCH_SYMBOLS("ConstantPool")) \ + f(VMConstMethod, MATCH_SYMBOLS("ConstMethod")) \ + f(VMContinuationEntry, MATCH_SYMBOLS("ContinuationEntry")) \ + f(VMFlag, MATCH_SYMBOLS("JVMFlag", "Flag")) \ + f(VMJavaFrameAnchor, MATCH_SYMBOLS("JavaFrameAnchor")) \ + f(VMKlass, MATCH_SYMBOLS("Klass")) \ + f(VMMethod, MATCH_SYMBOLS("Method")) \ + f(VMNMethod, MATCH_SYMBOLS("nmethod")) \ + f(VMSymbol, MATCH_SYMBOLS("Symbol")) \ + f(VMThread, MATCH_SYMBOLS("Thread")) + ``` + +- [x] **Step 2: Add field declarations in `DECLARE_TYPE_FIELD_DO`** + + 2a. Add `_cont_entry_offset` to the `VMJavaThread` block (around line 190): + ```cpp + type_begin(VMJavaThread, MATCH_SYMBOLS("JavaThread", "Thread")) + field(_thread_osthread_offset, offset, MATCH_SYMBOLS("_osthread")) + field(_thread_anchor_offset, offset, MATCH_SYMBOLS("_anchor")) + field(_thread_state_offset, offset, MATCH_SYMBOLS("_thread_state")) + field(_thread_vframe_offset, offset, MATCH_SYMBOLS("_vframe_array_head")) + field_with_version(_cont_entry_offset, offset, 21, MAX_VERSION, MATCH_SYMBOLS("_cont_entry")) + type_end() + ``` + + 2b. Add `_cont_return_barrier_addr` to the `VMStubRoutine` block (around line 250): + ```cpp + type_begin(VMStubRoutine, MATCH_SYMBOLS("StubRoutines")) + field(_call_stub_return_addr, address, MATCH_SYMBOLS("_call_stub_return_address")) + field_with_version(_cont_return_barrier_addr, address, 21, MAX_VERSION, MATCH_SYMBOLS("_cont_returnBarrier")) + type_end() + ``` + + 2c. Add a new `VMContinuationEntry` type block after the `VMJavaFrameAnchor` block: + ```cpp + type_begin(VMContinuationEntry, MATCH_SYMBOLS("ContinuationEntry")) + field_with_version(_cont_entry_parent_offset, offset, 21, MAX_VERSION, MATCH_SYMBOLS("_parent")) + field_with_version(_cont_entry_return_pc_addr, address, 21, MAX_VERSION, MATCH_SYMBOLS("_return_pc")) + type_end() + ``` + +- [x] **Step 3: Add three static variables to the `VMStructs` class body** + + In `vmStructs.h`, add these three lines directly after `static const void* _call_stub_return;` (around line 313): + ```cpp + static const void* _cont_return_barrier; + static const void* _cont_entry_return_pc; + static VMNMethod* _enter_special_nm; + ``` + +- [x] **Step 4: Add static helpers to `VMStructs`** + + In `vmStructs.h`, add these two static methods inside the `VMStructs` class, near `isEntryFrame` in `VMNMethod` or after the static variable declarations: + ```cpp + static bool isContReturnBarrier(const void* pc) { + return _cont_return_barrier != nullptr && pc == _cont_return_barrier; + } + + static VMNMethod* enterSpecialNMethod() { + return _enter_special_nm; + } + ``` + +- [x] **Step 5: Add the `VMContinuationEntry` DECLARE class** + + Add this new class after the `DECLARE_END` of `VMJavaFrameAnchor` (around line 680): + ```cpp + DECLARE(VMContinuationEntry) + public: + static bool isAvailable() { + return _cont_entry_parent_offset >= 0; + } + + VMContinuationEntry* parent() { + assert(_cont_entry_parent_offset >= 0); + return (VMContinuationEntry*) SafeAccess::loadPtr((void**) at(_cont_entry_parent_offset), nullptr); + } + + // Address of the enterSpecial frame's {saved_fp, return_addr} pair. + // Layout above this address: [saved_fp][return_addr_to_carrier][carrier_sp...] + uintptr_t entryFP() const { + return (uintptr_t)this + type_size(); + } + DECLARE_END + ``` + +- [x] **Step 6: Add `contEntry()` to `VMThread`** + + Inside the `VMThread` `DECLARE` block, add after `anchor()`: + ```cpp + VMContinuationEntry* contEntry() { + if (_cont_entry_offset < 0) return nullptr; + void* ptr = SafeAccess::loadPtr((void**) at(_cont_entry_offset), nullptr); + return ptr != nullptr ? VMContinuationEntry::cast(ptr) : nullptr; + } + ``` + +- [x] **Step 7: Add static-variable definitions in `vmStructs.cpp`** + + Add three lines after `const void* VMStructs::_call_stub_return = nullptr;` (around line 36): + ```cpp + const void* VMStructs::_cont_return_barrier = nullptr; + const void* VMStructs::_cont_entry_return_pc = nullptr; + VMNMethod* VMStructs::_enter_special_nm = nullptr; + ``` + +- [x] **Step 8: Initialise the new fields in `vmStructs.cpp`** + + In `vmStructs.cpp`, after the existing `_call_stub_return` init block (around line 394), add: + ```cpp + if (_cont_return_barrier_addr != NULL) { + _cont_return_barrier = *(const void**)_cont_return_barrier_addr; + } + if (_cont_entry_return_pc_addr != NULL) { + _cont_entry_return_pc = *(const void**)_cont_entry_return_pc_addr; + } + ``` + + Then, after the existing `_interpreter_nm` lookup (around line 443): + ```cpp + if (_enter_special_nm == NULL && _cont_entry_return_pc != NULL) { + _enter_special_nm = CodeHeap::findNMethod(_cont_entry_return_pc); + } + ``` + + This mirrors the existing pattern `_interpreter_nm = CodeHeap::findNMethod(_interpreter_start)`. + +- [x] **Step 9: Build** + + ```bash + ./gradlew :ddprof-lib:assemble -Pbuild_profile=debug 2>&1 | tail -20 + ``` + Expected: `BUILD SUCCESSFUL` + +- [x] **Step 10: Commit** + + ```bash + git add ddprof-lib/src/main/cpp/vmStructs.h ddprof-lib/src/main/cpp/vmStructs.cpp + git commit -m "feat(vmstructs): expose ContinuationEntry and enterSpecial nmethod" + ``` + +--- + +## Task 3: Add continuation unwind paths in `walkVM` + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/stackWalker.cpp` + +### What to implement + +Both paths produce the same carrier-thread `{sp, fp, pc}` triple from `entry->entryFP()` and `continue` the walk loop. The computation: + +``` +entry_fp = entry->entryFP() // = (uintptr_t)entry + ContinuationEntry::type_size() +carrier_fp = *(uintptr_t*)entry_fp // saved FP from enterSpecial's frame prologue +carrier_pc = ((const void**)entry_fp)[1] // return address baked into enterSpecial → Continuation.run() +carrier_sp = entry_fp + 2*sizeof(void*) // SP of the Continuation.run() call site +``` + +**Path A — `enterSpecial` nmethod detection** (CPU-bound VT, all frames thawed): +Placed at the *top* of the `isNMethod()` block, before the frame is emitted or `frameSize()` is used. When `nm == VMStructs::enterSpecialNMethod()`, skip directly to carrier via `contEntry()`. This prevents incorrect `frameSize()` usage on the `enterSpecial` frame and avoids emitting a profiler-internal stub as a user-visible frame. + +**Path B — `cont_returnBarrier` detection** (yielded VT, frozen frames remain): +New `else if` branch between the `isNMethod()` and `isEntryFrame()` blocks. Triggered when the current PC equals `StubRoutines::_cont_returnBarrier`. + +### Steps + +- [x] **Step 1: Add Path A — `enterSpecial` detection inside `isNMethod()`** + + Locate the opening of the `isNMethod()` block (around line 443): + ```cpp + } else if (nm->isNMethod()) { + // Check if deoptimization is in progress before walking compiled frames + if (vm_thread != NULL && vm_thread->inDeopt()) { + ``` + Insert the enterSpecial check at the very beginning of this block: + ```cpp + } else if (nm->isNMethod()) { + // enterSpecial is a generated native nmethod that acts as the + // continuation entry stub. It has no JavaCallWrapper, so + // isEntryFrame() will not fire for it. Detect it by identity + // and navigate to the carrier thread via ContinuationEntry. + if (nm == VMStructs::enterSpecialNMethod()) { + Counters::increment(WALKVM_ENTER_SPECIAL_HIT); + VMContinuationEntry* entry = vm_thread != nullptr ? vm_thread->contEntry() : nullptr; + if (entry == nullptr) { + Counters::increment(WALKVM_CONT_ENTRY_NULL); + fillFrame(frames[depth++], BCI_ERROR, "break_cont_entry_null"); + break; + } + uintptr_t entry_fp = entry->entryFP(); + if (!goodPtr((void*)entry_fp) || !aligned(entry_fp)) { + fillFrame(frames[depth++], BCI_ERROR, "break_cont_entry_fp"); + break; + } + uintptr_t carrier_fp = *(uintptr_t*)entry_fp; + const void* carrier_pc = ((const void**)entry_fp)[1]; + uintptr_t carrier_sp = entry_fp + 2 * sizeof(void*); + if (!sameStack(sp, carrier_sp, bottom) || !aligned(carrier_sp)) { + fillFrame(frames[depth++], BCI_ERROR, "break_cont_carrier_sp"); + break; + } + sp = carrier_sp; + fp = carrier_fp; + pc = carrier_pc; + continue; + } + // Check if deoptimization is in progress before walking compiled frames + if (vm_thread != NULL && vm_thread->inDeopt()) { + ``` + +- [x] **Step 2: Add Path B — `cont_returnBarrier` branch** + + Locate the `isEntryFrame` branch (around line 502): + ```cpp + } else if (nm->isEntryFrame(pc) && !features.mixed) { + ``` + Insert a new `else if` immediately before it: + ```cpp + } else if (VMStructs::isContReturnBarrier(pc)) { + // The bottommost thawed compiled frame's return PC has been + // patched to cont_returnBarrier, meaning frozen frames remain + // in the StackChunk. Navigate via ContinuationEntry to the + // carrier thread frames above the enterSpecial stub. + Counters::increment(WALKVM_CONT_BARRIER_HIT); + VMContinuationEntry* entry = vm_thread != nullptr ? vm_thread->contEntry() : nullptr; + if (entry == nullptr) { + Counters::increment(WALKVM_CONT_ENTRY_NULL); + fillFrame(frames[depth++], BCI_ERROR, "break_cont_entry_null"); + break; + } + uintptr_t entry_fp = entry->entryFP(); + if (!goodPtr((void*)entry_fp) || !aligned(entry_fp)) { + fillFrame(frames[depth++], BCI_ERROR, "break_cont_entry_fp"); + break; + } + uintptr_t carrier_fp = *(uintptr_t*)entry_fp; + const void* carrier_pc = ((const void**)entry_fp)[1]; + uintptr_t carrier_sp = entry_fp + 2 * sizeof(void*); + if (!sameStack(sp, carrier_sp, bottom) || !aligned(carrier_sp)) { + fillFrame(frames[depth++], BCI_ERROR, "break_cont_carrier_sp"); + break; + } + sp = carrier_sp; + fp = carrier_fp; + pc = carrier_pc; + continue; + } else if (nm->isEntryFrame(pc) && !features.mixed) { + ``` + +- [x] **Step 3: Build** + + ```bash + ./gradlew :ddprof-lib:assemble -Pbuild_profile=debug 2>&1 | tail -20 + ``` + Expected: `BUILD SUCCESSFUL` + +- [x] **Step 4: Commit** + + ```bash + git add ddprof-lib/src/main/cpp/stackWalker.cpp ddprof-lib/src/main/cpp/vmStructs.h + git commit -m "fix(walkvm): unwind through enterSpecial and cont_returnBarrier" + ``` + +--- + +## Task 4: Integration tests — both VT unwind paths + +**Files:** +- Create: `ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java` + +### What to test + +Two test methods, one per path: + +**`samplesCarrierFramesFromCpuBoundVT`** — Path A (enterSpecial detection). +A VT that never blocks (CPU-only spin loop) is never suspended, so all frames are always thawed and `cont_returnBarrier` never appears. The profiler must traverse `enterSpecial` via the nmethod-identity check to reach carrier frames. + +**`samplesCarrierFramesFromBlockingVT`** — Path B (cont_returnBarrier). +A VT that repeatedly parks and unparks will have its frames frozen and thawed. When remounted with frozen frames still in the StackChunk, `cont_returnBarrier` is the return PC of the bottommost thawed frame. + +Both methods assert: +1. At least one sample from the VT's lambda is captured. +2. No sample contains any `break_cont_*` or `break_entry_frame` sentinel in the stack trace string. + +The test class is skipped entirely on JDK < 21. + +- [x] **Step 1: Write the test** + + ```java + package com.datadoghq.profiler.wallclock; + + import com.datadoghq.profiler.CStackAwareAbstractProfilerTest; + import com.datadoghq.profiler.Platform; + import com.datadoghq.profiler.junit.CStack; + import com.datadoghq.profiler.junit.RetryTest; + import org.junit.jupiter.api.TestTemplate; + import org.junit.jupiter.params.provider.ValueSource; + import org.openjdk.jmc.common.item.IItem; + import org.openjdk.jmc.common.item.IItemCollection; + import org.openjdk.jmc.common.item.IItemIterable; + import org.openjdk.jmc.common.item.IMemberAccessor; + import org.openjdk.jmc.flightrecorder.jdk.JdkAttributes; + + import java.util.concurrent.CountDownLatch; + import java.util.concurrent.locks.LockSupport; + + import static org.junit.jupiter.api.Assertions.assertFalse; + import static org.junit.jupiter.api.Assertions.assertTrue; + + public class VirtualThreadWallClockTest extends CStackAwareAbstractProfilerTest { + + private volatile long sink; + + public VirtualThreadWallClockTest(@CStack String cstack) { + super(cstack); + } + + @Override + protected boolean isPlatformSupported() { + // Virtual threads require JDK 21+ + return Platform.isJavaVersionAtLeast(21); + } + + /** + * Path A: enterSpecial nmethod detection. + * A CPU-bound VT never yields, so all frames stay thawed and cont_returnBarrier + * never appears. The profiler must unwind through enterSpecial directly. + */ + @RetryTest(3) + @TestTemplate + @ValueSource(strings = {"vm", "fp"}) + public void samplesCarrierFramesFromCpuBoundVT(@CStack String cstack) throws Exception { + CountDownLatch started = new CountDownLatch(1); + Thread vt = Thread.ofVirtual().start(() -> { + started.countDown(); + long sum = 0; + for (long i = 0; i < 600_000_000L; i++) { + sum += i; + } + sink = sum; + }); + started.await(); + vt.join(15_000); + stopProfiler(); + + assertNoWalkErrors("VirtualThreadWallClockTest"); + } + + /** + * Path B: cont_returnBarrier detection. + * A VT that parks and unparks repeatedly will have frames frozen into a StackChunk. + * On remount with frozen frames remaining, the bottommost thawed frame has + * cont_returnBarrier as its return PC. + */ + @RetryTest(3) + @TestTemplate + @ValueSource(strings = {"vm", "fp"}) + public void samplesCarrierFramesFromBlockingVT(@CStack String cstack) throws Exception { + Thread[] carrier = new Thread[1]; + CountDownLatch started = new CountDownLatch(1); + Thread vt = Thread.ofVirtual().start(() -> { + carrier[0] = Thread.currentThread(); + started.countDown(); + // Park/unpark 200 times to exercise freeze/thaw cycle + for (int i = 0; i < 200; i++) { + LockSupport.park(); + } + }); + started.await(); + for (int i = 0; i < 200; i++) { + Thread.sleep(5); // give wall-clock sampler time to fire during freeze/thaw + LockSupport.unpark(vt); + } + vt.join(10_000); + stopProfiler(); + + assertNoWalkErrors("VirtualThreadWallClockTest"); + } + + /** Verifies at least one sample from this test class with no walk-error sentinels. */ + private void assertNoWalkErrors(String markerClass) throws Exception { + IItemCollection events = verifyEvents("datadog.MethodSample"); + boolean sawSample = false; + for (IItemIterable samples : events) { + IMemberAccessor traceAcc = + JdkAttributes.STACK_TRACE_STRING.getAccessor(samples.getType()); + if (traceAcc == null) continue; + for (IItem sample : samples) { + String trace = traceAcc.getMember(sample); + if (trace == null || !trace.contains(markerClass)) continue; + sawSample = true; + assertFalse(trace.contains("break_cont_entry_null"), + "cont_entry null walk error in: " + trace); + assertFalse(trace.contains("break_cont_entry_fp"), + "cont_entry fp walk error in: " + trace); + assertFalse(trace.contains("break_cont_carrier_sp"), + "cont carrier sp walk error in: " + trace); + assertFalse(trace.contains("break_entry_frame"), + "entry frame walk error in: " + trace); + } + } + assertTrue(sawSample, "No wall-clock samples captured from the virtual thread"); + } + + @Override + protected void after() {} + + @Override + protected String getProfilerCommand() { + return "wall=1ms"; + } + } + ``` + +- [x] **Step 2: Build** + + ```bash + ./gradlew :ddprof-test:compileTestJava 2>&1 | tail -20 + ``` + Expected: `BUILD SUCCESSFUL` + +- [x] **Step 3: Run on JDK 21+** + + ```bash + ./gradlew :ddprof-test:testRelease \ + --tests "com.datadoghq.profiler.wallclock.VirtualThreadWallClockTest" 2>&1 | tail -40 + ``` + Expected: both test methods pass. On JDK < 21 the entire class is skipped via `isPlatformSupported()`. + +- [x] **Step 4: Commit** + + ```bash + git add ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualThreadWallClockTest.java + git commit -m "test(wallclock): virtual thread continuation unwind smoke tests" + ``` + +--- + +## Self-Review + +### Spec coverage + +| Hypothesis | Task that addresses it | +|---|---| +| H1: `cont_returnBarrier` not recognised | Task 3 Path B | +| H2: No `JavaCallWrapper` for `enterSpecial` | Task 3 Path A (enterSpecial is bypassed, `isEntryFrame` never reached) | +| H3: Wrong blob-type classification for `enterSpecial` | Task 3 Path A (detection is by nmethod identity, not name/type) | +| H4: Broken FP chain at ContinuationEntry boundary | Task 3 both paths (use `entryFP()` instead of raw FP chain) | +| H5: `@Hidden` / non-standard frame at yield point | Resolved: `Continuation.enter` is a normal JIT nmethod; H5 was a misidentification of the real problem (the `enterSpecial` frame above it, now covered by Path A) | + +**H5 closure:** The `@Hidden` annotation and the non-local yield-return path do not affect the physical JIT frame layout or OopMap coverage of `Continuation.enter`. The profiler walks it correctly via the standard `isNMethod()` path. The actual difficulty was always the `enterSpecial` frame above it, which is now addressed by Path A. + +### Placeholder scan + +None found. + +### Type consistency + +- `VMContinuationEntry` is defined in Task 2 Steps 1–5, used in Task 3 Steps 1–2. `entryFP()` defined in Step 5, called in Steps 1 and 2 of Task 3. Consistent. +- `WALKVM_CONT_BARRIER_HIT`, `WALKVM_ENTER_SPECIAL_HIT`, `WALKVM_CONT_ENTRY_NULL` defined in Task 1, used in Task 3. Consistent. +- `isContReturnBarrier()` and `enterSpecialNMethod()` defined in Task 2 Step 4, called in Task 3 Steps 1 and 2. Consistent. +- `_enter_special_nm` declared in Task 2 Step 3, defined in Task 2 Step 7, initialised in Task 2 Step 8, returned by `enterSpecialNMethod()` in Step 4. Consistent. diff --git a/doc/plans/2026-04-01-remove-jmethodid-design.md b/doc/plans/2026-04-01-remove-jmethodid-design.md new file mode 100644 index 0000000000..fc7ff5e150 --- /dev/null +++ b/doc/plans/2026-04-01-remove-jmethodid-design.md @@ -0,0 +1,190 @@ +# Removing jmethodID from walkVM Stack Traces — Design Document + +**Date:** 2026-04-01 +**Author:** Jaroslav Bachorik +**Status:** DRAFT + +--- + +## Problem Statement + +The profiler currently uses `jmethodID` to identify Java methods in collected stack traces. This has several problems: + +1. **jmethodID lifecycle is tied to ClassLoaderData.** When a class is unloaded, its jmethodIDs become stale. The profiler must validate them before use, and any collected stack traces referencing unloaded methods lose their method identity. + +2. **jmethodID resolution requires JVMTI calls.** Converting a jmethodID to human-readable form (`GetMethodName`, `GetMethodDeclaringClass`, `GetClassSignature`) requires JVMTI calls that cannot run in a signal handler. This forces a two-phase approach: collect opaque IDs during sampling, resolve to strings later during flush. + +3. **jmethodID allocation forces JVM work.** Reading `VMMethod::id()` navigates `ConstMethod → ConstantPool → pool_holder (Klass) → _methods_jmethod_ids` array. If the jmethodID hasn't been allocated yet, it's NULL and the frame is lost. The JVM lazily allocates these on first JVMTI/JNI access, so methods that were never touched by JVMTI have no jmethodID. + +4. **Stale jmethodID detection is fragile.** `isStaleMethodId()` dereferences the jmethodID and checks validity — this races with GC and class unloading. + +## Proposed Solution + +Replace `jmethodID` with a **profiler-computed method identifier** derived from the JFR trace ID scheme, and install an **inline function hook** on `ClassLoaderDataGraph::purge()` to resolve method identifiers to symbol strings before class metadata is freed. + +### Method Identifier Design + +The JFR method ID formula is: + +``` +method_id = (klass_trace_id << 16) | orig_method_idnum +``` + +Where: +- `klass_trace_id` is read from `Klass::_trace_id` (8 bytes, JFR-assigned, unique per class) +- `orig_method_idnum` is read from `ConstMethod::_orig_method_idnum` (2 bytes, stable across method redefinition) + +This ID is computable in a signal handler with two memory loads and a bitwise OR. No allocation, no locks, no JNI. It's stable across method redefinition (uses `orig_method_idnum`, not `method_idnum`). + +**Scope:** HotSpot only. `Klass::_trace_id` is a JFR-specific field not present on J9 or Zing. On those JVMs, the profiler continues to use jmethodID as a fallback. + +### Klass Trace ID Epoch Bits + +The raw `_trace_id` value contains epoch/metadata bits in addition to the base class identity. The profiler must mask these consistently. The exact mask depends on JFR internals (`TRACE_ID_SHIFT`, epoch bit positions) but the key invariant is: the same mask applied at collection time and at resolution time produces matching IDs. We maintain a mapping from the masked klass trace ID to a Klass pointer, populated incrementally during stack walks, which also serves as the invalidation mechanism during class unloading. + +### ConstantPool Symbol Resolution + +To resolve a method's name and signature from raw memory, we need to navigate: + +``` +Method → _constMethod (ConstMethod*) → _name_index, _signature_index (u2 indices) +ConstMethod → _constants (ConstantPool*) +ConstantPool → entry array → Symbol* at given index +``` + +The ConstantPool entry array is an inline `intptr_t[]` starting immediately after the ConstantPool header: + +```cpp +// From constantPool.hpp:139 +intptr_t* base() const { return (intptr_t*)(((char*)this) + sizeof(ConstantPool)); } + +// symbol_at(index) = *(Symbol**)&base()[index] +``` + +`sizeof(ConstantPool)` is already available via `VMConstantPool::type_size()` (read from `gHotSpotVMTypes` at agent init). So the resolution is: + +```cpp +Symbol* cp_symbol_at(char* cp, int index) { + intptr_t* base = (intptr_t*)(cp + VMConstantPool::type_size()); + return *(Symbol**)&base[index]; +} +``` + +No new VMStructs offset needed for this — just the existing type size plus the new `_constmethod_name_index_offset` and `_constmethod_signature_index_offset`. + +### Hooking ClassLoaderDataGraph::purge() + +When classes are unloaded, the metadata (Klass*, Method*, Symbol*) is freed during `ClassLoaderDataGraph::purge()`. We hook this function to resolve any pending method IDs to strings before the memory is reclaimed. + +**Why purge()?** The unloading sequence across all GCs is: + +``` +① SystemDictionary::do_unloading() → marks CLDs dead, Method*/Klass* still valid +② nmethod unlink + purge + free → compiled code freed +③ ClassLoaderDataGraph::purge() → Metaspace freed (Klass*, Method*, Symbol* gone) +``` + +At purge() entry, nmethod memory is already freed but we don't need it — we have the method IDs from our stack traces. All Klass*/Method*/Symbol* metadata is still in metaspace and readable. + +**Hook mechanism:** Inline function patching (trampoline). We overwrite the first bytes of `purge()` with a jump to our hook function. The hook runs on the GC thread at safepoint — no signal handler constraints, but must be fast and avoid JVM locks. + +**Symbol availability:** +- Linux: `_ZN20ClassLoaderDataGraph5purgeEb` is dynamically exported (version script `global: *;`) +- macOS: Present in symbol table as local text (`t`), findable via Mach-O parsing (profiler already does this) + +**Walking the unloading list:** At purge() entry, `ClassUnloadingContext::_context->_cld_head` points to the linked list of ClassLoaderData being unloaded. Each CLD has `_klasses` (head of Klass chain via `_next_link`). Each InstanceKlass has `_methods` (Array\). The `_cld_head` field is at offset 0 in ClassUnloadingContext (CHeapObj has no vtable, _cld_head is the first instance field). + +### Safepoint Duration Impact + +The hook runs at a GC safepoint, extending pause time. For typical workloads (few classes unloaded per GC), the overhead is microseconds. For scripting-heavy workloads (Groovy, Clojure) that generate thousands of classes, the cost scales with the number of methods in unloaded classes. We accept this tradeoff and will instrument the hook duration for monitoring. + +### Data Flow Overview + +``` + ┌──────────────────────────────────────────────────┐ + │ Signal Handler (sampling) │ + │ │ + │ nmethod → Method* → {klass_trace_id, │ + │ orig_method_idnum} │ + │ Store profiler_method_id in ASGCT_CallFrame │ + │ Register klass_trace_id → Klass* in live map │ + └──────────────┬───────────────────────────────────┘ + │ + │ profiler_method_id stored in traces + ▼ + ┌──────────────────────────────────────────────────┐ + │ Flush / Recording Output │ + │ │ + │ For each frame: │ + │ Check MethodSymbolCache (for unloaded classes) │ + │ Or resolve from live Klass* (for live classes) │ + │ → {class_name, method_name, signature} │ + └──────────────────────────────────────────────────┘ + ▲ + │ cache populated before metadata freed + │ + ┌──────────────────────────────────────────────────┐ + │ purge() Hook (GC safepoint) │ + │ │ + │ Walk ClassUnloadingContext._cld_head chain: │ + │ CLD → _klasses → each InstanceKlass: │ + │ Read _trace_id, _methods array │ + │ For each Method*: │ + │ Compute profiler_method_id │ + │ Read name/signature via ConstantPool │ + │ Store in MethodSymbolCache │ + │ Call original purge() │ + └──────────────────────────────────────────────────┘ +``` + +### New VMStructs Offsets Required + +All discoverable from `gHotSpotVMStructs` (exported by HotSpot): + +| Variable | HotSpot Type.Field | Purpose | +|---|---|---| +| `_klass_next_link_offset` | `Klass._next_link` | Walk CLD's Klass chain | +| `_klass_trace_id_offset` | `Klass._trace_id` | JFR klass trace ID | +| `_constmethod_orig_idnum_offset` | `ConstMethod._orig_method_idnum` | Stable method index | +| `_constmethod_name_index_offset` | `ConstMethod._name_index` | CP index for name | +| `_constmethod_signature_index_offset` | `ConstMethod._signature_index` | CP index for signature | +| `_cld_klasses_offset` | `ClassLoaderData._klasses` | Head of Klass chain | +| `_cld_unloading_next_offset` | `ClassLoaderData._unloading_next` | Unloading CLD list | + +Symbols resolved via `libjvm` symbol table lookup (not gHotSpotVMStructs): + +| Symbol | Mangled Name | Purpose | +|---|---|---| +| `ClassUnloadingContext::_context` | `_ZN21ClassUnloadingContext8_contextE` | Singleton pointer | +| `ClassLoaderDataGraph::purge` | `_ZN20ClassLoaderDataGraph5purgeEb` | Hook target | + +### Scope of Change + +| Component | What Changes | +|---|---| +| `vmStructs.h/cpp` | New offset variables, `jfr_method_id()` on VMMethod, feature flag | +| `stackWalker.cpp` | Replace `method->id()` / `getMethodId()` with `method->jfr_method_id()` | +| `methodId.h` (new) | `profiler_method_id` type, pack/unpack helpers | +| `methodSymbolCache.h/cpp` (new) | Lock-free cache: method_id → {class, method, sig} | +| `classUnloadHook.h/cpp` (new) | purge() hook install/uninstall, pre-purge callback | +| `flightRecorder.cpp` | Replace JVMTI-based `fillJavaMethodInfo()` with cache/live lookup | +| `profiler.cpp` | Hook lifecycle (install on start, uninstall on stop) | + +### What Gets Removed + +- `getMethodId()` function in stackWalker.cpp +- `VMMethod::id()` and `VMMethod::validatedId()` +- `_jmethod_ids_offset` from VMStructs +- `_can_dereference_jmethod_id` flag and `isStaleMethodId()` check +- `check_jmethodID_hotspot()` / `check_jmethodID_J9()` validation functions +- JVMTI `GetMethodName` / `GetMethodDeclaringClass` / `GetClassSignature` calls in `fillJavaMethodInfo()` + +### Risks and Mitigations + +| Risk | Mitigation | +|---|---| +| J9/Zing has no `_trace_id` | Feature-gated: use jmethodID fallback when `_klass_trace_id_offset == -1` | +| Epoch bits in `_trace_id` change | Apply consistent masking at collection and resolution; documented separately | +| Safepoint pause extension | Instrument hook duration; profile with scripting workloads before ship | +| Version-specific struct offsets | All offsets from gHotSpotVMStructs — auto-adapts across JDK versions | +| Concurrent class unloading (ZGC/Shenandoah) | purge() is the convergence point for all GCs; hook is GC-agnostic | diff --git a/doc/plans/2026-04-01-remove-jmethodid-execution.md b/doc/plans/2026-04-01-remove-jmethodid-execution.md new file mode 100644 index 0000000000..67ef550900 --- /dev/null +++ b/doc/plans/2026-04-01-remove-jmethodid-execution.md @@ -0,0 +1,649 @@ +# Remove jmethodID from walkVM Traces — Execution Plan + +> **Status:** NOT STARTED +> +> **Companion doc:** [Design Document](2026-04-01-remove-jmethodid-design.md) + +**Goal:** Replace `jmethodID` with JFR-derived profiler method IDs in walkVM-based stack traces. Install an inline hook on `ClassLoaderDataGraph::purge()` to resolve method symbols before class unloading frees metadata. Remove all jmethodID usage from the HotSpot stack walking path. + +**Tech Stack:** C++ (C++14), HotSpot VMStructs introspection, inline function patching, lock-free data structures. + +**Scope:** HotSpot JDK 11+ only. J9/Zing retain existing jmethodID path behind feature gate. + +--- + +## File Map + +| File | Change | +|---|---| +| `ddprof-lib/src/main/cpp/hotspot/vmStructs.h` | Add 7 new offset variables, `jfr_method_id()` accessor on VMMethod, `_has_jfr_trace_ids` feature flag, `cp_symbol_at()` helper | +| `ddprof-lib/src/main/cpp/hotspot/vmStructs.cpp` | Register new offsets in `init_offsets_and_addresses()`, resolve `ClassUnloadingContext::_context` and `purge()` symbols, update `verify_offsets()` and `resolveOffsets()` | +| `ddprof-lib/src/main/cpp/methodId.h` | **New file.** `profiler_method_id` typedef, `make_method_id()`, `klass_id_from()`, `method_idnum_from()` | +| `ddprof-lib/src/main/cpp/methodSymbolCache.h` | **New file.** Pre-allocated, lock-free hash map: `profiler_method_id → {class_name, method_name, signature}` | +| `ddprof-lib/src/main/cpp/methodSymbolCache.cpp` | **New file.** Implementation | +| `ddprof-lib/src/main/cpp/classUnloadHook.h` | **New file.** Hook install/uninstall API, trampoline management | +| `ddprof-lib/src/main/cpp/classUnloadHook.cpp` | **New file.** Inline patching of `purge()`, pre-purge callback that walks unloading CLDs and populates MethodSymbolCache | +| `ddprof-lib/src/main/cpp/stackWalker.cpp` | Replace `getMethodId()`/`method->id()` calls with `method->jfr_method_id()` | +| `ddprof-lib/src/main/cpp/flightRecorder.cpp` | Replace `fillJavaMethodInfo()` JVMTI calls with MethodSymbolCache/live-metadata lookup | +| `ddprof-lib/src/main/cpp/flightRecorder.h` | Update MethodMap key type from `jmethodID` to `profiler_method_id` | +| `ddprof-lib/src/main/cpp/profiler.cpp` | Hook lifecycle: install on `start()`, uninstall on `stop()` | +| `ddprof-lib/src/main/cpp/profiler.h` | Remove `_class_unload_hook_trap` (replaced by inline hook), add `ClassUnloadHook*` member | + +--- + +## Task 1: Add methodId.h — Profiler Method ID Type + +**Files:** +- Create: `ddprof-lib/src/main/cpp/methodId.h` + +- [ ] **Step 1: Create the header with type definition and helpers** + + ```cpp + #ifndef _METHOD_ID_H + #define _METHOD_ID_H + + #include + + typedef uint64_t profiler_method_id; + + static const int METHOD_ID_NUM_BITS = 16; + static const profiler_method_id METHOD_ID_NUM_MASK = (1ULL << METHOD_ID_NUM_BITS) - 1; + + static inline profiler_method_id make_method_id(uint64_t klass_trace_id, uint16_t method_idnum) { + // Strip epoch/metadata bits from lower 16 bits of trace_id, + // then combine with method index + return (klass_trace_id & ~METHOD_ID_NUM_MASK) | method_idnum; + } + + static inline uint64_t klass_id_from(profiler_method_id id) { + return id & ~METHOD_ID_NUM_MASK; + } + + static inline uint16_t method_idnum_from(profiler_method_id id) { + return (uint16_t)(id & METHOD_ID_NUM_MASK); + } + + #endif // _METHOD_ID_H + ``` + + Note: The masking of `klass_trace_id` must be validated against JFR's `TRACE_ID_SHIFT` and epoch bit layout. The exact mask may need adjustment after empirical testing with JFR epoch rotation. The key invariant is: the same masking applied at collection time and resolution time produces identical `klass_id_from()` values. + +--- + +## Task 2: Add New VMStructs Offsets + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/hotspot/vmStructs.h` +- Modify: `ddprof-lib/src/main/cpp/hotspot/vmStructs.cpp` + +- [ ] **Step 1: Declare new offset variables in vmStructs.h** + + Add to the `DECLARE_TYPE_FIELD_DO` macros, in the appropriate type sections: + + In the `VMKlass` section (near line 195-198, after `_class_loader_data_offset`): + ```cpp + field(_klass_next_link_offset, offset, MATCH_SYMBOLS("_next_link")) + field(_klass_trace_id_offset, offset, MATCH_SYMBOLS("_trace_id")) + ``` + + In the `VMConstMethod` section (near line 188-189, after `_constmethod_idnum_offset`): + ```cpp + field(_constmethod_orig_idnum_offset, offset, MATCH_SYMBOLS("_orig_method_idnum")) + field(_constmethod_name_index_offset, offset, MATCH_SYMBOLS("_name_index")) + field(_constmethod_signature_index_offset, offset, MATCH_SYMBOLS("_signature_index")) + ``` + + In the `VMClassLoaderData` section (near line 201, after `_class_loader_data_next_offset`): + ```cpp + field(_cld_klasses_offset, offset, MATCH_SYMBOLS("_klasses")) + field(_cld_unloading_next_offset, offset, MATCH_SYMBOLS("_unloading_next")) + ``` + +- [ ] **Step 2: Add feature flag** + + In vmStructs.h, near the other feature flags (around line 362): + ```cpp + static bool _has_jfr_trace_ids; + ``` + + In vmStructs.cpp `resolveOffsets()`, set it: + ```cpp + _has_jfr_trace_ids = (_klass_trace_id_offset >= 0) + && (_constmethod_orig_idnum_offset >= 0); + ``` + +- [ ] **Step 3: Add symbol resolution for ClassUnloadingContext and purge()** + + In vmStructs.cpp, add static variables: + ```cpp + static void** _class_unloading_context_addr; // &ClassUnloadingContext::_context + static void* _purge_entry; // address of ClassLoaderDataGraph::purge(bool) + ``` + + In `initJvmFunctions()` or a new `initHookTargets()`: + ```cpp + _class_unloading_context_addr = (void**)_libjvm->findSymbol( + "_ZN21ClassUnloadingContext8_contextE"); + _purge_entry = _libjvm->findSymbol( + "_ZN20ClassLoaderDataGraph5purgeEb"); + ``` + + Note: Mangled names are for GCC/Clang on Linux/macOS. On macOS, symbols have a leading underscore: `__ZN21ClassUnloadingContext8_contextE`. The profiler's existing symbol resolution handles this. + +- [ ] **Step 4: Add jfr_method_id() to VMMethod** + + In vmStructs.h, inside the `VMMethod` declaration (DECLARE block): + ```cpp + profiler_method_id jfr_method_id() { + if (!_has_jfr_trace_ids) return 0; + + const char* const_method = (const char*)SafeAccess::load( + (void**)at(_method_constmethod_offset)); + if (!goodPtr(const_method)) return 0; + + uint16_t idnum = (uint16_t)SafeAccess::load16( + (int16_t*)(const_method + _constmethod_orig_idnum_offset)); + + const char* cpool = (const char*)SafeAccess::loadPtr( + (void**)(const_method + _constmethod_constants_offset), nullptr); + if (!goodPtr(cpool)) return 0; + + const char* holder = (const char*)SafeAccess::loadPtr( + (void**)(cpool + _pool_holder_offset), nullptr); + if (!goodPtr(holder)) return 0; + + uint64_t trace_id = *(uint64_t*)(holder + _klass_trace_id_offset); + + return make_method_id(trace_id, idnum); + } + ``` + +- [ ] **Step 5: Add cp_symbol_at() helper** + + In vmStructs.h, add a static helper: + ```cpp + static Symbol* cp_symbol_at(const char* cp, int index) { + intptr_t* base = (intptr_t*)(cp + VMConstantPool::type_size()); + return *(Symbol**)&base[index]; + } + ``` + + This works because ConstantPool entries are an inline `intptr_t[]` array starting at offset `sizeof(ConstantPool)` from the ConstantPool pointer. Each entry is 8 bytes (pointer-sized). The `name_index` and `signature_index` from ConstMethod index directly into this array, where UTF-8 entries hold `Symbol*` values. + +- [ ] **Step 6: Update verify_offsets()** + + Add assertions for new offsets. Gate `_klass_trace_id_offset` on JFR availability (don't assert on non-JFR builds or J9). All other offsets should be present on any HotSpot JDK 11+. + +**Testing gate:** Build and run existing tests. Verify new offsets resolve correctly on JDK 11, 17, 21, 25 by adding a debug log at init time. + +--- + +## Task 3: Implement MethodSymbolCache + +**Files:** +- Create: `ddprof-lib/src/main/cpp/methodSymbolCache.h` +- Create: `ddprof-lib/src/main/cpp/methodSymbolCache.cpp` + +- [ ] **Step 1: Design the cache data structure** + + Requirements: + - Pre-allocated (no malloc during purge() hook — runs at safepoint) + - Supports concurrent insert (from purge hook on GC thread) and read (from flush on profiler thread) — but in practice these don't overlap since purge runs at safepoint + - Open-addressing hash map keyed by `profiler_method_id` + - String storage in a pre-allocated arena (bump allocator) + + ```cpp + struct MethodSymbolEntry { + profiler_method_id id; // 0 = empty slot + const char* class_name; // pointer into string arena + uint16_t class_name_len; + const char* method_name; + uint16_t method_name_len; + const char* signature; + uint16_t signature_len; + }; + + class MethodSymbolCache { + MethodSymbolEntry* _table; + size_t _capacity; // power of 2 + std::atomic _size; + + char* _string_arena; // pre-allocated string storage + std::atomic _arena_pos; + size_t _arena_capacity; + + public: + MethodSymbolCache(size_t capacity, size_t arena_bytes); + ~MethodSymbolCache(); + + // Copy symbol bytes into arena and store entry. + // Safe to call from safepoint (no malloc). + bool put(profiler_method_id id, + const char* cls, int cls_len, + const char* name, int name_len, + const char* sig, int sig_len); + + const MethodSymbolEntry* get(profiler_method_id id) const; + void clear(); // reset for next recording + }; + ``` + +- [ ] **Step 2: Implement put/get with open addressing** + + Use linear probing. Key = `profiler_method_id`, hash = mix64(id). Empty slot sentinel = id == 0 (valid method IDs are never 0 since klass_trace_id > 0). On collision, linear probe. On arena exhaustion, stop inserting (best-effort — log a counter). + +- [ ] **Step 3: Implement string arena** + + Simple bump allocator: `_arena_pos` advances atomically. `put()` copies symbol bytes from HotSpot Symbol objects into the arena. Returns pointer into the arena for the entry. + +**Testing gate:** Unit test: create cache, insert entries, look up, verify strings match. Test capacity limits and arena exhaustion. + +--- + +## Task 4: Implement Class Unload Hook + +**Files:** +- Create: `ddprof-lib/src/main/cpp/classUnloadHook.h` +- Create: `ddprof-lib/src/main/cpp/classUnloadHook.cpp` + +- [ ] **Step 1: Implement inline function patching** + + On install: + 1. `mprotect()` the page containing `purge()` entry to `PROT_READ | PROT_WRITE | PROT_EXEC` (or `PROT_READ | PROT_WRITE` on W^X systems) + 2. Save original bytes (enough for the trampoline jump — 12-14 bytes on x86_64, 16 bytes on aarch64) + 3. Write unconditional jump to `pre_purge_hook`: + - x86_64: `mov rax, ; jmp rax` (10 bytes) or `jmp rel32` if within ±2GB + - aarch64: `ldr x16, .+8; br x16; .quad ` (16 bytes) + 4. Flush instruction cache (`__builtin___clear_cache` on aarch64) + 5. Restore page protection + + On uninstall: reverse — write back saved bytes, flush cache. + + Note: The profiler's existing `Trap` class handles `mprotect` and cache flush. Reuse the `Trap::patch()` infrastructure for page protection management. The actual patching differs (multi-byte trampoline vs single-instruction breakpoint) but the memory protection logic is identical. + +- [ ] **Step 2: Implement pre_purge_hook()** + + ```cpp + static void pre_purge_hook(bool at_safepoint) { + if (_class_unloading_context_addr != nullptr) { + void* ctx = *_class_unloading_context_addr; + if (ctx != nullptr) { + // _cld_head is at offset 0 in ClassUnloadingContext + char* cld = (char*)SafeAccess::loadPtr((void**)ctx, nullptr); + + while (cld != nullptr) { + resolve_cld_methods(cld); + cld = (char*)SafeAccess::loadPtr( + (void**)(cld + _cld_unloading_next_offset), nullptr); + } + } + } + + // Call original purge() + _original_purge(at_safepoint); + } + ``` + +- [ ] **Step 3: Implement resolve_cld_methods()** + + Walk each CLD's Klass chain, and for each InstanceKlass walk the methods array: + + ```cpp + static void resolve_cld_methods(char* cld) { + char* klass = (char*)SafeAccess::loadPtr( + (void**)(cld + _cld_klasses_offset), nullptr); + + while (klass != nullptr) { + // Read klass trace_id + uint64_t trace_id = *(uint64_t*)(klass + _klass_trace_id_offset); + + // Read class name from Klass._name (Symbol*) + char* klass_name_sym = (char*)SafeAccess::loadPtr( + (void**)(klass + _klass_name_offset), nullptr); + uint16_t cls_len = *(uint16_t*)(klass_name_sym + _symbol_length_offset); + const char* cls_body = klass_name_sym + _symbol_body_offset; + + // Read _methods array (Array*) + char* methods_arr = (char*)SafeAccess::loadPtr( + (void**)(klass + _methods_offset), nullptr); + if (methods_arr != nullptr) { + int num_methods = *(int*)methods_arr; // Array._length at offset 0 + // Array._data starts after _length + alignment padding + // Use GrowableArray _array_data_offset or compute: sizeof(int) padded to 8 + char** method_ptrs = (char**)(methods_arr + sizeof(intptr_t)); + + for (int i = 0; i < num_methods; i++) { + char* method = method_ptrs[i]; + if (method == nullptr) continue; + + // Navigate Method → ConstMethod + char* cm = (char*)SafeAccess::loadPtr( + (void**)(method + _method_constmethod_offset), nullptr); + if (cm == nullptr) continue; + + uint16_t idnum = *(uint16_t*)(cm + _constmethod_orig_idnum_offset); + profiler_method_id mid = make_method_id(trace_id, idnum); + + // Read name and signature via constant pool + uint16_t name_idx = *(uint16_t*)(cm + _constmethod_name_index_offset); + uint16_t sig_idx = *(uint16_t*)(cm + _constmethod_signature_index_offset); + char* cpool = (char*)SafeAccess::loadPtr( + (void**)(cm + _constmethod_constants_offset), nullptr); + if (cpool == nullptr) continue; + + // ConstantPool entry array at offset sizeof(ConstantPool) + char* name_sym = (char*)cp_symbol_at(cpool, name_idx); + char* sig_sym = (char*)cp_symbol_at(cpool, sig_idx); + if (name_sym == nullptr || sig_sym == nullptr) continue; + + uint16_t name_len = *(uint16_t*)(name_sym + _symbol_length_offset); + const char* name_body = name_sym + _symbol_body_offset; + uint16_t sig_len = *(uint16_t*)(sig_sym + _symbol_length_offset); + const char* sig_body = sig_sym + _symbol_body_offset; + + _cache->put(mid, cls_body, cls_len, name_body, name_len, sig_body, sig_len); + } + } + + // Next klass in CLD chain + klass = (char*)SafeAccess::loadPtr( + (void**)(klass + _klass_next_link_offset), nullptr); + } + } + ``` + +- [ ] **Step 4: Add trampoline saved-prologue execution** + + The trampoline must execute the original prologue bytes that were overwritten, then jump back into `purge()` past the patch point. Two approaches: + + **Approach A (simpler):** Allocate an executable trampoline buffer. Copy saved prologue bytes into it, followed by a jump to `purge + patch_size`. On call: `pre_purge_hook()` runs the notification logic, then calls the trampoline which executes the original prologue and continues into purge(). + + **Approach B (patch/unpatch):** Before calling original, uninstall the hook (restore original bytes), call `purge()` directly, then reinstall the hook. This is simpler but not thread-safe if multiple GC threads could call purge() concurrently. In practice, purge() is called from a single GC thread at a time, so this is safe. + + Recommend **Approach B** for simplicity. The hook is short-lived (active only during profiling) and purge() is called serially. + +**Testing gate:** Unit test hook install/uninstall. Integration test: run profiler with class-unloading workload (custom ClassLoader that loads + discards classes in a loop), verify no crash and symbols resolve for unloaded methods. + +--- + +## Task 5: Replace jmethodID in walkVM + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/stackWalker.cpp` + +- [ ] **Step 1: Add JFR method ID path for interpreted frames** + + In `walkVM()`, the interpreted frame path (around lines 512-533): + + Current code: + ```cpp + VMMethod* method = ((VMMethod**)fp)[InterpreterFrame::method_offset]; + jmethodID method_id = getMethodId(method); + // ... + fillFrame(frames[depth++], FRAME_INTERPRETED, bci, method_id); + ``` + + Replace with: + ```cpp + VMMethod* method = ((VMMethod**)fp)[InterpreterFrame::method_offset]; + if (VMStructs::hasJfrTraceIds()) { + profiler_method_id mid = method->jfr_method_id(); + if (mid != 0) { + fillFrame(frames[depth++], FRAME_INTERPRETED, bci, (jmethodID)(uintptr_t)mid); + } + } else { + // J9/Zing fallback + jmethodID method_id = getMethodId(method); + if (method_id != NULL) { + fillFrame(frames[depth++], FRAME_INTERPRETED, bci, method_id); + } + } + ``` + + Note: Storing `profiler_method_id` in the `jmethodID` union field via cast. Both are 8 bytes. The frame type or a global flag distinguishes the two formats downstream. + +- [ ] **Step 2: Add JFR method ID path for compiled frames** + + In the compiled frame path (around lines 569, 586): + + Current code: + ```cpp + fillFrame(frames[depth++], type, 0, nm->method()->id()); + ``` + + Replace with: + ```cpp + if (VMStructs::hasJfrTraceIds()) { + profiler_method_id mid = nm->method()->jfr_method_id(); + fillFrame(frames[depth++], type, 0, (jmethodID)(uintptr_t)mid); + } else { + fillFrame(frames[depth++], type, 0, nm->method()->id()); + } + ``` + + Apply the same pattern for inlined frames (around line 586): + ```cpp + fillFrame(frames[depth++], type, scope.bci(), scope.method()->jfr_method_id()); + ``` + +- [ ] **Step 3: Apply same pattern to all other getMethodId() call sites** + + There are 4 call sites for `getMethodId()` in stackWalker.cpp (lines 513, 530, 739, 897). Apply the same conditional pattern to each. + +- [ ] **Step 4: Add hasJfrTraceIds() to VMStructs** + + In vmStructs.h: + ```cpp + static bool hasJfrTraceIds() { return _has_jfr_trace_ids; } + ``` + +**Testing gate:** Run profiler on a Java application. Compare collected stack traces against baseline (jmethodID path). Verify all methods are captured. Run with both HotSpot and J9 to verify fallback. + +--- + +## Task 6: Replace JVMTI Symbol Resolution in FlightRecorder + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/flightRecorder.cpp` +- Modify: `ddprof-lib/src/main/cpp/flightRecorder.h` + +- [ ] **Step 1: Update MethodMap key handling** + + In `flightRecorder.h`, the `MethodMap` class uses `jmethodID` as key. When JFR trace IDs are active, the key becomes a `profiler_method_id` stored in the same `unsigned long` slot. No structural change needed — just ensure the key generation in `makeKey()` handles both formats: + + ```cpp + static unsigned long makeKey(jmethodID method) { + return (unsigned long)method; // works for both jmethodID and profiler_method_id + } + ``` + +- [ ] **Step 2: Add JFR-based fillJavaMethodInfo()** + + Create a new resolution path that does NOT use JVMTI: + + ```cpp + void Lookup::fillJavaMethodInfoFromCache(profiler_method_id mid, MethodInfo* mi) { + // 1. Check MethodSymbolCache (populated by purge hook for unloaded classes) + const MethodSymbolEntry* cached = _method_symbol_cache->get(mid); + if (cached != nullptr) { + // Copy cached strings into MethodInfo + mi->_class = copyString(cached->class_name, cached->class_name_len); + mi->_name = copyString(cached->method_name, cached->method_name_len); + mi->_sig = copyString(cached->signature, cached->signature_len); + mi->_modifiers = 0; + return; + } + + // 2. Class still loaded — resolve from live metadata + // Use the klass_trace_id → Klass* mapping built during stack walks, + // or walk ClassLoaderDataGraph to find the Klass. + // Then: Klass → _methods → Method[idnum] → ConstMethod → CP → Symbol + resolveFromLiveMetadata(mid, mi); + } + ``` + +- [ ] **Step 3: Wire into resolveMethod()** + + In `Lookup::resolveMethod()`, add a branch before the existing JVMTI path: + + ```cpp + if (VMStructs::hasJfrTraceIds() && bci != BCI_NATIVE_FRAME && ...) { + profiler_method_id mid = (profiler_method_id)(uintptr_t)method; + fillJavaMethodInfoFromCache(mid, mi); + } else { + // Existing JVMTI path for J9/Zing or native frames + fillJavaMethodInfo(method, mi); + } + ``` + +- [ ] **Step 4: Implement resolveFromLiveMetadata()** + + For classes that are still loaded (not yet unloaded), resolve directly: + - Decompose `mid` into `klass_id` and `method_idnum` + - Look up Klass* from a `klass_id → Klass*` map (populated during stack walks — see Task 7) + - Validate the Klass* is still alive (read `_class_loader_data`, check CLD is not unloading) + - Navigate: `Klass → _name → Symbol` for class name + - Navigate: `Klass → _methods → Method[idnum] → ConstMethod → {_name_index, _signature_index} → ConstantPool → Symbol` for method name and signature + - Use the same `cp_symbol_at()` helper from VMStructs + +**Testing gate:** Run profiler, produce JFR recording. Compare method names against baseline recording produced with jmethodID path. They must be identical. + +--- + +## Task 7: Add Klass Trace ID → Klass* Live Map + +**Files:** +- Create or extend: `ddprof-lib/src/main/cpp/klassMap.h` (or add to existing structures) +- Modify: `ddprof-lib/src/main/cpp/stackWalker.cpp` + +- [ ] **Step 1: Create a lock-free map from masked klass_trace_id to Klass*** + + Requirements: + - Written from signal handler (during stack walks) — must be async-signal-safe + - Read from flush thread (during symbol resolution) — must handle concurrent access + - Entries invalidated by purge() hook when class is unloaded + + Use the same open-addressing pattern as MethodSymbolCache, pre-allocated. + +- [ ] **Step 2: Populate during stack walks** + + After computing `jfr_method_id()`, register the Klass* in the map: + + ```cpp + profiler_method_id mid = method->jfr_method_id(); + if (mid != 0) { + uint64_t kid = klass_id_from(mid); + const char* holder = /* already loaded in jfr_method_id() — refactor to return it */; + _klass_map->put_if_absent(kid, holder); + } + ``` + + This means `jfr_method_id()` should be refactored to also expose the Klass* it traversed, to avoid re-navigating the pointer chain. + +- [ ] **Step 3: Invalidate entries in purge() hook** + + In `pre_purge_hook()`, after walking unloading CLDs: + ```cpp + // Remove entries for unloaded classes from the klass map + _klass_map->remove(klass_id_from(make_method_id(trace_id, 0))); + ``` + +**Testing gate:** Verify live map correctly tracks and invalidates entries across class load/unload cycles. + +--- + +## Task 8: Hook Lifecycle and Integration + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/profiler.cpp` +- Modify: `ddprof-lib/src/main/cpp/profiler.h` + +- [ ] **Step 1: Add ClassUnloadHook to Profiler** + + In `profiler.h`: + ```cpp + ClassUnloadHook* _class_unload_hook; + MethodSymbolCache* _method_symbol_cache; + ``` + + Remove: `Trap _class_unload_hook_trap;` and `NotifyClassUnloadedFunc`. + +- [ ] **Step 2: Install hook on profiler start** + + In `Profiler::start()`, after VMStructs is ready: + ```cpp + if (VMStructs::hasJfrTraceIds() && VMStructs::purgeEntry() != nullptr) { + _method_symbol_cache = new MethodSymbolCache(64 * 1024, 4 * 1024 * 1024); + _class_unload_hook = new ClassUnloadHook( + VMStructs::purgeEntry(), + VMStructs::classUnloadingContextAddr(), + _method_symbol_cache); + _class_unload_hook->install(); + } + ``` + +- [ ] **Step 3: Uninstall hook on profiler stop** + + In `Profiler::stop()`: + ```cpp + if (_class_unload_hook != nullptr) { + _class_unload_hook->uninstall(); + delete _class_unload_hook; + _class_unload_hook = nullptr; + } + // Flush remaining cache entries to output before clearing + // ... (during final recording flush) + if (_method_symbol_cache != nullptr) { + delete _method_symbol_cache; + _method_symbol_cache = nullptr; + } + ``` + +**Testing gate:** Start/stop profiler multiple times. Verify no crash, no leaked trampolines, no stale hooks. + +--- + +## Task 9: Clean Up Dead Code + +**Files:** +- Modify: `ddprof-lib/src/main/cpp/stackWalker.cpp` +- Modify: `ddprof-lib/src/main/cpp/hotspot/vmStructs.h` +- Modify: `ddprof-lib/src/main/cpp/hotspot/vmStructs.cpp` +- Modify: `ddprof-lib/src/main/cpp/flightRecorder.cpp` + +- [ ] **Step 1: Remove jmethodID validation code (HotSpot path only)** + + After confirming all tests pass with the new path: + + - Remove `getMethodId()` from stackWalker.cpp (lines 74-80) + - Remove `VMMethod::id()` and `VMMethod::validatedId()` from vmStructs.h + - Remove `check_jmethodID_hotspot()` from vmStructs.cpp + - Remove `_can_dereference_jmethod_id` flag + - Remove `isStaleMethodId()` from vmStructs.h + + Keep all of the above for the J9/Zing fallback path, gated behind `!hasJfrTraceIds()`. + +- [ ] **Step 2: Remove JVMTI symbol resolution (HotSpot path only)** + + - Remove `GetMethodName` / `GetMethodDeclaringClass` / `GetClassSignature` calls from the HotSpot code path in `fillJavaMethodInfo()` + - Keep for J9/Zing fallback + +- [ ] **Step 3: Remove _jmethod_ids_offset from VMStructs** + + If no remaining code references `_jmethod_ids_offset` on the HotSpot path, remove it. Keep if needed for J9 fallback. + +**Testing gate:** Full test suite. Verify no jmethodID usage remains in HotSpot code paths. Verify J9 fallback still works. + +--- + +## Execution Order + +| Phase | Tasks | Risk | Gate | +|---|---|---|---| +| **1 — Foundation** | Tasks 1, 2 | Low | Offsets resolve on JDK 11/17/21/25; `jfr_method_id()` returns nonzero values | +| **2 — Cache** | Task 3 | Low | Unit tests pass for cache operations | +| **3 — Hook** | Task 4 | High | Stress test with classloader churn; no crashes; symbols resolve for unloaded methods | +| **4 — Collection** | Tasks 5, 7 | Medium | Stack traces captured with profiler_method_id; compare against jmethodID baseline | +| **5 — Resolution** | Task 6 | Medium | Output recordings identical to jmethodID-based output | +| **6 — Integration** | Task 8 | Medium | Start/stop cycles clean; no resource leaks | +| **7 — Cleanup** | Task 9 | Low | Full test suite; J9 fallback verified | diff --git a/doc/plans/2026-04-21-signal-origin-validation.md b/doc/plans/2026-04-21-signal-origin-validation.md new file mode 100644 index 0000000000..1f82467163 --- /dev/null +++ b/doc/plans/2026-04-21-signal-origin-validation.md @@ -0,0 +1,197 @@ +# Plan: Signal Origin Validation for ddprof Signal Handlers + +## Goal +Make ddprof's signal handlers process only signals that originated from ddprof's own timers / inter-thread kills. Foreign signals are forwarded to the previously-installed handler. + +Context: A Go-cgo shared library (e.g. trivyjni) linked into the JVM can install +`setitimer(ITIMER_PROF)` via `dd-trace-go`'s CPU profiler. That timer is +process-wide and delivers SIGPROF to arbitrary threads — including threads that +were never registered with ddprof's `pthread_create` interceptor. ddprof's +SIGPROF handler currently processes every SIGPROF, which can land on threads +whose dynamic-TLS blocks for libjavaProfiler.so have not been allocated yet. +First-touch access of any `thread_local` on such a thread inside the signal +handler triggers `__tls_get_addr` → `calloc`, which can deadlock against a +malloc lock held by the interrupted thread. + +## Scope + +- SIGPROF (CTimer — default CPU engine, `timer_create`-based) +- SIGVTALRM (WallClock engine, ASGCT variant) +- **Out of scope**: + - SIGSEGV / SIGBUS — synchronous faults use a different discrimination + mechanism (SafeAccess PC range check). + - ITimer (alternate CPU engine, `setitimer(ITIMER_PROF)`-based) — setitimer + signals carry `si_code == SI_KERNEL` with no `sigval` payload, so an + origin cookie cannot be attached. Migrating ITimer to `timer_create` + would duplicate CTimer. ITimer is a fallback for platforms where + `timer_create` is unavailable; deployments vulnerable to the Go + `ITIMER_PROF` deadlock should use CTimer (the default). + +## Design + +### Discriminators by signal source + +| Source | `si_code` | Additional validation | +|---|---|---| +| `timer_create` (CTimer; ITimer post-migration) | `SI_TIMER` (-2) | `siginfo->si_value.sival_ptr == CPU_COOKIE` | +| `rt_tgsigqueueinfo` (WallClock — new) | `SI_QUEUE` (-1) | `siginfo->si_value.sival_ptr == WALLCLOCK_COOKIE` | +| `setitimer` (foreign — Go's CPU profiler) | `SI_KERNEL` / kernel-defined | none → untrusted, must forward | +| `kill` / `raise` | `SI_USER` (0) | none → forward | +| `tgkill` / `pthread_kill` (foreign) | `SI_TKILL` (-6) | none → untrusted, must forward | +| `sigqueue` from another process | `SI_QUEUE` (-1) + foreign cookie | cookie mismatch → forward | + +### Cookie strategy + +Static per-engine sentinels in a shared header. Use non-null, easily +identifiable constants: + +Cookies live in `signalCookie.h` and are exposed via inline functions +(`reinterpret_cast` is not allowed in `constexpr`): + +```cpp +namespace SignalCookie { + inline void* cpu() { return reinterpret_cast(0xDDDD01); } + inline void* wallclock() { return reinterpret_cast(0xDDDD02); } +} +``` + +One cookie for CTimer (the only CPU engine that can carry a cookie), one for +wallclock. Rationale for wallclock needing a cookie: `SI_TKILL` + +`si_pid == getpid()` is **not** sufficient — `si_pid` for `SI_TKILL` is the +sending process's PID, and all threads in our process share the same PID, so +that check accepts any in-process `tgkill` of SIGVTALRM (including from other +libraries). Real isolation requires a payload. + +### Portable per-thread queued signal + +`pthread_sigqueue(3)` is a glibc-only extension; musl does not implement it. +To keep musl support, send via the `rt_tgsigqueueinfo(2)` syscall directly — +it is the kernel primitive both glibc and musl expose via `syscall(...)`: + +```cpp +#include +#include // siginfo_t + +static int tg_sigqueue_thread(pid_t tgid, pid_t tid, int sig, void* cookie) { + siginfo_t si = {}; + si.si_signo = sig; + si.si_code = SI_QUEUE; // must be < 0 to be accepted by kernel + si.si_value.sival_ptr = cookie; + si.si_pid = tgid; // convention: sender's TGID + si.si_uid = getuid(); + return (int)syscall(SYS_rt_tgsigqueueinfo, tgid, tid, sig, &si); +} +``` + +The receiving handler then sees: +- `siginfo->si_code == SI_QUEUE` +- `siginfo->si_value.sival_ptr == WALLCLOCK_COOKIE` + +The kernel rejects any positive `si_code` from userspace (reserved for +kernel-generated signals), so this is safe. + +### Handler gate (common helper) + +```cpp +// os.h +bool OS::isOwnTimerSignal(siginfo_t* si, void* expected_cookie); // SI_TIMER + cookie +bool OS::isOwnQueuedSignal(siginfo_t* si, void* expected_cookie); // SI_QUEUE + cookie +int OS::queueSignalToThread(pid_t tid, int sig, void* cookie); // rt_tgsigqueueinfo wrapper +void OS::forwardForeignSignal(int signo, siginfo_t* si, void* uctx); +``` + +Handler skeleton: + +```cpp +void Engine::signalHandler(int signo, siginfo_t* si, void* uctx) { + if (!OS::isOwnTimerSignal(si, ENGINE_COOKIE)) { + Counters::increment(ENGINE_SIGNAL_FOREIGN); + OS::forwardForeignSignal(signo, si, uctx); + return; + } + Counters::increment(ENGINE_SIGNAL_OWN); + // existing handler body +} +``` + +### Foreign-signal forwarding + +Must preserve correctness for other libraries sharing the signal (e.g. Go's +`runtime.sighandler`). Store the previous `sigaction` at install time and +forward via: + +```cpp +if (prev.sa_flags & SA_SIGINFO) { + prev.sa_sigaction(signo, si, uctx); +} else if (prev.sa_handler != SIG_DFL && prev.sa_handler != SIG_IGN) { + prev.sa_handler(signo); +} +// SIG_DFL / SIG_IGN: drop +``` + +`OS::installSignalHandler` already captures this via the `oldact` out-parameter +of `sigaction` — just need to make it retrievable by signal number. + +## Implementation steps + +1. **`os.h` / `os_linux.cpp`**: add the three helper functions + previous-action + storage keyed by signal number. +2. **`ctimer_linux.cpp`**: set `sev.sigev_value.sival_ptr = CPU_COOKIE` in the + `timer_create` call; add origin gate at top of `signalHandler` (current + line 145). +3. **`itimer.cpp`**: left unchanged — see scope note. Callers who need the + origin check must configure the profiler to use CTimer (the default). +4. **`wallClock.cpp` + `os_linux.cpp`**: + - Replace the current `OS::sendSignalToThread(tid, SIGVTALRM)` implementation + (uses `tgkill` under the hood) with `OS::queueSignalToThread(tid, + SIGVTALRM, WALLCLOCK_COOKIE)` which invokes `rt_tgsigqueueinfo` + directly — portable across glibc and musl. + - Add `SI_QUEUE` + `WALLCLOCK_COOKIE` gate in + `WallClockASGCT::sharedSignalHandler` (current line 52) and the JVMTI + variant. Forward on mismatch. +5. **`counters.h`**: add `CPU_SIGNAL_FOREIGN` / `CPU_SIGNAL_OWN` and + `WALLCLOCK_SIGNAL_FOREIGN` / `WALLCLOCK_SIGNAL_OWN` counters. +6. **Feature flag**: gate the whole change behind `DDPROF_SIGNAL_ORIGIN_CHECK` + (default ON) so it can be disabled if a regression emerges. + +## Testing + +1. **Unit tests** (per engine): + - CPU: `timer_create` + `timer_settime` with `CPU_COOKIE` → assert `OWN` + counter increments. + - CPU: `setitimer(ITIMER_PROF)` → assert `FOREIGN` counter increments and + signal is forwarded. + - CPU: `raise(SIGPROF)` (SI_USER, no cookie) → assert `FOREIGN`. + - Wallclock: `rt_tgsigqueueinfo(..., WALLCLOCK_COOKIE)` → assert `OWN`. + - Wallclock: `tgkill(SIGVTALRM)` from another thread (no cookie) → assert + `FOREIGN` and forwarded. + - Wallclock: `rt_tgsigqueueinfo` with a foreign cookie → assert `FOREIGN`. + - Musl build: confirm `rt_tgsigqueueinfo` path compiles and signals are + delivered with correct `si_code` / `si_value`. +2. **Integration**: a test process that installs `setitimer(ITIMER_PROF)` + alongside ddprof; verify ddprof samples are all attributed to registered + threads (none from kernel-selected pre-existing threads). +3. **Regression**: reproduce the trivyjni scenario — load a Go-cgo lib that + calls `profiler.Start(CPUProfile)`, run with ddprof active, confirm no + lockup and no ddprof samples from Go-runtime threads that weren't explicitly + registered. + +## Metrics to watch post-deploy + +- `CPU_SIGNAL_FOREIGN > 0` on services with Go cgo + dd-trace-go CPU profiler + → confirms the fix is discriminating. +- `CPU_SIGNAL_OWN` unchanged vs. pre-fix baseline → confirms no false negatives. +- `WALLCLOCK_SIGNAL_OWN` unchanged vs. pre-fix baseline → confirms the + `rt_tgsigqueueinfo` migration preserves delivery. +- `WALLCLOCK_SIGNAL_FOREIGN` should be 0 under normal conditions; non-zero + indicates another in-process sender or external `sigqueue` to us. +- P99 request latency on trivy-affected services → should lose the lockup tail. + +## What this does NOT fix + +- SIGSEGV on pre-existing threads touching a not-yet-allocated `thread_local` + for the first time (separate audit — need to enumerate profiler-owned + `thread_local`s and either eliminate, make allocation-free, or pre-touch at + registration). +- If Go ever adds a `SIGEV_SIGNAL`-targeted `SIGPROF` with its own sival → + our cookie is non-null and distinct, so it is still filtered. diff --git a/doc/review-pr-488-comparison.md b/doc/review-pr-488-comparison.md new file mode 100644 index 0000000000..eb98e95938 --- /dev/null +++ b/doc/review-pr-488-comparison.md @@ -0,0 +1,122 @@ +# PR #488 Review Comparison: Simple vs Muse + +## Overview + +| Dimension | Simple | Muse | +|-----------|--------|------| +| Findings | 11 | 24 | +| HIGH | 2 | 5 | +| MEDIUM | 6 | 10 | +| LOW | 3 | 4 | +| NIT | 0 | 5 | +| Files reviewed | 34 | ~20 unique files | +| Narrative summary | Yes | No | +| Tier classification | No | Yes (tier 2 = actionable, tier 3 = advisory) | +| Confidence scores | No | Yes (40–95%) | + +--- + +## Findings Agreement Matrix + +### Agreed (both reviewers flagged the same root cause) + +| Issue | Simple | Muse | Severity agreement | +|-------|--------|------|--------------------| +| PLT/GOT `__ATOMIC_RELAXED` + `_orig_*` plain stores | HIGH #2 | HIGH #3 | **Full** — both flag the same ARM64 ordering hazard | +| fd-type-cache not invalidated on fd reuse | HIGH #1 | MEDIUM #7 | **Partial** — Simple rates it HIGH (wrong events produced); Muse MEDIUM (confidence 70%) | +| fd-cache TOCTOU / stale address on reuse | HIGH #1 (implicit) | MEDIUM #6 | **Partial** — Simple collapses both into one HIGH; Muse splits them | +| Rate-limit test upper bound not asserted | MEDIUM #8 | NIT #24 | **Diverged** — Simple MEDIUM, Muse NIT (confidence 65%) | + +### Unique to Simple (missed by Muse) + +| # | Severity | File | Issue | +|---|----------|------|-------| +| 3 | MEDIUM | `hotspotSupport.cpp:1109` | `walkJavaStack` split into two byte-for-byte duplicate branches for BCI_NATIVE_MALLOC and BCI_NATIVE_SOCKET | +| 4 | MEDIUM | `hotspotSupport.cpp:70` | `BCI_NATIVE_SOCKET` missing from `eventTypeFromBCI()` — falls to `default: EXECUTION_SAMPLE` causing wrong EventType for socket events with `cstack=vm` | +| 5 | MEDIUM | `nativeSocketSampler.cpp:631` | `std::mutex _fd_cache_mutex` held while SIGPROF can fire — not async-signal-safe, potential deadlock | +| 6 | MEDIUM | `nativeSocketSampler.cpp:565` | `inet_ntop()` return value ignored; `host[]` may be uninitialised on failure | +| 9 | MEDIUM | `NativeSocketEnabledTest.java` | Redundant test — `NativeSocketEventFieldsTest` makes strictly stronger assertions | +| 10 | LOW | `livenessTracker.cpp:214` | Only `_record_heap_usage` moved above the `_initialized` guard; other args fields not updated on repeat calls with no documentation | + +### Unique to Muse (missed by Simple) + +| # | Severity | File | Issue | +|---|----------|------|-------| +| 1 | HIGH | `libraryPatcher.h:52` | `_socket_active` plain bool read/written without sync — C++11 data race (confidence 95%) | +| 2 | HIGH | `libraryPatcher_linux.cpp:386` | Duplicate-patch detection matches by lib pointer not per-GOT-slot — can skip unpatched slots | +| 4 | HIGH | `nativeSocketSampler_ut.cpp:79` | `write_hook`/`read_hook` have zero unit-test coverage; `isSocket()` guard path untested | +| 5 | HIGH | `nativesocket/` (Java tests) | No start → transfer → stop → start → transfer lifecycle test; stale cache or non-reset epoch would silently suppress events | +| 8 | MEDIUM | `nativeSocketSampler.h:989` | Non-GLIBC stub `check()` returns `Error::OK` — feature silently does nothing on musl with no warning | +| 9 | MEDIUM | `NativeSocketStackTraceTest.java:2748` | Stack-trace test only runs default cstack; AGCT/dwarf/fp paths introduced by new `BCI_NATIVE_SOCKET` branch untested | +| 10 | MEDIUM | `arguments.cpp:389` | Negative `natsock` interval validation branch has no test coverage | +| 11 | MEDIUM | `nativeSocketSampler.cpp:125` | `std::string addr` in hot sampled path causes heap allocation for IPv6 addresses | +| 12 | MEDIUM | `libraryPatcher_linux.cpp:373` | `realpath()` I/O call inside `ExclusiveLockGuard` — blocks lock under FUSE/proc paths | +| 13 | MEDIUM | `nativeSocketSampler.cpp:659` | In-flight hooks call `recordSample` after `stop()` without drain; safety depends on undocumented internal guard | +| 14 | MEDIUM | `rateLimiter.h:102` | PID controller loses one count per window due to `exchange(_event_count, 0)` before signal computation | +| 15 | LOW | `nativeSocketSampler.h:960` | O(N=65536) reset loop on every `start()` adds ~65 µs startup latency unnecessarily | +| 16 | LOW | `nativeSocketSampler.cpp:264` | `clearFdCache()` only called in `stop()`, not `start()` — stale entries survive restart if `start()` called without prior `stop()` | +| 17 | LOW | `nativeSocketSampler.cpp:544` | `thread_local PoissonSampler` may be destroyed during JVM shutdown while hooks still installed | +| 19 | LOW | `libraryPatcher_linux.cpp:363` | `_orig_*` reassignment on re-patching may overwrite with already-hooked pointer via `RTLD_NEXT` | + +--- + +## Severity Disagreements + +### fd-type-cache staleness on fd reuse +- **Simple → HIGH**: justified by correctness impact — wrong events emitted or sampling silently skipped. +- **Muse → MEDIUM** (#7, confidence 70%): acknowledges the same root cause but rates it lower, noting the fix options clearly. +- **Verdict**: Simple's HIGH is defensible given the user-visible impact (incorrect JFR events). Muse's lower confidence suggests hedging, not a real disagreement on the bug. + +### Rate-limit test upper bound +- **Simple → MEDIUM**: argues the test provides "essentially no signal" given the 100× ceiling. +- **Muse → NIT** (#24, confidence 65%): same observation framed as a nit. +- **Verdict**: Simple is more actionable here — calling it MEDIUM puts it in the must-fix category for a useful test suite. + +--- + +## Coverage Analysis + +### What Simple did better +- **Code-logic bugs** that require understanding cross-file control flow: the `eventTypeFromBCI()` omission (#4) and the duplicate `walkJavaStack` branches (#3) are pure correctness bugs that Muse missed entirely. +- **Signal-safety**: the `std::mutex` + SIGPROF deadlock risk (#5) is a correctness hazard Muse did not flag. +- **Test redundancy**: identified the entirely redundant `NativeSocketEnabledTest` (#9). +- **Narrative synthesis**: the summary paragraph ties findings together and names the three risk centres (fd reuse, memory ordering, test quality) explicitly. + +### What Muse did better +- **Concurrency surface area**: caught `_socket_active` data race (#1), duplicate-patch detection by lib pointer (#2), `_orig_*` reassignment on re-patching (#19), and in-flight hook drain (#13) — four distinct concurrency hazards Simple missed. +- **Test coverage gaps**: systematic enumeration of untested code paths (write/read hooks, cstack modes, restart lifecycle, negative interval validation). +- **Platform correctness**: non-GLIBC silent no-op (#8) and musl disabled-test inconsistency (#22) are platform-specific gaps Simple skipped. +- **Performance**: heap allocation in hot path (#11), O(N) start loop (#15), `realpath()` under lock (#12). +- **Lifecycle bugs**: `clearFdCache()` not called on `start()` (#16), `thread_local` destructor vs active hooks (#17). +- **Confidence scores and tier classification** make it easier to triage actionable vs advisory findings. + +--- + +## Recommended Priority Fix List (union, deduplicated) + +| Priority | Issue | Source | +|----------|-------|--------| +| P0 | `_socket_active` data race (plain bool, C++11 UB) | Muse #1 | +| P0 | `_orig_*` plain stores + `__ATOMIC_RELAXED` GOT publish — ARM64 ordering hole | Both | +| P0 | fd-type-cache never invalidated on fd reuse → wrong JFR events | Both | +| P0 | `BCI_NATIVE_SOCKET` missing from `eventTypeFromBCI()` → wrong EventType under `cstack=vm` | Simple #4 | +| P0 | `std::mutex` held while SIGPROF can fire → potential deadlock | Simple #5 | +| P1 | Duplicate `walkJavaStack` branches — maintenance hazard | Simple #3 | +| P1 | Duplicate-patch detection by lib pointer skips individual GOT slots | Muse #2 | +| P1 | Add restart lifecycle integration test | Muse #5 | +| P1 | Add write/read hook unit tests | Muse #4 | +| P1 | Non-GLIBC stub must log warning, not silently succeed | Muse #8 | +| P1 | `inet_ntop()` return value unchecked — possible garbage in output | Simple #6 | +| P2 | Tighten accuracy and rate-limit test assertions | Both | +| P2 | `clearFdCache()` at `start()`, not only `stop()` | Muse #16 | +| P2 | Stack-trace test parameterised over cstack modes | Muse #9 | +| P3 | IPv6 `std::string` heap allocation in hot path | Muse #11 | +| P3 | O(N) fd-type-cache reset loop on start | Muse #15 | +| P3 | `realpath()` inside `ExclusiveLockGuard` | Muse #12 | +| P3 | Remove redundant `NativeSocketEnabledTest` | Simple #9 | + +--- + +## Conclusion + +The two reviews are **complementary, not redundant**. Simple excels at cross-file logic tracing (finding the `eventTypeFromBCI` omission and the duplicated `walkJavaStack` branch) and signal-safety analysis. Muse provides broader concurrency coverage, systematic test-gap enumeration, and platform-specific edge cases. Together they surface **five distinct P0-severity correctness and safety bugs** that neither review alone would have caught in full. Using both reviews together is the right call for a feature of this complexity. diff --git a/doc/review-pr-488-muse.json b/doc/review-pr-488-muse.json new file mode 100644 index 0000000000..aed3ca3879 --- /dev/null +++ b/doc/review-pr-488-muse.json @@ -0,0 +1,386 @@ +[ + { + "number": 1, + "severity": "HIGH", + "file": "ddprof-lib/src/main/cpp/libraryPatcher.h", + "line": 52, + "concern": "data-race on _socket_active bool read/write without synchronization", + "issue": "_socket_active is a plain bool written under _lock in patch/unpatch_socket_functions but read without _lock in install_socket_hooks (called from dlopen_hook on arbitrary application threads concurrently with stop()). This creates a C++11 data race (undefined behavior); a dlopen thread may observe stale true and invoke patch_socket_functions after stop has completed.", + "suggestion": "Declare _socket_active as std::atomic and use load(std::memory_order_acquire) in install_socket_hooks; store with memory_order_release in patch/unpatch_socket_functions to establish proper synchronization.", + "confidence": 95, + "voice": "merged: bastet-nexus, khonsu-guardian, nova-herald, pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 2, + "severity": "HIGH", + "file": "ddprof-lib/src/main/cpp/libraryPatcher_linux.cpp", + "line": 386, + "concern": "duplicate-patch detection by lib pointer instead of per-GOT-slot tracking", + "issue": "_socket_entries stores one entry per GOT slot but the already_patched check matches by _lib pointer; if patch_socket_functions is called again (e.g. via install_socket_hooks after dlopen) for a library whose send but not recv slot was patched, the lib-pointer match will skip all four slots even though the newly discovered slots are unpatched, causing selective skipping of GOT slots.", + "suggestion": "Track patched (lib, import_id) pairs, not just the lib pointer, so each GOT slot is guarded independently; or reset _socket_size = 0 before re-patching and re-scan all known libs from scratch on each call.", + "confidence": 82, + "voice": "bastet-nexus", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 3, + "severity": "HIGH", + "file": "ddprof-lib/src/main/cpp/libraryPatcher_linux.cpp", + "line": 363, + "concern": "PLT/GOT stores and _orig_* pointers lack proper atomic memory ordering on weak architectures", + "issue": "GOT entries are written with __ATOMIC_RELAXED and _orig_send/_recv/_write/_read are assigned with relaxed semantics, and _socket_active is set to true without an intervening release fence. On ARM64 a thread in install_socket_hooks or a hook body may observe _socket_active==true before the hook pointer stores are visible, or read stale _orig_* pointers, causing null dereference or calling through incorrect function pointers.", + "suggestion": "Use __ATOMIC_RELEASE for the _orig_* stores and __ATOMIC_ACQUIRE for the GOT stores that install the hooks. Insert __atomic_thread_fence(__ATOMIC_RELEASE) after all GOT __atomic_store_n calls and before setting _socket_active = true to establish happens-before between pointer assignment and hook body reads.", + "confidence": 77, + "voice": "merged: bastet-nexus, khonsu-guardian, pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 4, + "severity": "HIGH", + "file": "ddprof-test/src/test/cpp/nativeSocketSampler_ut.cpp", + "line": 79, + "concern": "write_hook and read_hook not tested in unit test (missing isSocket branch coverage)", + "issue": "The C++ unit test covers only send_hook and recv_hook (error-return path); write_hook and read_hook have an additional isSocket() branch that gates the sampling path and have zero unit-test coverage, so a regression in the fd-type-cache decision path would be silent.", + "suggestion": "Add TEST_F cases that install stub _orig_write/_orig_read pointers and call write_hook/read_hook with ret<=0; add cases with positive return values to cover the socket classification guard, using fd numbers outside FD_TYPE_CACHE_SIZE to bypass the cache and redirect getsockopt via a mock.", + "confidence": 80, + "voice": "nemesis-moravec", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 5, + "severity": "HIGH", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket", + "line": null, + "concern": "no stop-then-restart lifecycle test for nativesocket (missing restart scenario)", + "issue": "The patch adds start/stop lifecycle management for NativeSocketSampler (PLT patch/unpatch, fd-type cache reset, rate-limiter epoch bump), but no Java integration test verifies that a second profiler session after stop() produces valid events; stale fd-type-cache entries or a non-reset epoch would silently suppress events.", + "suggestion": "Add a NativeSocketRestartTest that calls profiler start, does a TCP transfer, calls stop, then start again with a fresh JFR file, performs another transfer, and asserts NativeSocketEvent events appear in the second recording.", + "confidence": 80, + "voice": "nemesis-moravec", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 6, + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 133, + "concern": "fd-cache TOCTOU window and stale-entry misattribution on fd reuse", + "issue": "recordEvent checks _fd_cache under _fd_cache_mutex, releases the lock before calling resolveAddr (getpeername/inet_ntop), then re-acquires the lock at line 135 before insert. Between unlock and re-lock, another thread can insert the same fd, resulting in redundant getpeername syscalls or silently discarded duplicate emplace. Additionally, when an fd is closed and reused for a different connection, the old address silently shadows the cache, causing misattribution of events.", + "suggestion": "For TOCTOU: Check the cache again inside the second lock_guard scope before calling emplace; or restructure to do a single lock for cache-check + conditional resolveAddr; or accept and document the known double-resolve race as benign (emplace is idempotent for existing key). For fd-reuse staleness: Hook close() to clear the cached entry, or document as known trade-off and bound worst case to one misattributed event per fd reuse.", + "confidence": 75, + "voice": "merged: bastet-nexus, khonsu-guardian, nova-herald, pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 7, + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 87, + "concern": "fd-type-cache stale classification after fd reuse without close() hook", + "issue": "FD numbers are reused by the kernel; _fd_type_cache is only reset at start(), not on individual close(). A newly opened fd reusing a number classified as FD_TYPE_SOCKET from a prior connection skips getsockopt in write_hook/read_hook and is treated as a socket even if it is a regular file, producing incorrect NativeSocketEvent records.", + "suggestion": "Hook close() (via PLT patch) to store FD_TYPE_UNKNOWN for the closed fd, or use a generation counter incremented on start() to invalidate cached entries; or document that stale classification is accepted and bound the worst case to one erroneous event per fd reuse.", + "confidence": 70, + "voice": "pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 8, + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.h", + "line": 989, + "concern": "non-GLIBC stub check() always returns OK, masking misconfiguration without warning", + "issue": "On non-GLIBC Linux (musl) and macOS, the stub check() returns Error::OK even when args._nativesocket is true, so Profiler::check() silently succeeds, EM_NATIVESOCKET is set in _event_mask, and start() returns OK; but no hooks are installed and the feature silently does nothing without any diagnostic log or warning to the user.", + "suggestion": "On non-GLIBC the stub check() should return an informational Error (e.g. 'nativesocket is not supported on this platform') so the profiler::start() path logs the recoverable warning already present in profiler.cpp:1265; macOS silent no-op is intentional and correct.", + "confidence": 80, + "voice": "merged: bastet-nexus, nova-herald, pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 2 + }, + { + "number": 9, + "severity": "MEDIUM", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketStackTraceTest.java", + "line": 2748, + "concern": "stack-trace test does not verify cstack modes (missing AGCT path coverage)", + "issue": "hotspotSupport.cpp introduces a new BCI_NATIVE_SOCKET branch that forks between walkVM (cstack>=CSTACK_VM) and AsyncGetCallTrace (cstack Profiler::instance()->recordSample() after the profiler has stopped, which is safe only if recordSample guards against the stopped state internally.", + "suggestion": "Verify and document that recordSample returns early when _state != RUNNING, or add an explicit guard in recordEvent (e.g. check Profiler::instance()->isRunning()).", + "confidence": 60, + "voice": "nova-herald", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 14, + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/rateLimiter.h", + "line": 102, + "concern": "PID feedback loses one count per window in rate-limiter due to exchange-before-compute", + "issue": "maybeUpdateInterval does exchange(_event_count, 0) before computing the PID signal. A concurrent recordFire() between the exchange and _interval.store() silently loses that count from the feedback, causing the controller to under-estimate the actual rate.", + "suggestion": "Acknowledge this as an accepted imprecision in the rate-limiter design (one-count-per-window loss is negligible over seconds); add an explicit comment documenting it.", + "confidence": 55, + "voice": "pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 15, + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.h", + "line": 960, + "concern": "O(N) reset loop for 65536-entry atomic array on every profiler start adds startup latency", + "issue": "std::atomic _fd_type_cache[65536] inside NativeSocketSampler is re-initialised in a sequential loop on start(), adding ~65 us of unnecessary startup latency on cold cache lines, even though initial state FD_TYPE_UNKNOWN==0 is already correct from zero-initialization.", + "suggestion": "Use a generation counter (u8 _cache_gen) incremented on start(); treat a cached value as valid only when its generation byte matches the current generation. This avoids the O(N) reset loop entirely.", + "confidence": 45, + "voice": "bastet-nexus", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 16, + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 264, + "concern": "fd address cache not cleared at start on profiler restart without stop", + "issue": "start() resets _fd_type_cache but not _fd_cache; clearFdCache() is only called from stop(). If start() is ever called without a preceding stop() (e.g. via a future API path), stale address entries survive across sessions causing silent misattribution of events.", + "suggestion": "Call clearFdCache() at the beginning of start(), mirroring the pattern used for _fd_type_cache.", + "confidence": 55, + "voice": "nova-herald", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 17, + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 544, + "concern": "static thread_local PoissonSampler destruction during shutdown with active hooks", + "issue": "thread_local PoissonSampler _send_sampler/_recv_sampler are file-scope statics; on glibc they are reset via the epoch mechanism, but if the GLIBC TLS destructor sequence runs during JVM shutdown the objects may be destroyed while hooks are still installed, leading to use-after-destruction on a subsequent I/O call.", + "suggestion": "Add a guard in the hook body (check _socket_active or a separate atomic flag) before dereferencing the thread_local sampler so that calls after TLS destructors have run fall through to _orig_send/_orig_recv safely.", + "confidence": 55, + "voice": "bastet-nexus", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 18, + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/rateLimiter.h", + "line": 109, + "concern": "rate-limiter interval store uses relaxed atomics on weak architectures, delaying visibility", + "issue": "In maybeUpdateInterval, the winning CAS thread stores the updated _interval with memory_order_relaxed; interval() also loads with relaxed. On ARM the new value may be delayed in becoming visible across cores.", + "suggestion": "Use memory_order_release for the _interval.store in maybeUpdateInterval so threads reading it with memory_order_acquire see the updated value promptly. Or accept the eventual-consistency model and document it.", + "confidence": 55, + "voice": "khonsu-guardian", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 19, + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/libraryPatcher_linux.cpp", + "line": 363, + "concern": "_orig_* pointer reassignment on re-patching may use already-hooked functions", + "issue": "patch_socket_functions() assigns _orig_send/recv/write/read via RTLD_NEXT; if patch_socket_functions is called again while already-patched entries exist (via install_socket_hooks from dlopen_hook), it may overwrite _orig_* with a pointer to the hook itself if the GOT was already patched in some libraries.", + "suggestion": "Guard _orig_* assignment with a check: only assign when _socket_size == 0 (no hooks yet installed), to avoid overwriting originals with potentially already-hooked function pointers.", + "confidence": 50, + "voice": "nova-herald", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 20, + "severity": "NIT", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 569, + "concern": "IPv6 formatted address buffer size calculation is fragile and undocumented", + "issue": "char buf[INET6_ADDRSTRLEN + 10] for '[addr]:port'; INET6_ADDRSTRLEN is 46, plus 2 brackets + 1 colon + 5 digits + NUL = 55 bytes within the 56-byte buffer, leaving no margin if port formatting changes or buffer definition is moved without updating the arithmetic.", + "suggestion": "Define a named constant for the formatted address buffer size (e.g. #define FORMATTED_ADDR_SIZE (INET6_ADDRSTRLEN + 2 + 1 + 5 + 1)) rather than relying on the arithmetic to stay correct.", + "confidence": 40, + "voice": "bastet-nexus", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 21, + "severity": "NIT", + "file": "ddprof-lib/src/main/cpp/poissonSampler.h", + "line": 1143, + "concern": "float-precision underflow to 0 in weight calculation is undocumented", + "issue": "When value >> interval, 1.0f - expf(-value/interval) underflows to 0.0f in float arithmetic, so the guard defaults weight to 1.0f instead of the theoretically correct (unbounded) value; this is a conservative floor but it is not documented.", + "suggestion": "Add a comment noting the float-precision floor and that 1.0f is a conservative lower bound for large measurements, confirming it is intentional.", + "confidence": 50, + "voice": "pallas-augur", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 22, + "severity": "NIT", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketDisabledTest.java", + "line": 24, + "concern": "isPlatformSupported inconsistency with other nativesocket tests", + "issue": "NativeSocketDisabledTest.isPlatformSupported() returns Platform.isLinux() (includes musl) while all other nativesocket tests return Platform.isLinux() && !Platform.isMusl(); on musl the feature is gated by __GLIBC__ so the disabled-test still runs but the nativesocket check() path is a stub that always returns OK, making the negative assertion trivially true.", + "suggestion": "Either explicitly exclude musl from the disabled-test (align with other tests), or add a comment explaining why musl is deliberately included (feature is compiled out on musl so negative assertion is still meaningful).", + "confidence": 50, + "voice": "nemesis-moravec", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 23, + "severity": "NIT", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketMacOsNoOpTest.java", + "line": 2241, + "concern": "macOS no-op test asserts only event absence, not profiler start success", + "issue": "The non-GLIBC stub's check() always returns OK, so the profiler silently starts without error; the test only verifies events.hasItems()==false but does not assert that the profiler started without returning an error string, leaving the scenario where check() accidentally returns an error undetected.", + "suggestion": "Capture the return value of profiler.execute('start,natsock,...') and assert it does not contain 'error' or is empty before proceeding to the transfer and event assertion.", + "confidence": 60, + "voice": "nemesis-moravec", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + }, + { + "number": 24, + "severity": "NIT", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketRateLimitTest.java", + "line": 2406, + "concern": "rate-limit test upper bound not asserted", + "issue": "The test asserts eventCount < operations but does not enforce an upper ceiling on recorded events, so a broken PID controller that fires on every call (100% sampling) would pass if operations > 1; the guard is trivially satisfied with even one suppressed event.", + "suggestion": "Add a guard: assertTrue(eventCount < operations / 2) or assert eventCount < TARGET_RATE_PER_SECOND * RECORDING_SECONDS * MARGIN to verify the rate limiter actually constrains throughput to a fraction of the operation count.", + "confidence": 65, + "voice": "nemesis-moravec", + "chain_of_thought": "", + "suggestion_code": null, + "fixable": true, + "disputed": false, + "tier": 3 + } +] \ No newline at end of file diff --git a/doc/review-pr-488-simple.json b/doc/review-pr-488-simple.json new file mode 100644 index 0000000000..fbd80406fb --- /dev/null +++ b/doc/review-pr-488-simple.json @@ -0,0 +1,118 @@ +{ + "summary": "Adds a Linux glibc-only PLT hook based 'natsock' (native socket I/O) sampling engine, plus a thread-local Poisson sampler, PID-based rate limiter, JFR metadata, integration tests and a JMH overhead benchmark. Also touches LivenessTracker init order, hotspotSupport stack-walk dispatch, and import enumeration. The new feature is conceptually sound but has several robustness and correctness issues centred on fd reuse, memory ordering during hook (un)patching, and weak/redundant tests. The hotspotSupport split is pure code duplication.", + "branch": "muse/impl-20260416-133933", + "files_reviewed": 34, + "findings": [ + { + "severity": "HIGH", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 578, + "concern": "correctness", + "issue": "_fd_type_cache entries are written-once per fd value and never invalidated on close()/dup()/dup2(). When a TCP socket is closed and the same fd number is reopened as a regular file (or vice-versa), the cached classification is wrong: write/read on a former socket fd that is now a file goes through the sampling path (recordEvent + getpeername — which fails — yielding events with empty remoteAddress); a former regular fd reused as a socket bypasses sampling entirely.", + "evidence": "isSocket() caches FD_TYPE_SOCKET / FD_TYPE_NON_SOCKET at first observation and never resets it except in start(); there is no close()/dup() hook or invalidation path.", + "suggestion": "Either probe getsockopt(SO_TYPE) on every call (acceptable cost) or hook close()/dup() to invalidate the slot, or limit the cache to a small bounded LRU keyed by (fd, generation_counter).", + "confidence": "high" + }, + { + "severity": "HIGH", + "file": "ddprof-lib/src/main/cpp/libraryPatcher_linux.cpp", + "line": 367, + "concern": "concurrency", + "issue": "Memory-ordering hole between _orig_send/_recv/_write/_read assignment (plain pointer store, non-atomic) and the __atomic_store_n RELAXED publish of the hook into the GOT slot. A second core that loads the hook pointer from the GOT before its own cache view of _orig_send is updated will execute send_hook with the prior (or, on first start, NULL) function pointer.", + "evidence": "NativeSocketSampler::_orig_send = pre_send; // plain store\n__atomic_store_n(send_location, (void*)NativeSocketSampler::send_hook, __ATOMIC_RELAXED);", + "suggestion": "Make _orig_* std::atomic<...> with release stores in patch_socket_functions and acquire loads in *_hook, and switch the GOT __atomic_store_n to __ATOMIC_RELEASE.", + "confidence": "high" + }, + { + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp", + "line": 1109, + "concern": "necessity", + "issue": "The walkJavaStack dispatch was split into two separate else-if branches (BCI_NATIVE_MALLOC || BCI_NATIVE_SOCKET) and (BCI_CPU || BCI_WALL) with byte-for-byte identical bodies. This is pure duplication — the next maintainer will fix one branch and forget the other.", + "evidence": "Lines 1109-1132 and 1133-1159 are the same code modulo the comment on line 1137.", + "suggestion": "Restore the single original branch: 'else if (event_type == BCI_CPU || event_type == BCI_WALL || event_type == BCI_NATIVE_MALLOC || event_type == BCI_NATIVE_SOCKET)'.", + "confidence": "high" + }, + { + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp", + "line": 70, + "concern": "completeness", + "issue": "BCI_NATIVE_SOCKET was added but eventTypeFromBCI() was not updated — it falls into the 'default' branch returning EXECUTION_SAMPLE. walkVM is therefore invoked with the wrong EventType for socket events when cstack=vm.", + "evidence": "Switch in hotspotSupport.cpp:54 has cases for BCI_NATIVE_MALLOC but not BCI_NATIVE_SOCKET.", + "suggestion": "Add a case for BCI_NATIVE_SOCKET returning the appropriate EventType and document the choice.", + "confidence": "high" + }, + { + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 631, + "concern": "correctness", + "issue": "recordEvent holds std::mutex _fd_cache_mutex while a SIGPROF handler can fire on the same thread. std::mutex is not async-signal-safe. If the SIGPROF path enters any code that tries to acquire the same mutex (directly or via Profiler::recordSample → call-trace-storage flush), the thread deadlocks.", + "evidence": "recordEvent acquires _fd_cache_mutex twice; SIGPROF can fire while the lock is held; std::mutex is not reentrant and not signal-safe.", + "suggestion": "Either replace _fd_cache_mutex with try_lock + best-effort skip, block SIGPROF around the critical section, or move address resolution outside the hook path (resolve eagerly on connect/accept).", + "confidence": "medium" + }, + { + "severity": "MEDIUM", + "file": "ddprof-lib/src/main/cpp/nativeSocketSampler.cpp", + "line": 565, + "concern": "correctness", + "issue": "inet_ntop() return value is ignored in resolveAddr. On NULL return (unsupported AF, insufficient buffer), host[] may be uninitialised and snprintf reads garbage.", + "evidence": "inet_ntop(AF_INET, &s->sin_addr, host, sizeof(host)); // return value ignored", + "suggestion": "Check inet_ntop return value and return empty string on failure.", + "confidence": "medium" + }, + { + "severity": "MEDIUM", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketBytesAccuracyTest.java", + "line": 36, + "concern": "consistency", + "issue": "Test would pass against a stub emitting a single event with weight=1 and any positive duration. The sole assertion is 'sum(weight*duration) > 0 AND <= 100x wall time'. A 100x ceiling provides essentially no signal.", + "evidence": "assertTrue(scaledDurationNs <= wallNs * 100.0, ...) — no lower bound, no order-of-magnitude check.", + "suggestion": "Add a lower bound (e.g. > wallNs / 100) and tighten the upper bound, or run with a known-deterministic interval and assert the scaled estimate is within ~3 sigma of the analytical expectation.", + "confidence": "high" + }, + { + "severity": "MEDIUM", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketRateLimitTest.java", + "line": 46, + "concern": "completeness", + "issue": "Test would pass if the sampler were stubbed to emit a single event with weight=1.5. Both assertions ('eventCount < operations' and 'foundWeightAboveOne') are trivially satisfied by any non-degenerate sampler. The test never validates that the rate is anywhere near the documented ~83/sec target.", + "evidence": "assertions only check eventCount < 20000 and at-least-one weight > 1.0", + "suggestion": "Assert eventCount is within an envelope of expected_rate × wall_time; otherwise rename to 'sampler emits something' since rate-limiting correctness is not validated.", + "confidence": "high" + }, + { + "severity": "MEDIUM", + "file": "ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketEnabledTest.java", + "line": 18, + "concern": "necessity", + "issue": "Redundant test. NativeSocketEventFieldsTest performs the same workload with strictly stronger assertions (verifies operation, remoteAddress, bytesTransferred, weight, duration, eventThread, stackTrace, foundSend && foundRecv). NativeSocketEnabledTest only checks events.hasItems().", + "evidence": "Both tests share parent class and same workload; EventFieldsTest assertions imply EnabledTest's assertion.", + "suggestion": "Remove NativeSocketEnabledTest.", + "confidence": "high" + }, + { + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/livenessTracker.cpp", + "line": 214, + "concern": "completeness", + "issue": "Only _record_heap_usage was moved above the _initialized fast-path return. Other per-call toggles (subsample_ratio, table_cap, JNI references) remain pinned to the first-ever args. Behaviour is now asymmetric across args fields with no documentation explaining why.", + "evidence": "Only _record_heap_usage moved above the _initialized guard; all other args fields remain inside 'if (!_initialized)'.", + "suggestion": "Document why only _record_heap_usage is re-read on repeat calls, or move all per-call toggles together.", + "confidence": "medium" + }, + { + "severity": "LOW", + "file": "ddprof-lib/src/main/cpp/libraryPatcher_linux.cpp", + "line": 459, + "concern": "robustness", + "issue": "unpatch_socket_functions leaves _orig_* set so in-flight hooks can dereference them — documented. But on the next patch_socket_functions call, those slots are unconditionally overwritten with plain stores while hooks from a prior session may still be in flight, creating a window where a hook reads a partially-updated pointer.", + "evidence": "Unconditional reassignment in patch_socket_functions with no release fence.", + "suggestion": "Use atomic release stores for _orig_* assignments and document the start/stop ordering invariant.", + "confidence": "medium" + } + ], + "clean": false +} diff --git a/doc/sframe-feasibility.md b/doc/sframe-feasibility.md new file mode 100644 index 0000000000..a360ae1680 --- /dev/null +++ b/doc/sframe-feasibility.md @@ -0,0 +1,163 @@ +# SFrame Stack Walking Mode: Feasibility Analysis + +## What is SFrame? + +SFrame ("Simple Frame") is a stack unwinding metadata format designed as a simpler, faster-to-decode +alternative to DWARF `.eh_frame`. Originated at Oracle, inspired by the Linux kernel's ORC unwinder. +Instead of executing a DWARF bytecode VM to recover frame info, SFrame directly encodes CFA/FP/RA +offsets in flat, binary-searchable tables. No interpreter needed. + +The `.sframe` ELF section (`SHT_GNU_SFRAME`, `SHF_ALLOC`) has three parts: +- **Header** (~28 bytes): magic, version, ABI/arch, fixed RA offset, FDE/FRE counts +- **FDE sub-section**: fixed-size entries sorted by start PC (binary searchable) +- **FRE sub-section**: variable-length entries encoding CFA offset, FP save offset, RA save offset + +Spec: https://sourceware.org/binutils/docs/sframe-spec.html + +## Current Toolchain & Kernel Status + +| Component | SFrame Status | +|-------------------|---------------| +| GNU binutils/gas | `--gsframe` since 2.40. Linker merges `.sframe`. `readelf --sframe` works. | +| GCC | Via assembler: `gcc -Wa,--gsframe`. No direct GCC flag. | +| LLVM/Clang | RFC posted Jun 2025, prototype exists. **Not upstreamed yet** (controversial in LLVM project). | +| glibc | Merged in 2.42 (Aug 2025). `ld.so` registers `.sframe` with kernel via `prctl()`. | +| Linux kernel | Merged in 6.19 (Feb 2026). `perf_events` can do SFrame-based user-space unwinding. | +| Fedora | `.sframe` in all packages since Fedora 43. | +| Architectures | x86_64 (V1+), aarch64 (V1+), s390x (V2 errata 1). | +| macOS | **Not applicable** -- ELF-only format. | + +**Critical gap: No LLVM support means clang-built binaries (musl distros, Alpine, Android, ChromeOS) +lack `.sframe` sections.** + +## Current Profiler Architecture (relevant parts) + +The profiler has 6 CStack modes (`arguments.h:63-70`): + +``` +CSTACK_DEFAULT → CSTACK_FP → CSTACK_DWARF → CSTACK_LBR → CSTACK_VM +``` + +For native frame collection (`profiler.cpp:271-278`): +- `CSTACK_DWARF` → `StackWalker::walkDwarf()` +- Everything else → `StackWalker::walkFP()` + +DWARF loading path: +1. `ElfParser::parseDwarfInfo()` finds `PT_GNU_EH_FRAME` program header +2. `DwarfParser` parses `.eh_frame_hdr` → sorted `FrameDesc*` table +3. `CodeCache::setDwarfTable()` stores the table per-library +4. At walk time, `findFrameDesc(pc)` does binary search → yields CFA reg, CFA offset, FP offset, PC offset + +The `FrameDesc` struct (16 bytes) stores exactly what SFrame encodes: + +```cpp +struct FrameDesc { + u32 loc; // PC offset relative to module base + u32 cfa; // CFA register + offset packed + int fp_off; // FP save location relative to CFA + int pc_off; // RA save location relative to CFA +}; +``` + +## Feasibility Assessment + +### What SFrame gives us that we don't already have + +**Honestly: not much for in-process unwinding.** The profiler already parses `.eh_frame` into +the same logical structure (`FrameDesc`) that SFrame encodes natively. The DWARF bytecode VM +interpretation happens once at load time, not on the hot path. During stack walking, both approaches +reduce to: binary search → read offsets → step. + +Where SFrame could help: + +1. **Simpler parser.** The current `DwarfParser` (`dwarf.cpp`) is ~400 lines of LEB128 decoding, + CIE/FDE parsing, and DW_CFA_* opcode interpretation. An SFrame parser would be ~100 lines of + direct struct reads. Less code = fewer bugs, easier maintenance. + +2. **Faster library load time.** Parsing `.eh_frame` is O(n) in the number of DWARF opcodes. + SFrame FDEs are already in their final form -- just validate the header and memcpy the tables. + For libraries with large `.eh_frame` sections (libc, libpthread, libstdc++), this saves + measurable time during profiler startup. + +3. **Future kernel perf_events integration.** On Linux 6.19+, `perf_events` can produce user-space + stacks via SFrame without any in-process unwinding. This could replace `CSTACK_DWARF` entirely + for the `cpu` event type on modern kernels, similar to how `CSTACK_DEFAULT` already uses + perf_event_open stacks when available. + +### Implementation effort + +**Low.** The extension points are clean: + +1. Add `CSTACK_SFRAME` to enum in `arguments.h` +2. Write `SFrameParser` (~100 LOC) that produces the same `FrameDesc*` table +3. In `symbols_linux.cpp::parseDwarfInfo()`, check for `PT_GNU_SFRAME` program header first, + fall back to `PT_GNU_EH_FRAME` +4. No changes to `StackWalker::walkDwarf()` -- it already works on `FrameDesc` tables + +Actually, option (3) means we might not even need a new CStack mode. SFrame could be a +**transparent optimization** of DWARF loading: if `.sframe` is present, parse that (faster); +otherwise fall back to `.eh_frame`. The walk-time code stays identical. + +If we want `CSTACK_SFRAME` as an explicit mode, the additional work is: +5. Add mode selection logic in `profiler.cpp:1115-1121` +6. Add `"sframe"` string in argument parsing and `cstack()` method +7. Add `SFRAME_SUPPORTED` compile-time flag (Linux only, not macOS) + +### What we'd need to handle + +- **Mixed environments.** Some libraries will have `.sframe`, others won't. Need per-library + fallback to `.eh_frame`. This is straightforward -- check in `parseDwarfInfo()`. +- **Version detection.** SFrame V1 is obsolete, V2 is current, V3 is emerging. Header has + a version field; we only need to support V2+. +- **Architecture differences.** On x86_64, RA is at fixed CFA offset (-8) so the FRE doesn't + encode it. On aarch64, RA offset is per-FRE. This maps directly to how `FrameDesc::pc_off` + already works. +- **No macOS support.** SFrame is ELF-only. macOS path stays on `__eh_frame` parsing (recently + added in PR #430). This is fine -- SFrame would be a Linux-only optimization. + +### Risks and concerns + +1. **LLVM gap is the dealbreaker for mandatory use.** Until clang emits `.sframe`, we cannot + make it the default. It must always fall back to `.eh_frame`. + +2. **Marginal benefit for in-process unwinding.** The DWARF parse cost is paid once at startup. + Walk-time performance is identical (both are binary search + offset reads). The benefit is + modest: slightly faster startup, simpler parser code. + +3. **Kernel-side SFrame unwinding helps less than it appears.** The kernel's SFrame unwinder + (Linux 6.19+) only produces **native** frames. Java frames still require + `AsyncGetCallTrace` / JVMTI from a signal handler. So signal-handler unwinding cannot be + eliminated -- you'd replace only the native portion while keeping all the signal machinery + for Java frames. Coordinating kernel-provided native stacks with signal-handler-provided + Java stacks adds complexity that likely exceeds the current unified in-process approach. + +4. **Format is still evolving.** V2 → V3 changed terminology, added signal frame marking, + flexible FDE types, 64-bit offsets. Supporting a moving target has a maintenance cost. + +## Recommendation + +**Feasible but not urgent.** Two practical approaches, in order of effort/value: + +### Approach A: Transparent SFrame loading (low effort, do now if desired) + +Add SFrame as an alternative `.eh_frame` loading path in `symbols_linux.cpp`. No new CStack +mode needed. When a library has both `.sframe` and `.eh_frame_hdr`, prefer `.sframe` for +faster load. Fall back seamlessly. ~200 LOC total (parser + integration). + +**Value:** Cleaner parser, faster profiler startup on Fedora 43+ / newer distros. + +### ~~Approach B: Kernel perf_events SFrame unwinding~~ (not viable) + +The kernel's SFrame unwinder (Linux 6.19+) only produces native frames. Java frames still +require `AsyncGetCallTrace` from a signal handler, so we cannot eliminate in-process signal +handling. Coordinating kernel-provided native stacks with signal-handler-provided Java stacks +would add complexity without removing the fundamental signal-handler machinery. **Not worth +pursuing.** + +### What I would not do + +- Add `CSTACK_SFRAME` as a user-visible mode distinct from `CSTACK_DWARF`. The walk-time + behavior is identical; exposing it as a separate mode adds configuration surface for no + user-facing benefit. +- Target macOS. SFrame is inherently Linux/ELF. +- Block on LLVM support. The transparent fallback approach works fine with mixed toolchains. diff --git a/doc/specs/sframe-transparent-loading.md b/doc/specs/sframe-transparent-loading.md new file mode 100644 index 0000000000..6b945776ee --- /dev/null +++ b/doc/specs/sframe-transparent-loading.md @@ -0,0 +1,411 @@ +# Specification: Transparent SFrame Loading as Alternative to .eh_frame + +## Objective + +Add SFrame V2 parsing as a transparent optimization in the native unwind table loading path. When the profiler loads a shared library, it checks for a `.sframe` section first. If present and valid, it parses SFrame data into the existing `FrameDesc*` table. If absent or invalid, it falls back to the existing `.eh_frame` DWARF path. No new user-visible CStack mode. No changes to walk-time code. + +## Background + +The profiler loads native unwind tables from `.eh_frame` sections via DWARF CFI opcode interpretation at library load time. SFrame is a simpler ELF section format (`.sframe`) that directly encodes CFA/FP/RA offsets in flat, binary-searchable tables without requiring a bytecode interpreter. On modern Linux distros (Fedora 43+, glibc 2.42+, binutils 2.40+), libraries ship `.sframe` sections alongside `.eh_frame`. + +SFrame is Linux/ELF only. macOS is unaffected. + +Reference: https://sourceware.org/binutils/docs/sframe-spec.html + +## Scope + +### In scope +- SFrame V2 parser producing `FrameDesc*` tables identical to what the DWARF parser produces +- x86_64 and aarch64 architecture support +- Per-library fallback: use `.sframe` when present, `.eh_frame` otherwise +- Aarch64 GCC vs Clang default frame detection (matching existing DwarfParser behavior) +- Unit tests following existing `dwarf_ut.cpp` patterns +- Bounds-checking for all reads from the SFrame section + +### Out of scope +- No new `CSTACK_SFRAME` user-visible mode +- No SFrame V3 support (add later when V3 stabilizes) +- No PCMASK FDE type support (rare, used for PLT stubs; fall back to DWARF for those libraries) +- No macOS changes +- No changes to `StackWalker::walkDwarf()` or `CodeCache::findFrameDesc()` + +## Architecture + +### Current loading path (unchanged for .eh_frame) + +``` +ElfParser::parseDwarfInfo() [symbols_linux.cpp:593] + -> findProgramHeader(PT_GNU_EH_FRAME) + -> DwarfParser(name, image_base, eh_frame_hdr_ptr) [dwarf.cpp] + - Interprets DWARF CFI opcodes (DW_CFA_*) + - Builds sorted FrameDesc* table + -> CodeCache::setDwarfTable(table, count, defaultFrame) [codeCache.cpp:397] + +At walk time: + CodeCache::findFrameDesc(pc) [codeCache.cpp:403] + - Binary search: target_loc = (char*)pc - _text_base + - Returns FrameDesc with cfa, fp_off, pc_off + StackWalker::walkDwarf() [stackWalker.cpp:81] + - Uses FrameDesc fields to step through frames +``` + +### New loading path (SFrame, tried first) + +``` +ElfParser::parseDwarfInfo() [symbols_linux.cpp:593] + -> findProgramHeader(PT_GNU_SFRAME) // NEW: try SFrame first + -> SFrameParser(name, section_base, size, section_offset) [sframe.cpp] + - Direct struct reads, no opcode interpretation + - Builds sorted FrameDesc* table + -> CodeCache::setDwarfTable(table, count, defaultFrame) + -> return on success; fall through to DWARF on failure + + -> findProgramHeader(PT_GNU_EH_FRAME) // existing fallback + -> DwarfParser(...) + ... +``` + +Walk-time code is identical -- both paths produce the same `FrameDesc` table format. + +## Files to Create + +### `ddprof-lib/src/main/cpp/sframe.h` + +Header with SFrame format definitions and parser class declaration. + +**Constants** (define locally; older `` may lack them): + +```cpp +#ifndef PT_GNU_SFRAME +#define PT_GNU_SFRAME 0x6474e554 +#endif + +// SFrame header constants +const uint16_t SFRAME_MAGIC = 0xDEE2; +const uint8_t SFRAME_VERSION_2 = 2; +const uint8_t SFRAME_F_FDE_SORTED = 0x01; + +// ABI/architecture identifiers +const uint8_t SFRAME_ABI_AARCH64_ENDIAN_LITTLE = 2; +const uint8_t SFRAME_ABI_AMD64_ENDIAN_LITTLE = 3; + +// FDE info byte accessors +// bit 0: FDE type (0=PCINC, 1=PCMASK) +// bits 1-2: FRE type (start address size: 0=1B, 1=2B, 2=4B) +#define SFRAME_FUNC_FDE_TYPE(info) ((info) & 0x1) +#define SFRAME_FUNC_FRE_TYPE(info) (((info) >> 1) & 0x3) + +// FRE info byte accessors +// bit 0: CFA base register (0=SP, 1=FP) +// bits 1-2: offset encoding size (0=1B, 1=2B, 2=4B) +// bit 3: RA tracked +// bit 4: FP tracked +// bit 7: mangled RA (PAC on aarch64) +#define SFRAME_FRE_BASE_REG(info) ((info) & 0x1) +#define SFRAME_FRE_OFFSET_SIZE(info) (((info) >> 1) & 0x3) +#define SFRAME_FRE_RA_TRACKED(info) (((info) >> 3) & 0x1) +#define SFRAME_FRE_FP_TRACKED(info) (((info) >> 4) & 0x1) + +// Offset size codes +const int SFRAME_FRE_OFFSET_1B = 0; +const int SFRAME_FRE_OFFSET_2B = 1; +const int SFRAME_FRE_OFFSET_4B = 2; + +// FRE start address size codes (from FDE info) +const int SFRAME_FRE_TYPE_ADDR1 = 0; +const int SFRAME_FRE_TYPE_ADDR2 = 1; +const int SFRAME_FRE_TYPE_ADDR4 = 2; +``` + +**On-disk structs** (packed, little-endian): + +```cpp +struct __attribute__((packed)) SFrameHeader { // 28 bytes + uint16_t magic; + uint8_t version; + uint8_t flags; + uint8_t abi_arch; + int8_t cfa_fixed_fp_offset; + int8_t cfa_fixed_ra_offset; // -8 on x86_64, 0 on aarch64 (per-FRE) + uint8_t auxhdr_len; + uint32_t num_fdes; + uint32_t num_fres; + uint32_t fre_len; // total bytes in FRE sub-section + uint32_t fdeoff; // offset to FDE array from end of header+auxhdr + uint32_t freoff; // offset to FRE section from end of header+auxhdr +}; + +struct __attribute__((packed)) SFrameFDE { // 20 bytes + int32_t start_addr; // signed, relative to .sframe section start (V2) + uint32_t func_size; + uint32_t fre_off; // byte offset into FRE sub-section + uint32_t fre_num; // number of FREs for this function + uint8_t info; // FDE type (bit 0) | FRE type (bits 1-2) + uint8_t rep_size; // for repeated block encoding + uint16_t padding; +}; +``` + +**SFrameParser class**: + +```cpp +class SFrameParser { + private: + const char* _name; // library name for diagnostics + const char* _section_base; // runtime address of .sframe section + size_t _section_size; + u32 _section_offset; // section_runtime_addr - library_base (for loc computation) + + int _capacity; + int _count; + FrameDesc* _table; + int _linked_frame_size; // for aarch64 GCC vs Clang detection; -1 = undetected + + bool parseFDE(const SFrameHeader* hdr, const SFrameFDE* fde, + const char* fre_section, const char* fre_end); + FrameDesc* addRecord(u32 loc, u32 cfa, int fp_off, int pc_off); + + public: + SFrameParser(const char* name, const char* section_base, + size_t section_size, u32 section_offset); + ~SFrameParser(); + + bool parse(); // returns false on invalid/unsupported section (triggers DWARF fallback) + + // Ownership of table() transfers to caller on success. + // Caller frees with free(). After calling table(), do not use the parser further. + FrameDesc* table() const { return _table; } + int count() const { return _count; } + + const FrameDesc& detectedDefaultFrame() const; +}; +``` + +Include `dwarf.h` for `FrameDesc`, `DW_REG_SP`, `DW_REG_FP`, `DW_SAME_FP`, `DW_LINK_REGISTER`, `LINKED_FRAME_SIZE`, `LINKED_FRAME_CLANG_SIZE`, `DWARF_SUPPORTED`. + +### `ddprof-lib/src/main/cpp/sframe.cpp` + +**Constructor:** +- Store name, section_base, section_size, section_offset +- Allocate `_table = (FrameDesc*)malloc(128 * sizeof(FrameDesc))`, `_capacity = 128`, `_count = 0` +- Set `_linked_frame_size = -1` + +**Destructor:** +- Free `_table` if non-null (handles the case where `parse()` fails but caller doesn't call `table()`) + +**`parse()` method:** +1. If `_section_size < sizeof(SFrameHeader)` return false +2. Cast `_section_base` to `const SFrameHeader*` +3. Validate `magic == SFRAME_MAGIC` +4. Validate `version == SFRAME_VERSION_2` +5. Validate `abi_arch` matches current architecture: + - `__x86_64__` -> `SFRAME_ABI_AMD64_ENDIAN_LITTLE` + - `__aarch64__` -> `SFRAME_ABI_AARCH64_ENDIAN_LITTLE` + - Other -> return false +6. Compute: `data_start = _section_base + sizeof(SFrameHeader) + hdr->auxhdr_len` +7. Compute: `fde_array = (const SFrameFDE*)(data_start + hdr->fdeoff)` +8. Compute: `fre_section = data_start + hdr->freoff` +9. Compute: `fre_end = fre_section + hdr->fre_len` +10. Bounds check: `(const char*)fde_array + hdr->num_fdes * sizeof(SFrameFDE) <= _section_base + _section_size` +11. Bounds check: `fre_end <= _section_base + _section_size` +12. For each FDE i in 0..num_fdes-1: + - Skip if `SFRAME_FUNC_FDE_TYPE(fde->info) != 0` (PCMASK not supported) + - Skip if `fde->fre_num == 0` + - Call `parseFDE(hdr, fde, fre_section, fre_end)`. On false, continue (skip corrupt FDE) +13. Sort: `qsort(_table, _count, sizeof(FrameDesc), FrameDesc::comparator)` +14. Return `_count > 0` + +**`parseFDE()` method:** +1. Determine FRE start address size from `SFRAME_FUNC_FRE_TYPE(fde->info)`: + - 0 -> 1 byte (uint8_t) + - 1 -> 2 bytes (uint16_t) + - 2 -> 4 bytes (uint32_t) + - other -> return false +2. Compute `fre_ptr = fre_section + fde->fre_off` +3. For each FRE j in 0..fde->fre_num-1: + a. Bounds check: `fre_ptr < fre_end` + b. Read FRE start address offset (unsigned, 1/2/4 bytes per step 1) + c. Read FRE info byte (1 byte) + d. Determine offset encoding size from `SFRAME_FRE_OFFSET_SIZE(fre_info)`: + - 0 -> 1 byte (int8_t) + - 1 -> 2 bytes (int16_t) + - 2 -> 4 bytes (int32_t) + - other -> return false + e. Bounds check remaining read size + f. Read CFA offset (signed, size from step d) + g. Read FP offset if `SFRAME_FRE_FP_TRACKED(fre_info)` (signed, same size) + h. Read RA offset if `hdr->cfa_fixed_ra_offset == 0 && SFRAME_FRE_RA_TRACKED(fre_info)` (signed, same size) + i. Translate to FrameDesc (see translation rules below) + j. Call `addRecord(loc, cfa, fp_off, pc_off)` +4. Return true + +**SFrame to FrameDesc translation rules:** + +``` +// Address: section-relative -> image-base-relative +loc = _section_offset + (u32)((int32_t)fde->start_addr + fre_start_addr_offset) + +// CFA register mapping +cfa_reg = SFRAME_FRE_BASE_REG(fre_info) ? DW_REG_FP : DW_REG_SP +cfa = ((u32)cfa_offset << 8) | cfa_reg + +// Frame pointer +fp_off = SFRAME_FRE_FP_TRACKED(fre_info) ? fre_fp_offset : DW_SAME_FP + +// Return address / program counter +if (hdr->cfa_fixed_ra_offset != 0) { + // x86_64: RA is always at CFA + fixed offset (typically -8) + pc_off = (int)hdr->cfa_fixed_ra_offset; +} else if (SFRAME_FRE_RA_TRACKED(fre_info)) { + // aarch64: RA location varies per frame + pc_off = fre_ra_offset; +} else { + // aarch64 leaf: RA in link register, not on stack + pc_off = DW_LINK_REGISTER; +} +``` + +**Default frame detection** (for aarch64 GCC vs Clang): +```cpp +const FrameDesc& SFrameParser::detectedDefaultFrame() const { + if (_linked_frame_size == LINKED_FRAME_CLANG_SIZE && + LINKED_FRAME_CLANG_SIZE != LINKED_FRAME_SIZE) { + return FrameDesc::default_clang_frame; + } + return FrameDesc::default_frame; +} +``` + +In `parseFDE()`, when translating a FRE: if `_linked_frame_size < 0` and `cfa_reg == DW_REG_FP` and `cfa_offset > 0`, set `_linked_frame_size = cfa_offset`. This matches `DwarfParser::addRecord()` logic at `dwarf.cpp:512-514`. + +**`addRecord()` method:** +```cpp +FrameDesc* SFrameParser::addRecord(u32 loc, u32 cfa, int fp_off, int pc_off) { + if (_count >= _capacity) { + FrameDesc* resized = (FrameDesc*)realloc(_table, _capacity * 2 * sizeof(FrameDesc)); + if (!resized) return NULL; + _capacity *= 2; + _table = resized; + } + FrameDesc* fd = &_table[_count++]; + fd->loc = loc; + fd->cfa = cfa; + fd->fp_off = fp_off; + fd->pc_off = pc_off; + return fd; +} +``` + +## Files to Modify + +### `ddprof-lib/src/main/cpp/symbols_linux.cpp` + +Add `#include "sframe.h"` near existing `#include "dwarf.h"`. + +Replace `parseDwarfInfo()` (currently at line 593): + +```cpp +void ElfParser::parseDwarfInfo() { + if (!DWARF_SUPPORTED) return; + + // Try SFrame first (simpler format, faster parsing, no opcode interpretation) + ElfProgramHeader* sframe_phdr = findProgramHeader(PT_GNU_SFRAME); + if (sframe_phdr != NULL && sframe_phdr->p_vaddr != 0) { + const char* section_base = at(sframe_phdr); + u32 section_offset = (u32)(section_base - _base); + SFrameParser sframe(_cc->name(), section_base, + (size_t)sframe_phdr->p_memsz, section_offset); + if (sframe.parse()) { + _cc->setDwarfTable(sframe.table(), sframe.count(), + sframe.detectedDefaultFrame()); + return; + } + // SFrame parse failed; fall through to DWARF + } + + // Existing DWARF path (unchanged) + ElfProgramHeader* eh_frame_hdr = findProgramHeader(PT_GNU_EH_FRAME); + if (eh_frame_hdr != NULL) { + if (eh_frame_hdr->p_vaddr != 0) { + DwarfParser dwarf(_cc->name(), _base, at(eh_frame_hdr)); + _cc->setDwarfTable(dwarf.table(), dwarf.count(), + dwarf.detectedDefaultFrame()); + } else if (strcmp(_cc->name(), "[vdso]") == 0) { + FrameDesc* table = (FrameDesc*)malloc(sizeof(FrameDesc)); + *table = FrameDesc::empty_frame; + _cc->setDwarfTable(table, 1); + } + } +} +``` + +**Address translation explained:** +- `_base` is the library's runtime load address (same value stored as `_text_base` in CodeCache) +- `at(sframe_phdr)` returns the runtime address of the `.sframe` section +- `section_offset = at(sframe_phdr) - _base` = section's offset within the loaded image +- SFrame FDE `start_addr` is relative to section start +- So `loc = section_offset + fde->start_addr + fre_offset` = offset from library base +- At walk time: `findFrameDesc(pc)` computes `target_loc = pc - _text_base` = offset from library base +- These match. + +## Test File + +### `ddprof-lib/src/test/cpp/sframe_ut.cpp` + +Guard with `#ifdef __linux__` (SFrame is Linux/ELF only). + +Use gtest framework matching `dwarf_ut.cpp` patterns. Include `gtest_crash_handler.h`. + +**Test helpers** — functions to construct in-memory SFrame binary data: +- `buildHeader(buf, abi_arch, cfa_fixed_ra_offset, num_fdes, num_fres, fre_len, fdeoff, freoff)` -- appends a valid SFrameHeader +- `buildFDE(buf, start_addr, func_size, fre_off, fre_num, fre_type)` -- appends an SFrameFDE +- `buildFRE_1B(buf, start_offset, fre_info, cfa_off, [fp_off], [ra_off])` -- appends a 1-byte-offset FRE +- Similar helpers for 2B and 4B FREs + +**Test cases:** + +| Test | What it verifies | +|------|-----------------| +| `InvalidMagic` | `parse()` returns false for wrong magic | +| `UnsupportedVersion` | `parse()` returns false for version != 2 | +| `WrongArch` | `parse()` returns false for mismatched abi_arch | +| `TruncatedSection` | `parse()` returns false when section_size < sizeof(SFrameHeader) | +| `EmptyFDEArray` | `parse()` returns false when num_fdes == 0 | +| `SingleFDE_SingleFRE_SPBased` | Single function, SP-based CFA. Verify loc, cfa reg=SP, cfa offset, fp_off=DW_SAME_FP, pc_off from fixed RA | +| `SingleFDE_SingleFRE_FPBased` | FP-based CFA with FP tracked. Verify cfa reg=FP, fp_off set | +| `FixedRAOffset` | x86_64 style: `cfa_fixed_ra_offset=-8`. Verify pc_off=-8 regardless of FRE flags | +| `PerFRE_RA` | aarch64 style: `cfa_fixed_ra_offset=0`, RA tracked. Verify pc_off from FRE | +| `PerFRE_RA_Untracked` | aarch64 leaf: RA not tracked. Verify pc_off=DW_LINK_REGISTER | +| `MultipleFDEs` | 3 FDEs with multiple FREs each. Verify all entries present and sorted by loc | +| `OffsetSize_2B` | FRE with 2-byte offset encoding. Verify correct value decoding | +| `OffsetSize_4B` | FRE with 4-byte offset encoding | +| `AddressTranslation` | Verify `loc = section_offset + fde.start_addr + fre.start_offset` with non-zero section_offset | +| `EmptyFDE_Skipped` | FDE with fre_num=0 produces no records | +| `PCMASK_Skipped` | FDE with PCMASK type is skipped | +| `BoundsCheck_FREOverrun` | FRE data extends past fre_end. parseFDE returns false for that FDE, remaining FDEs still parsed | +| `ParseFailure_FreesTable` | On failure, verify parse() returns false and no memory is leaked (destructor frees) | + +## Build Integration + +No build file changes needed. The build system (`NativeBuildPlugin` / `GtestTaskBuilder`) auto-discovers source files in `ddprof-lib/src/main/cpp/` and test files in `ddprof-lib/src/test/cpp/`. New `.cpp` and `.h` files are picked up automatically. + +Verify: `./gradlew ddprof-lib:compileRelease` compiles the new files; `./gradlew ddprof-test:testDebug` runs the new tests. + +## Verification Plan + +1. **Unit tests**: Run `./gradlew ddprof-test:testDebug` -- new SFrame tests pass, existing DWARF tests unaffected +2. **Build**: `./gradlew ddprof-lib:compileRelease` succeeds on Linux (both x86_64 and aarch64 CI) +3. **macOS**: Build succeeds (SFrame code compiles but `PT_GNU_SFRAME` is never found since macOS uses Mach-O, not ELF) +4. **Integration on modern Linux**: On a system with `.sframe` support (Fedora 43+), profile a Java app with `cstack=dwarf`. Verify native stacks are collected. Optionally add a `Log::info` in the SFrame path during development to confirm it's taken. +5. **Fallback on older Linux**: On a system without `.sframe`, verify the DWARF path is taken as before (no behavior change) +6. **Correctness comparison**: On a system with `.sframe`, compare profiler output between SFrame path and DWARF-only path (disable SFrame temporarily). Stack traces should be identical. + +## Constraints + +- Linux only. macOS path (`symbols_macos.cpp`) is not modified. +- No external dependencies. Parser is self-contained. +- No heap allocation at walk time. All parsing happens at library load time. +- Allocation during parsing (malloc/realloc for table) is acceptable. +- Must handle mixed environments: some libraries have `.sframe`, others don't. +- Must support both x86_64 and aarch64. +- Older `` headers may lack `PT_GNU_SFRAME` -- define locally with `#ifndef` guard.