Reduce the time taken to validate the commit chain #8

New Issue

2025-05-28T05:08:29Z

phil commented

2025-05-28 05:08:29 +00:00

It is taking hours for CI to validate the whole commit chain of a repo. We want CI to be quicker!
For example, in accounting we have:

phil@phil-Inspiron-5468:~/git/JankariTechUG/accounting$ git rev-list main
9a4a08c6beece26b857fc7d0014a2d330a565c5a
d45f390324ce20837948771384b89aae41cf9554
42278a64ea1ef205d0484fc93d56d6c7eb81d4b3
581e44a232b6ed6474647a64ca1ce729397514f6
866f59a61c105cd97663e796e584225ceb58ca5e
14c6c5ef0290fa6f421de46d93e45bbcb71832c6
888bd3385b0224feef1d8a52768d48ab0676ffe0
c352097f9952e7c3369f1034695157e073245929
b5e41c60de330b471c655ca30ea01d5a27b667d7
6f1e602d8bcb103704221c34468911635d2fcf17
fbe6c7d41146544741a06fa65317fcf1d9165d97
2ac7bbbbf6e1c5a1bbcbb5bba9835ace697d5d78
9715cc30c84b013444b020843f6d174f66f9d2e8
5eb4697fecd15e79263c16f942bc196d5d83c3b9
c5f2b61bae70dc10188f3c642b633c4677225f91
8e94bf5689a3bf99e8e9440484041c724b171113
3e2f36b583eb97a8fdc676a3378ae3e6e6175217
05272be96dd5fc4657e1710a1ad2271bca1bf509
ccb85560b13c4e720beeda9907d2face4f5b1195
fbb6dca2c3aef5a087fc285e1e49a30def8d2421
826c113f782ac08bd1902c82e2c3c16fa1f2fa29
4956100b48f305db54eacf701da19368a9d9a99d
32a453193da8994720b291363643c3b7529119cf
f1ecdcc90f948e3d4fc153b1328c054413497833
aa7cee78f21525d6c5c5bf790f782d8108f97fce
91293ebefbe635cc2bd4a41ff30bff38e064c0f5
7344aff8dd17c0913ecb96112e5e26b1738c8d50
44324f193b6980b404c3c85a878cd1602bb65f0f
ceacb4567608fc19abddc237c65ffab97117eeb2
35531cf91a5ae6e10d5a9e6ac6f6e3aeb3837c43
87ee6d1d56038f2edaf50b4261c471c68fac11cc
640c1d832e637d1724c05cf52919d9d731689649
207bc1a943aae00695b7055a75117cf8c3cd8207
fdf0c72865fc0c46410d3d119591fdc23490afa7
871f3e35215e51e34192afc4586a146ed88bd102
147d07ea34be8c47e02589c04c8abeb3a0cb9052
19813747ce50b6ff9769a0d51eb12ce62f02a880
3af208fba66798bef32f9b8e42c4c8fb83a5a3f9
6dd9a878af45c7b71cb30ecd1322e2136dda90e6
d1615ddd89f49652165525c984a44d9e4174f2b1
99440e6f761935446cd86756abd5667a18be13d2
61437e135267f5a8dde45c0313ebc22da185e6cb
50730dc512e0a9388b1fec6d4dae4a8f4fb5728b
17d70f8a45ec6ef502f9d2617d1f2197e6d5f2ca
f04f4e6ef90e37ed2563fb425072ecab41a7a26f
3b499f1cdfce01192a5aadc94092234eb9ffe366
6c593e3e6a78ba01b3b751849d68d54109f8b314
c01e10906ade17fe729d485c2019508cb6d2d244
7afa99fa8a85a4209e30a16571ff7d174c1f7dfa
2d748415e006913c295530e37ba0a29f0ab7b326
a06e3b87cd987c51499797bd953dc513703041ec
e05e8a5c1580f24029371ec9f942dc14848e66e6
5ea3a2017f28d41e28aa7a366dc946da931ae18c
5b8d47eedb09eb08d46303fce4ec86d6f6db20b2
26df610618b6921da605ffc962b3b883abb60f75
3831fe0cdfd9c9baef37ca36c83a1d6405903433
a50db36196e84eb5abe38e17b36f2c22d12a727e
f5f168a183e5df7ff74b7c7c765d06edc6ce32d6
9b8b241dbe40ff84cd25823d9053c849ac6b28ca
316187712fa9c3bd7106525dfb16507c825e5eb6
6c8b1cb22bc6be6e6a990f1b951bf43e64e13302
5994c3c7c0c09d70cf0949e23e3a094e7cf1d49a
3a51e6ef7d715151d484750993b5c802725755c9
5acfea24a8fde5b013f853621bb0c9063cb9461c
phil@phil-Inspiron-5468:~/git/JankariTechUG/accounting$ git rev-list --count main
64
phil@phil-Inspiron-5468:~/git/JankariTechUG/accounting$ git rev-list --count --no-merges main
43

main sees 64 commits, 21 are merge commits and 43 are "real" commits. The merge commits don't have any timestamp to validate. The 43 "real" commits are all being validated, and it is taking a long time to do the checks back to the timestamp servers.

There could be 2 approaches:

only check a few commits on the end of the branch being checked (maybe maximum of 20)
investigate what are all the requests sent to the timestamp servers, and see if any of those are repetitious and can be done only once and then cached.

It is taking hours for CI to validate the whole commit chain of a repo. We want CI to be quicker! For example, in accounting we have: ``` phil@phil-Inspiron-5468:~/git/JankariTechUG/accounting$ git rev-list main 9a4a08c6beece26b857fc7d0014a2d330a565c5a d45f390324ce20837948771384b89aae41cf9554 42278a64ea1ef205d0484fc93d56d6c7eb81d4b3 581e44a232b6ed6474647a64ca1ce729397514f6 866f59a61c105cd97663e796e584225ceb58ca5e 14c6c5ef0290fa6f421de46d93e45bbcb71832c6 888bd3385b0224feef1d8a52768d48ab0676ffe0 c352097f9952e7c3369f1034695157e073245929 b5e41c60de330b471c655ca30ea01d5a27b667d7 6f1e602d8bcb103704221c34468911635d2fcf17 fbe6c7d41146544741a06fa65317fcf1d9165d97 2ac7bbbbf6e1c5a1bbcbb5bba9835ace697d5d78 9715cc30c84b013444b020843f6d174f66f9d2e8 5eb4697fecd15e79263c16f942bc196d5d83c3b9 c5f2b61bae70dc10188f3c642b633c4677225f91 8e94bf5689a3bf99e8e9440484041c724b171113 3e2f36b583eb97a8fdc676a3378ae3e6e6175217 05272be96dd5fc4657e1710a1ad2271bca1bf509 ccb85560b13c4e720beeda9907d2face4f5b1195 fbb6dca2c3aef5a087fc285e1e49a30def8d2421 826c113f782ac08bd1902c82e2c3c16fa1f2fa29 4956100b48f305db54eacf701da19368a9d9a99d 32a453193da8994720b291363643c3b7529119cf f1ecdcc90f948e3d4fc153b1328c054413497833 aa7cee78f21525d6c5c5bf790f782d8108f97fce 91293ebefbe635cc2bd4a41ff30bff38e064c0f5 7344aff8dd17c0913ecb96112e5e26b1738c8d50 44324f193b6980b404c3c85a878cd1602bb65f0f ceacb4567608fc19abddc237c65ffab97117eeb2 35531cf91a5ae6e10d5a9e6ac6f6e3aeb3837c43 87ee6d1d56038f2edaf50b4261c471c68fac11cc 640c1d832e637d1724c05cf52919d9d731689649 207bc1a943aae00695b7055a75117cf8c3cd8207 fdf0c72865fc0c46410d3d119591fdc23490afa7 871f3e35215e51e34192afc4586a146ed88bd102 147d07ea34be8c47e02589c04c8abeb3a0cb9052 19813747ce50b6ff9769a0d51eb12ce62f02a880 3af208fba66798bef32f9b8e42c4c8fb83a5a3f9 6dd9a878af45c7b71cb30ecd1322e2136dda90e6 d1615ddd89f49652165525c984a44d9e4174f2b1 99440e6f761935446cd86756abd5667a18be13d2 61437e135267f5a8dde45c0313ebc22da185e6cb 50730dc512e0a9388b1fec6d4dae4a8f4fb5728b 17d70f8a45ec6ef502f9d2617d1f2197e6d5f2ca f04f4e6ef90e37ed2563fb425072ecab41a7a26f 3b499f1cdfce01192a5aadc94092234eb9ffe366 6c593e3e6a78ba01b3b751849d68d54109f8b314 c01e10906ade17fe729d485c2019508cb6d2d244 7afa99fa8a85a4209e30a16571ff7d174c1f7dfa 2d748415e006913c295530e37ba0a29f0ab7b326 a06e3b87cd987c51499797bd953dc513703041ec e05e8a5c1580f24029371ec9f942dc14848e66e6 5ea3a2017f28d41e28aa7a366dc946da931ae18c 5b8d47eedb09eb08d46303fce4ec86d6f6db20b2 26df610618b6921da605ffc962b3b883abb60f75 3831fe0cdfd9c9baef37ca36c83a1d6405903433 a50db36196e84eb5abe38e17b36f2c22d12a727e f5f168a183e5df7ff74b7c7c765d06edc6ce32d6 9b8b241dbe40ff84cd25823d9053c849ac6b28ca 316187712fa9c3bd7106525dfb16507c825e5eb6 6c8b1cb22bc6be6e6a990f1b951bf43e64e13302 5994c3c7c0c09d70cf0949e23e3a094e7cf1d49a 3a51e6ef7d715151d484750993b5c802725755c9 5acfea24a8fde5b013f853621bb0c9063cb9461c phil@phil-Inspiron-5468:~/git/JankariTechUG/accounting$ git rev-list --count main 64 phil@phil-Inspiron-5468:~/git/JankariTechUG/accounting$ git rev-list --count --no-merges main 43 ``` `main` sees 64 commits, 21 are merge commits and 43 are "real" commits. The merge commits don't have any timestamp to validate. The 43 "real" commits are all being validated, and it is taking a long time to do the checks back to the timestamp servers. There could be 2 approaches: 1) only check a few commits on the end of the branch being checked (maybe maximum of 20) 2) investigate what are all the requests sent to the timestamp servers, and see if any of those are repetitious and can be done only once and then cached.

artur commented

2025-05-28 08:14:47 +00:00

merge commits don't have a timestamp in our case, but in theory could.
Do commits without any stamp also take so much time to be validated?

merge commits don't have a timestamp in our case, but in theory could. Do commits without any stamp also take so much time to be validated?

phil referenced a pull request that will close this issue

2025-05-29 04:20:58 +00:00

feature: limit the number of commits to be validated #9

phil closed this issue

2025-06-02 04:10:48 +00:00

phil commented

2025-06-02 04:14:51 +00:00

merge commits don't have a timestamp in our case, but in theory could.
Do commits without any stamp also take so much time to be validated?

The validation code "skips" quickly through commits that have no timestamps. The validation is checking that commits with timestamps do have valid timestamps.

> merge commits don't have a timestamp in our case, but in theory could. > Do commits without any stamp also take so much time to be validated? The validation code "skips" quickly through commits that have no timestamps. The validation is checking that commits with timestamps do have valid timestamps.

phil referenced this issue

2025-06-02 11:48:36 +00:00

fix: stop correctly when MAX_COMMITS_TO_CHECK is reached #10

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: JankariTechUG/GitTrustedTimestamps#8