phase1,phase2: improve round robin builds

There seems to be some issue with database updates, where database
updates are asynchronous and thus the status update of the previous
build (complete_at DB column) might take little bit longer, then
preparation of the next/current build.

This is issue during job prioritization as this might result in the
build of the same builder twice in a row, for example:

 2021-06-22 02:42:54+0000 [-]  <Build mipsel_24kc number:95 results:success>: build finished
 2021-06-22 02:42:55+0000 [-] prioritizeBuilders:          mipsel_24kc complete_at: 2021-06-21 20:17:14+00:00
 2021-06-22 03:14:30+0000 [-] prioritizeBuilders:          mipsel_24kc complete_at: 2021-06-22 02:42:54+00:00

Build finishes at 02:42:54, scheduler then asks for next build at
02:42:55, but the build still has the old complete_at timestamp
2021-06-21 20:17:14 instead of the correct one 2021-06-22 02:42:54, thus
scheduling the build of oldest mipsel_24kc builder one more time.

This is so far very promising workaround attempt which checks latest
builder complete_at in builds table which seems to be updated faster,
thus using greater complete_at value seems to work for now.

References: https://github.com/buildbot/buildbot/issues/4592#issuecomment-801163587
Signed-off-by: Petr Štetiar <ynezz@true.cz>

diff --git a/phase1/master.cfg b/phase1/master.cfg
index a85382ae4fd2ee52d0d102fc90be7f721a2dfe86..be3a8941bb81d26f7a31a55c8835f64399cf35f2 100644 (file)
--- a/phase1/master.cfg
+++ b/phase1/master.cfg
@@ -154,7 +154,21 @@ def getNewestCompleteTime(bldr):
        if not completed:
                return
 
-       return completed[0]['complete_at']
+       complete_at = completed[0]['complete_at']
+
+       last_build = yield bldr.master.data.get(
+                       ('builds', ),
+                       [
+                               resultspec.Filter('builderid', 'eq', [bldrid]),
+                       ],
+                       order=['-started_at'], limit=1)
+
+       if last_build and last_build[0]:
+               last_complete_at = last_build[0]['complete_at']
+               if last_complete_at and (last_complete_at > complete_at):
+                   return last_complete_at
+
+       return complete_at
 
 @defer.inlineCallbacks
 def prioritizeBuilders(master, builders):

diff --git a/phase2/master.cfg b/phase2/master.cfg
index b9342cfc557a907d6a9728c6871398e06b04c58a..7742ad6c1e826646fe341a3bd20438c1ca9825c3 100644 (file)
--- a/phase2/master.cfg
+++ b/phase2/master.cfg
@@ -351,7 +351,21 @@ def getNewestCompleteTime(bldr):
        if not completed:
                return
 
-       return completed[0]['complete_at']
+       complete_at = completed[0]['complete_at']
+
+       last_build = yield bldr.master.data.get(
+                       ('builds', ),
+                       [
+                               resultspec.Filter('builderid', 'eq', [bldrid]),
+                       ],
+                       order=['-started_at'], limit=1)
+
+       if last_build and last_build[0]:
+               last_complete_at = last_build[0]['complete_at']
+               if last_complete_at and (last_complete_at > complete_at):
+                   return last_complete_at
+
+       return complete_at
 
 @defer.inlineCallbacks
 def prioritizeBuilders(master, builders):