## Wednesday, October 20, 2010

### Buildbot - Issue with svn poller

SVN poller may miss a check-in based on poll interval.

The current behavior of the poller is

The poller polls the version control system and stores last change (version number).  Subsequent changes are notified as log entries. These log entries are marked with the Time Stamp when the changes are noticed. These log entries are used to create change objects that is then passed to scheduler to trigger builds.Scheduler sees these change objects with the same timestamp and picks the latest change object to trigger the build.

The issue with this model is that if there are multiple changes within a single polling interval, this poller will result in triggering build only for the last one.

Some solution though half baked is to
1. Decrease the polling interval to say 30 sec. While this may decrease the number of errors or missed build. It does not guarantee that the builds may not be missed. It may only decrease their numbers.
2. The other solution is to use a script ( a subversion hook-script which submits changes to svn ) to trigger buildbot builds when a developer checks in the code. If this is server side hook then it is great. The issue is not every one has the privilege of setting up server side hooks.
3. Use a watcher which behaves like svn poller except that it needs to watch for every commit and send the change objects per commit to buildbot.

There is a file - "svn_watcher.py" distributed with buildbot. This scripts does the usual polling but suffers from the same lacunae as that of svn poller.

The way to run this file is

./svn_watcher.py subversion_url host_buildbot_master:port watch


Overcoming limitations of svn_watcher.py

Issue 1: Does not work in –non-interactive mode

Run the following command for the first time to enable appropriate authentication
svn log --xml --verbose --limit=1 subversion_url


Issue 2: Making it notice every commit and notify it to buildbot

# Check if multiple revisions have been made during this interval
if oldRevision != -1:
# Have to discount the state loss
revJump = int(revision) - int(oldRevision)
#print "Jump in Revision: " + str(revJump)
if revJump > 0:
# We have to get information about every change
cmd_multilog = ["svn",
"log",
"--xml",
"--verbose",
"--non-interactive",
"--limit="+ str(revJump + 1),
repo]
if sys.platform == 'win32':
f = win32pipe.popen(cmd_multilog)
f.close()
else:
xml2 = getoutput(cmd_multilog)
# Process multiple log Entries
doc2 = xml.dom.minidom.parseString(xml2)
logentries = doc2.getElementsByTagName("logentry")
changedetails = []
#print  "Len of logentries: " + str(len(logentries))
for logentry in logentries:
el_revision = logentry.getAttribute("revision")
if int(el_revision) > int(oldRevision):
el_author = "".join([t.data for t in
logentry.getElementsByTagName("author")[0].childNodes])
el_comments = "".join([t.data for t in
logentry.getElementsByTagName("msg")[0].childNodes])
el_pathlist = logentry.getElementsByTagName("paths")[0]
el_paths = []
for p in el_pathlist.getElementsByTagName("path"):
el_paths.append(
"".join([t.data for t in p.childNodes
]))
changedetails.append([el_revision,
el_author,
el_paths])
changedetails.reverse()
#print "Len of change details: " + str(len(changedetails))
# Send change details
for changedetail in changedetails:
chng_cmd = ["../../bot/bin/buildbot",
"sendchange",
"--master=%s"%master,
"--revision=%s"%changedetail[0],
]
chng_cmd += changedetail[3]
if verbose == True:
print chng_cmd
if sys.platform == 'win32':
f = win32pipe.popen(chng_cmd)
print time.strftime("%H.%M.%S ") + \
"Revision "+changedetail[0]+ ": "+ \
f.close()
else:
print time.strftime("%H.%M.%S ") + \
"Revision "+changedetail[0]
# Send buildbot command
out_xml  = getoutput(chng_cmd)
currentRevision = changedetail[0]
else:
print time.strftime("%H.%M.%S ") + \
"nothing has changed since revision "+ \
revision
currentRevision =  revision
else:
#currentRevision = revision
currentRevision = str(int(revision) - 2)
print time.strftime("%H.%M.%S ") + \
"Monitoring for change after revision " + \
currentRevision

return currentRevision