MRPP_AddBuildTags occasionally fails with 401 error

[ogerovich]

We are using the version from July 31 in TC 2018.1.1. We specified particular credentials in the "Add Build Tags" build step. It works 99% of the time, but on rare occasions in random builds we get this:

[15:02:23]prepare-internal-credentials:
[15:02:23]
[15:02:23]prepare-custom-credentials:
[15:02:23]     [echo] using custom credentials
[15:02:23]
[15:02:23]prepare-tags:
[15:02:23]
[15:02:23]addTags:
[15:02:23]     [http] HTTP Request
[15:02:23]     [http] ********************
[15:02:23]     [http] URL:		https://buildserver/httpAuth/app/rest/8.0/builds/id:1302210/tags
[15:02:23]     [http] Method:		POST
[15:02:23]     [http] 	Content-Type: application/xml
[15:02:23]     [http] Entity:		yes
[15:02:23]     [http] ------ BEGIN ENTITY ------
[15:02:23]     [http] <tags><tag>ObfuscatedTag1</tag><tag>ObfuscatedTag2</tag></tags>
[15:02:23]     [http] ------- END ENTITY -------
[15:02:23]     [http] 
[15:02:23]     [http] HTTP Response
[15:02:23]     [http] ********************
[15:02:23]     [http] Status:		401
[15:02:23]
[15:02:23]BUILD FAILED
[15:02:23]E:\Obfuscated\Obfuscated\build9079597898286185054.tmp:73: Expected Status [204] but got [401] for URI [https://buildserver/httpAuth/app/rest/8.0/builds/id:1302210/tags]

I can't figure out the cause. Credentials in the build steps do not change between builds. Is using "8.0" in the URL a problem? Should it use the most recent endpoint?

Is it possible that TeamCity cannot handle these REST calls if multiple calls happen at the same time?

[yaegor]

There might be some details on 401 error in the server logs which are worth checking.

[ogerovich]

Indeed! Thanks, Yegor! This is what I found:

[2018-12-01 00:34:57,767]   WARN -     jetbrains.buildServer.AUTH - Error occurred during HTTP authentication while processing request POST '/httpAuth/app/rest/8.0/builds/id:1396984/tags', from client 10.175.52.113 (127.0.0.1:59642), user-agent "Java/1.8.0_45", no auth: jetbrains.buildServer.serverSide.auth.AuthenticationFailedException: LDAP search operation returned an error while searching for user entry by 'OBFUSCATED' to perform bind. While initializing LDAP connection. Inaccessible or wrong server specified in the LDAP configuration? Check 'java.naming.provider.url' property (current value: 'ldaps://OBFUSCATED:636/DC=foo,DC=bar,DC=buzz'). Original error: org.springframework.ldap.CommunicationException: foo.bar.buzz:636; nested exception is javax.naming.CommunicationException: foo.bar.buzz:636 [Root exception is java.net.ConnectException: Connection timed out] (enable debug to see stacktrace)

This could indicate a temporary outage of our LDAP server. We currently specify credentials in Add Build Tags step. I suppose that's why it goes to LDAP to authenticate. If I omit the credentials, so it goes through "prepare-internal-credentials" target, will that skip LDAP authentication?