We have a single TeamCity server that has multiple build agents on other servers connecting to it. We also have multiple multi-GB source trees, each with multiple build configurations, so we have our TeamCity server configured to do server-side updates and to not clean the source tree before every build; so, the server should be sending the updated source code to the agents every time they need to run a build.
The problem is that occasionally the source code on the build agents will stop running. We'll check in some source code, and TeamCity will clearly detect there are updates and trigger a build as it should; however, the results of the build are identical to the previous build, and when we check the source tree on the agent computer, it has not been updated with the new source code. After the build agent has gotten into this state, it will not update the source tree again; we have to tell it to clean the source tree before the next build, and then it will pull everything.
Unfortunately, I haven't figured out how to reliably reproduce it, and it's very hard to catch in the logs due to the fact that we usually don't notice the sources have stopped updating until long after it happens, until somebody tries to download an artifact and discovers it doesn't have changes in it that it should have. We have seen this happen a number of times, though, so it's definitely not an isolated incident, but it's hard to say when it could happen again.
Actually I have encountered this problem too, however like you say it is a difficult one to reproduce because it is an intermittent type of bug.
Also what I've observed about this problem is that sometimes zero-byte files get created in the workspace, other times they don't get updated at all. This leads me to suspect that the AccuRev UPDATE command may be responsible.
We're on a slightly older version of AccuRev (4.7.2) so I might check the release notes to see if anything was addressed in later versions. Can you confirm which version you are running on?
I hadn't noticed the problem happening more often when there are stranded or overlapping files, but I'll keep an eye out for it.
I have a pile of logs from the last time it happened. I reviewed them and see nothing of note at the time of the checkout and build, but I would be happy to post whatever you think would be helpful.
Also, I would be happy to elevate the log levels or even load an instrumented version of the plugin and pass logs to you if there is a chance that it might help you out.
As it is I can find rhyme nor reason to the behaviour but now that its been mentioned a few times now I'll keep a very close eye on it, let you know what I find.