Clean/Rebuild deletes stats and cache objects

[jdonald]

Apologies if this has been brought up before. I did not find mention of it in the currently open issues and pull requests.

I'm using Visual Studio 2013 Update 5 with CMake-generated vcxproj/sln files. I am able to get reliable cache hit rates from what I see in clcache.exe -s. However, one unfortunate behavior is that anytime I select Clean, all the cache stats get reset to zero, apparently because stats.txt has been removed, and several cache objects are removed as well.

This unfortunately applies to the Rebuild command too, or even Project Only -> Rebuild Only .

It appears that the cleaning mechanism in Visual Studio 2013 works by tracking any files written by cl.exe so it can know how to do a reliable clean. This would not be a problem for nmake projects, and I'm not sure if this nifty mechanism was in earlier versions of Visual Studio.

Has anyone else encountered this? Any solutions or practical workarounds?

Or if you are using Visual 2013 or 2015 and have not seen this problem, that would be good to know. I do have one extension installed (VSCommands) but I doubt that it would affect the cleaning behavior.

[frerich]

I'm not sure how Clean in Visual Studio works exactly, but I'd assume that if you set the CLCACHE_DIR environment variable to a path which is outside of the project structure then any operations performed by VS don't apply to the cache.

[jdonald]

My CLCACHE_DIR is on the C:\ drive in my home directory, while the project source and sln+vcxproj/build directory are on a separate drive altogether. Thus, it does not appear that Visual Studio discriminates clean targets based on the enclosing directory.

Sounds like you have primarily tested under nmake, @frerich?

[frerich]

I cannot recall having ever tried clcache with Visual Studio project files (or anything else built on msbuild, AFAIK Visual Studio builds via msbuild) myself, no. I used it with custom build systems and in some cases with CMake-based build systems. I seem to recall that there were other people who successfully used clcache with Visual Studio though; I do not recall hearing about this 'Cleaning in VS deletes stats' symptom before though.

[jdonald]

To narrow things down, I uninstalled the VScommands extension and tried Visual Studio 2010, and tested with a non-CMake-generated sln. This behavior still reproduces in VS2010 with no extensions, and I suspect it may have existed even further back.

Grepping for stats.txt in my local project, I found that it is mentioned in Sample.Build.CppClean.log :

C:\USERS\JDONALD\CLCACHE\MANIFESTS\54\5492E2EA5E56990C01AD563E001825AE.DAT
C:\USERS\JDONALD\CLCACHE\OBJECTS\6B\6BA581C0457E9640C07310FB59CD925F\OBJECT
C:\USERS\JDONALD\CLCACHE\OBJECTS\6B\6BA581C0457E9640C07310FB59CD925F\OUTPUT.TXT
C:\USERS\JDONALD\CLCACHE\STATS.TXT
C:\Users\jdonald\Documents\LeapSDK\samples\x64\Release\cl.command.1.tlog
C:\Users\jdonald\Documents\LeapSDK\samples\x64\Release\CL.read.1.tlog
C:\Users\jdonald\Documents\LeapSDK\samples\x64\Release\CL.write.1.tlog
C:\Users\jdonald\Documents\LeapSDK\samples\x64\Release\CL-cl_real.read.1.tlog
C:\Users\jdonald\Documents\LeapSDK\samples\x64\Release\CL-cl_real.write.1.tlog
C:\USERS\JDONALD\DOCUMENTS\LEAPSDK\SAMPLES\X64\RELEASE\SAMPLE.OBJ
C:\Users\jdonald\Documents\LeapSDK\samples\x64\Release\SampleVS2010.write.1.tlog

(note: new project is on the same drive as the cache, but it seems unlikely that would matter.)

If I delete the log, I can recreate it by selecting Clean or Rebuild, and the log is still named *.Build.CppClean.log. So it is definitely the act of cleaning that clears the stats and cache objects.

I used it with custom build systems and in some cases with CMake-based build systems.

CMake still requires selecting a generator though (e.g. Visual Studio 10 2010 Win64). I am guessing you used CMake with the nmake or MSYS Makefiles generators in particular?

I agree it is strange that nobody else has mentioned this. If someone here could confirm if they have or have not seen this behavior on Visual Studio 2010, 2012, 2013, or 2015, that would help point me where to look next.

[webmaster128]

Sounds like a very sophisticated way of cleaning, where the build system tracks which files are created during the build.

This will not happen in any Linux-like Makefile (nmake, jom, MinGW make) because there you explicitly write down in the Makefile what to delete when running make clean. This can be a list of object files or an expression like builddir\*.obj. But it will not touch side effects of the compiler.

Maybe one could work around that by writing the cache entries in a database (e.g. an SQlite file), such that only a file change, not a file create is visible to the build system. But that would be a major performance loss, especially compared to hardlinking.

[jdonald]

Visual Studio may be tracking file changes as well as file creation. Or at least in practice stats.txt gets deleted too, so if we tried this with an SQLite database, it might just delete the whole database!

[webmaster128]

Is it possible that the first compiler run did create stats.txt? Could you try it with a new project when stats.txt already exists?

[jdonald]

Confirmed stats.txt is included in the clean-list regardless of whether stats.txt is created by clcache or written manually.

[webmaster128]

Okay, next random try: what happens if you set CLCACHE_DIR to a place on a different partition or physical drive?

[jdonald]

Tried that already. See comment from Oct 25, 2015.

I would be interested in hearing whether this problem reproduces on your system. It's strange that this has not been reported by any other users, yet seems to happen as far back as Visual Studio 2010 or earlier.

[webmaster128]

We use the Microsoft compiler cl.exe but always with a Makefile build system. I have no clue how Visual Studio works and even setting up a basic sample project would be hours wasted for me. So sorry, I cannot really help with this question.

[jdonald]

Creating a default project might not take quite as long as you think. Visual Studio's wizards are better than that of some other IDEs.

• Open up Visual Studio
• File menu -> New -> Project...
• select Win32 Console Application, Next, Finish
• Build menu -> Build ConsoleApplication1

It would still take a bit more learning to debug clcache in this context. Note the Clean command can also be found in the Build menu.

[webmaster128]

@jdonald Assuming I might find some effect quickly, like in half an hour. What is next? Debugging and maybe adjusting clcache for a system I have no interest in ever using?

I am always happy to provide some quick feedback to push this open source project but if you need me to help in your particular setup, I need to ask for a freelance job paid per hour without any guaranteed success. If that is how we can work together, drop me a line to simon@kullo.net I think this works as a CV: https://github.com/frerich/clcache/commits?author=webmaster128

[frerich]

@webmaster128 There are a couple of other people who are perfectly capable of fixing clcache issues. I think having @jdonald provide a sequence of steps which supposedly reproduce the issue is a good first step. Having somebody else confirm that these steps indeed reproduce the issue is a second step.

Whether, who and when this issue will be debugged and fixed is a totally separate issue and independent of whether you personally use Visual Studio. That being said, I'd rather not have people use this little open source project as an advertising forum for professional services, thanks. I think such offers are better discussed in private communication.

[webmaster128]

This was somehow turning in my direction because I was the only one adding anything to this issue in months. But you're right, this is entirely open to everyone willing to dig into this.

I'd rather not have people use this little open source project as an advertising forum for professional services, thanks

Sure. Sorry about that.

Btw.: you're a good moderator.

[frerich]

@jdonald it's been a while since you reported this issue, but I wanted to bring this up again because I just learned a few things.

By now, I think this is not an issue in clcache itself. Instead, as you already suspected, Visual Studio (in fact: MSBuild) can be made to track file accesses done during the build, and it can create logs of those accesses. The CL Task documentation explains:

[..]
TLogFileWrites - Optional ITaskItem[] parameter.
Specifies an array of items that represent the write file tracking logs.
A write-file tracking log (.tlog) contains the names of the output files that are written by a task, and is used by the project build system to support incremental builds. For more information, see the TrackerLogDirectory and TrackFileAccess parameters in this table.
[..]
TrackFileAccess - Optional Boolean parameter.
If true, tracks file access patterns.
For more information, see the TLogReadFiles and TLogWriteFiles parameters in this table.

Alas, I'm not sure how to pass such MSBuild flags via the Visual Studio API, but in any case it may be worth checking whether file access tracking is enabled for you, and if so - whether disabling helps with avoiding that stats.txt gets deleted. For what it's worth, it appears that file accesses are tracked for incremental builds, so disabling those might also have the same effect.

It would be great if you could let me know if you find something. As it is, this ticket is kind of stuck. :-]

[jdonald]

Great job on that investigation.

I edited my *.vcxproj file and added:

  <PropertyGroup Label="Globals">
    ...
    <TrackFileAccess>false</TrackFileAccess>
  </PropertyGroup>

Once this is set, I no longer lose stats.txt on Clean. As soon as I set it back to true, the problematic deletions return.

[frerich]

Thank you for confirming! I'll close this issue then.