I had a couple of times of this error message while merging packages:


/var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/build-amd64-x86_64-pc-linux-gnu-nptl/catgets/gencat.o: file not recognized: File format not recognized
collect2: ld returned 1 exit status
make[2]: *** [/var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/build-amd64-x86_64-pc-linux-gnu-nptl/catgets/gencat] Error 1
make[2]: Leaving directory `/var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/glibc-2.14.1/catgets'
make[1]: *** [catgets/others] Error 2
make[1]: Leaving directory `/var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/glibc-2.14.1'
make: *** [all] Error 2

When I first saw this error message, I didn’t realize it was caused by corrupted object file in ccache, which was resulted from computer crash. I didn’t make that connection at first, searched for answers as best as I could, but nothing had helped out until I ran ccache -C to clean up the cache. And the package successfully merged.

So, every time my computer crashed during merging, I cleaned up ccache’s cache. If the merging process has only taken a few minutes when crashes, then cleaning up cache would be fine. However, if the process has been going for a half hour, then that’s really bad because you have to completely start over the compilation.

Today, I had this case again and yes, my Gentoo does crash. I knew I had to find a way to deal with this, this probably won’t be the last time. Disabling ccache certainly isn’t the correct resolution, because having the compiled cache in the first is why you use it and since the problem isn’t the compilation but crash, therefore no reason to support disabling it.

By the way, though handbook says:

Warning: ccache is known to cause numerous compilation failures. Sometimes ccache will retain stale code objects or corrupted files, which can lead to packages that cannot be emerged. If this happens (if you receive errors like “File not recognized: File truncated”), try recompiling the application with ccache disabled (FEATURES="-ccache" in /etc/make.conf) before reporting a bug. Unless you are doing development work, do not enable ccache.

I have never had problem with it except this crash issue. If you will re-merge same package and same version (of the package, not of ebuild revision) more than one time, you definitely needs it to cut the compilation time. Re-merging usually happens when you are trying out a new packages and you are not sure what USE flags to use. It’s possible that you re-merge with different USEs. Or when a package’s ebuild isn’t working well like Firefox last month, the source isn’t changed but build process in Gentoo side is, this is how the ccache comes in to help a lot.

To resolve the issue, I began with ccache’s website, nothing was mentioned about it, then I saw the following section in ccache manpage, which says:

Corrupt object files
It should be noted that ccache is susceptible to general storage problems. If a bad object file sneaks into the cache for some reason, it will of course stay bad. Some possible reasons for erroneous object files are bad hardware (disk drive, disk controller, memory, etc), buggy drivers or file systems, a bad CCACHE_PREFIX command or compiler wrapper. If this happens, you can either find out which object file is broken by reading the debug log and then delete the bad object file from the cache, or you can simply clear the whole cache with ccache -C if you don’t mind losing other cached results.

The problem here is I didn’t have CCACHE_LOGFILE set up beforehand and since never actually is the problem of ccache, not entirely to say the least. It seems rather ineffective for enabling this debug log. The fact is that it’s already too late for this incident.

My chance to resolve the problem lies in ccache’s cache files, but I look into the problematic file first:


$ file /var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/build-amd64-x86_64-pc-linux-gnu-nptl/catgets/gencat.o
/var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/build-amd64-x86_64-pc-linux-gnu-nptl/catgets/gencat.o: data

A normal object file gives:


$ file /var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/build-amd64-x86_64-pc-linux-gnu-nptl/csu/crtn.o
/var/tmp/portage/sys-libs/glibc-2.14.1-r3/work/build-amd64-x86_64-pc-linux-gnu-nptl/csu/crtn.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

So, now I know what I should look in output of file:


$ cd /var/tmp/ccache
$ find -name *.o -mtime 0 -exec file {} \; | grep 'data$' | cut -f1 -d: | xargs ls -l

This should give you a list of files which were modified within 24 hours and file recognizes them as data. Once you are certain one of them are corrupted file by checking the modified time, you remove them all:


$ find -name *.o -mtime 0 -exec file {} \; | grep 'data$' | cut -f1 -d: | xargs rm

Yes, you kill them all. I have no way to tell which one really is corrupted object file. The easier way is to delete all of them. It’s cache, if it’s not there, then the compiler can always compile again. It is better than clean up the whole cache.

Once you remove the files, you resume the merge. The problem should be gone.