This is probably a case of my not reading the manual.
I use W3 Total Cache not because I get a ton of hits. I run it because it’s cool and interesting. With the current 0.9.1 version it suggests that you add to your .htaccess file a section to handle 404 for missing static files. So non-existent txt, css, jpg, etc. will just get the web server’s 404 page and save on CPU processing via WordPress PHP.
Sensible, but robots.txt is an imaginary file for my WordPress blog. Requests to that file get sent to index.php and the output is dynamically created. Without the exclusion, that file URL always gets 404’ed.
Easy to fix. Right above the W3TC lines just add these lines to your .htaccess file.
<IfModule mod_rewrite.c>
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^robots.txt index.php [L]
</IfModule>
The ReWriteCond is not really needed, I’m just putting it in to be complete. If a request comes in for robots.txt it will be sent to WordPress for handling.