Drupal generates nicely styled 404 pages that are easy to customize. And since 404 responses can be served from the page cache, the performance hit from the occasional stray image can be minimal.
However, if you're embedding 3rd-party content (such as ads or social widgets), there is an additional risk that a misbehaving app can generate bogus requests with random-ish query strings. The unpredictable URLs will totally defeat the page cache, so on a really busy site the added load can be crushing.
On Emmys.com we added a few extra rewrite rules to handle this possibility, taking care to make sure Drupal still handled imagecache requests:
These directives replace the main ErrorDocument and RewriteRule settings in .htaccess.
# Use default 404 handler
ErrorDocument 404 default
# Rewrite URLs of the form 'x' to the form 'index.php?q=x'.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} /imagecache/ [OR]
RewriteCond %{REQUEST_URI} !\.(png|gif|jpe?g|css|js|cgi|ico|xml|swf|flv|dll|exe)$ [NC]
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]