To exclude one category i.e. category 3, add the following to the root index.php before the include to wp-blog-header.php:
if(isset($_GET['feed'])) {
$cat=-3;
}
To exclude multiple categories i.e. 3,4 and 5, you’ll have to work backwards and include only the categories that you want i.e.
if(isset($_GET['feed'])) {
$cat='1 2 6';
}