A well-optimized application should rely on a well-optimized server. You should know the basics of server performance to make sure there is no bottleneck outside symfony. Here are a few things to check to make sure that your server isn't unnecessarily slow.
Having magic_quotes_gpc turned on in the php.ini slows down an application, because it tells PHP to escape all quotes in request parameters, but symfony will systematically unescape them afterwards, and the only consequence will be a loss of time--and quotes-escaping problems on some platforms. Therefore, turn this setting off if you have access to the PHP configuration.
The more recent PHP release you use, the better. PHP 5.2 is faster than PHP 5.1, and PHP 5.1 is a lot faster than PHP 5.0. So make sure you upgrade your PHP version to benefit from the latest performance improvements.
The use of a PHP accelerator (such as APC, XCache, or eAccelerator) is almost compulsory for a production server, because it can make PHP run an average 50% faster, with no tradeoff. Make sure you install one of the accelerator extensions to feel the real speed of PHP.
On the other hand, make sure you deactivate any debug utility, such as the Xdebug or APD extension, in your production server.
Note You might be wondering about the overhead caused by the mod_rewrite extension: it is negligible. Of course, loading an image with rewriting rules is slower than loading an image without, but the slowdown is orders of magnitude below the execution of any PHP statement.
Some symfony developers like to use syck, which is a YAML parser packaged as a PHP extension, as an alternative to the symfony internal parser. It is faster, but symfony's caching system already minimizes the overhead of YAML parsing, so the benefit is nonexistent in a production environment. You should also be aware that syck isn't completely mature yet, and that it may cause errors when you use it. However, if you are interested, install the extension (http://whytheluckystiff.net/syck/), and symfony will use it automatically.
Tip When one server is not enough, you can still add another and use load balancing. As long as the uploads/ directory is shared and you use database storage for sessions, a symfony project will react seamlessly in a load-balanced architecture.