A highly important issue of any notable website is performance. You may have created the best website in the world but if it dies under load, you're gonna lose customers. User experience is very important today and having a slow website doesn't help at all.
Optimization is, however, not a trivial issue and requires expertise in different fields. There are so many different places where you can optimize that it's not always that easy to know what to focus on. Though this post will adhere to its title I'll still list here where optimization can occur in a website.
* Correct usage of HTTP headers to make client browsers request less information.
* Smaller responses (gZIP / more CSS - less HTML / use of Ajax to return instead of reloading complete pages).
* Optimization of your server machine(s) hardware AKA "I need more CPU, I need more memory and 'how much is another 1U?".
* Server software optimizations: Webserver (such as Apache) / Scripting engine (such as PHP) / DBMS (such as MySQL) / cache engines (such as memcached, APC) could and should be tweaked heavily. Failing to define an appropriate index in your DBMS or making some wrong choices on where and when the webserver saves user sessions, for example, could carry a heavy toll.
* Network optimizations: anyone said CDNs?
The fun part is that all these parts are well entangled.
I've read an interesting post about prefering MySQL cache over the popular memcached in some situations. Though it was pretty much one-sided (ignoring the overhead of a database connection), it rose some interesting points and is well worth reading.
An advantage towards the DBMS that I consider relevant is greater flexibility. For example: you allow outdated information to persist (such as statistics). Say you want it updated about every 5 minutes. If you cache it for 5 minutes it'll expire and then you may face a situation in which a few threads query the database again to get this information. If you use a Memory table for this information you can read it and, if expired, set some writing lock that'll cause other thread to keep reading the expired information until it's well updated.
Another interesting older post about performance showed some interesting benchmarks. The biggest problem of relying on others' benchmarks is there can always be one single parameter different on your system that would mean the results for you would be totally different. For example:
* A new version of a product has just changed everything about it.
* A configuration option made a product completely flunk its benchmark tests.
* Your queries may not be similar at all to what is tested (though you may think it is).
So these were my 2c about performance for now. The bottom line is simple: there's always a part of your system that's not properly optimized. The best is to check the painful spots and remedy them while maintaining an overall look of what your system has to provide.