Ok how embarassing. Turns out it was a configuration problem with DNS and sendmail. You will not encounter this problem if you are on a hosting service so you can let your eyes glaze over now.
But if you were like me testing this out on a system in the office or at home, then this may be of interest to you.
* technical geek stuff ahead *
Was pointed to the right direction on the WordPress support forums.
Sendmail utilises reverse DNS to look up the host sending the mail. If it is “localhost” with an IP of “127.0.0.1” then it will try to resolve that address and eventually it will fail. The mail will still be sent but only after a prolonged timeout.
This affected the comments of WordPress because the commenting system tries to send out mails to you notifying you of the new comment. And it is sending the mail from “localhost”, ie. the machine that it is installed on.
How I fixed it: editing “/etc/hosts” by putting a fully qualified domain name on the line showing “127.0.0.1”.
Result: blazing fast comments!