When I first set up the Empathy chat client to connect to my Google apps account it took a ridiculous amount of time to load the contacts, usually over 10 minutes, and this was every time I logged in.
By changing the settings on the account though I got it to work pretty much instantaneously.
Login ID: your email address
Password: your password
Under advanced settings:
Check ‘Encryption Required’ checkbox
Check ‘Use old SSL’ checkbox