The Unity notification processes only care about events relating to the inbox and, of course, only have access to the inbox and deleted items folders. The inbox is indexed for content such that the conversations run a bit quicker when you log in to check messages and such. However, the MAPI login process is independant of this and the size of the inbox doesn't have much to do with the login (authentication) time for Exchange I don't believe.
The size of the inbox will certainly make it take longer to get a query back from Exchange on the state of the inbox - the total size is what matters there. But the indexed views added and kept up to date in the background by Exchange should mitigate most of that (i.e. we aren't asking Exchange to do a scratch recalculation of the inbox contents on the fly every time). But certainly huge inboxes are going to slow that indexing and query process down.