Currently a new connection is opened for each new layers, which works fine for local database, but is very slow for remote databases.
I've managed to speed the listing of layers in both repository and Database layers dialog. In both cases the same instance of datasource is reused under the hood which saves time on establishing new connections. It can be sped up even further by using GEOMETRY_COLUMNS table in the database (no need to do additional query to determine geometry type).
I've also tried to use shared dataset / connection for layers added to the map. But it wasn't stable without critical sections (works relatively stable with remote databases, but produces crashes and deadlocks with the local ones). I disabled it for now, but it's available in the settings: Data -> Vector -> OGR datasources -> Share connection.
I've also implemented async adding of layers to the map.
I can confirm this has been fixed. It is much faster now.