Is there a particular reason you add every folder as a separate collection instead of the parent folder?
Of course this shouldn’t cause the problems you describe, but I would expect it to be slower.
If it’s the exact same files, did you already try to copy those and only those to another location and add that as a collection to digikam - to see if the problem occurs as well?