I never heard about this problem before (the only close situation is on Windows (only), when browsing a network path that points to many files). Therefore, this is something specific to your system. The only advise I can give is to use one the available OSX applications to clean your system.