This may seem like a very obvious tip, however, it is something that can easily be overlooked. Check your server to make sure that the ~/App_Data folder of your Kentico installation is totally excluded from its antivirus engine. If the antivirus engine is scanning that directory then you might already be a victim of a performance hit.
Let me give a little background first. The Kentico CMS application utilizes the ~/App_Data folder to perform various tasks like importing and exporting objects, publishing pre-compiled sites via the VirtualPathProvider, and reading and writing temporary files for the Smart Search index.
That last one is where I really noticed a performance issue. You might think that the Smart Search index would be more of a database based solution. Not true. The Smart Search functionality is based off of the open source Lucene.Net search engine library.
To create indexes Lucene needs to read and write a bunch of small files that contain raw data chunks. These chunks make it easier and quicker for the search engine to do its job which is to query the data. Kentico places the Lucene.Net files of each Smart Search index under the ~/App_Data/CMSModules/SmartSearch folder.
Back to my painful real world example. Over at BizStream, we are currently working on a realty site for one of our clients. The site has over 25,000 nodes in the the content tree that represent house listings here in West Michigan. And as you might be able to guess, searching those listings is one of the main feature of the site.
Every night we download the latest data from the local real estate association, and automatically update any new or changed listings. After that process is finished, we trigger the Smart Search index to rebuild so that the Search page can work correctly on the site.
This process was taking over 6 hours on our server. And it was really killing us because we knew on a single, fairly powerful, development machine it was running closer to a third of that time.
After a lot of trial and error we decided to look at the antivirus client settings and sure enough there were no exclusions on the Kentico folder. After we added the proper folder exclusion the time to run the Smart Search index rebuild was down to about 2 hours.
One more bonus tip; also make sure that the ~/App_Data/CMSModules/SmartSearch/indexname folder has write permissions so that the site can read and write all those index files that Lucene.Net generates. You will see an error about locking violation if that folder doesn’t have the necessary permissions.