In the past I have blogged about various DotNetNuke performance and management issues, focusing on how to keep the Site Log, Event Log, and other database-related functions in place. In this post, I'm going to take a look at a few other aspects, not all 100% DotNetNuke based, that should be closely monitored and addressed as well. For those of you that follow me on Twitter, my recent posting about clearing almost 40Gb of rogue data across servers is all part of what lead me to this posting.
In the end, my problems were all caused by the fact that one server that I am responsible for had used all available disk space and as such, the sites on the server were taken offline. Come to find out, the server ran out of space due to rogue static log files, at this point in time I wasn't even aware that DotNetNuke stored these logs, and I had to use a tool called "Tree Size" to ultimately find the data.
Why Disk Space is a Concern
This may seem like an obvious answer, but I think that this issue, first of all, is worth noting. On one regard we all know it is best to keep the size of a site as small as possible, that it helps with backup, and depending on the item we are discussing size of, it can help with performance. However, there is a more urgent and important aspect of size management that comes in to play. I will break this down into two sub-components: cost and system stability.
Disk Space and Cost
Overall this should be pretty self-explanatory, the more disk space that you use the more you will have to pay. If you are on a shared hosting plan, or a plan that has "unlimited" space this doesn't impact you from a cost perspective directly but consider downstream uses and costs. If you have a site that is unnecessarily bloated and you are paying for managed backups with your hosting provider or possibly backing up your assets to the cloud, you are often paying by the GB not only for storage but also for transfer. If your site has 20Gb of extra, un-necessary data this can add substantial costs.
If you are in an environment that is space limited you can encounter additional costs to get larger disk allocations etc. All in all there can be, and often are costs associated with additional data use, however, those costs are sometimes hard to immediately notice.
System Stability
From a business perspective, this is the most critical aspect of any application setup. For example, in previous posts I have discussed the performance aspects of the DotNetNuke EventLog table growing excessively and how that can slow down the site due to database performance. This problem is typically slow to appear and leaves the site still usable, although slow.
When we start talking about disk space usage through the scenario is a lot more impactful in extreme situations. In a case where a systems' drive is filled to capacity, it is very common for the entire system to come down, due to the fact that the database and other systems cannot allocate disk space for processing. This is the critical point in any system management process.
What Types of Static Log Data Exist?
With DotNetNuke 6.2.0 and later we have two different types of log data that we need to be aware of, that are "hidden" from the regular views of the site and the DotNetNuke database. These are Log4Net logs generated by DotNetNuke and IIS Logs that are part of ANY website that is hosted by IIS, not just DotNetNuke.
Log4Net
In I believe it was DNN 6.2.0 DotNetNuke started to include an additional logging mechanism, a commonly used framework called Log4Net. (Note it was included in 6.x, but not enabled until what appears to be 6.2.0) As a system administrator Log4Net is amazing, the logs are dumped to a file and are much more easily read. However, the default DotNetNuke configuration does not have any "Retention" rules set in place for what to do with these logs after a period of time. In my case, I had a site that was launched in March 2012 and received about 100,000 visitors a month. It also just so happened that this site had a module that generates a hidden error message on a number of requests, we are waiting for a fix from the vendor, but there are higher than average errors on this site.
Following my standard best practices we were managing the site of the Event Log and the overall database was just under 50mb in size. However, on disk the site was 20Gb. Using Tree Size I tracked down 19.56Gb of that data to be in the /portals/_default/logs folder. Come to find out this is the storage location for the Log4Net logs, all of the items that were regularly cleared from the EventLog still existed on disk. Simply deleting the old files and we were done.
IIS Logs
If you are managing the server yourself you also have IIS logs that you want to be sure are not going rogue. I'm not going to spend much time on this as the exact process of cleaning them is dependent upon your environment. The key is that IIS logs will contain a history of each request to the site, another common killer of disk space. You want to make sure that the server is configured to manage the retention of these files.
Fixing Log4Net?
So, how do we keep DNN and Log4Net under control? Sadly, at this point in time, there isn't really any true "automated" solution for this issue. Log4Net supposedly has a configuration item for "Rolling File Limit", however, this option does not work with daily files. This leaves us to manually delete files. I have reported this to DotNetNuke corporation and it has been logged as a feature request to get this resolved.
As a solution, I have a member of my team working on a module that will provide a solution to this issue. We are working to try to get something rolled out this week. I'll post an update when it happens.
Updated Fix - Brute Force Approach
There is one additional solution that you could use to control the Log4Net logs. This is to simply turn them off. Not the best solution, but it 100% prevents any issues with this happening. You can do this by changing the value for the "Level" XML element in the DotNetNuke.Log4Net.config to "Off" instead of the default "Error" value. This will disable all future logs. Not ideal, but a great solution for the time being. Special thanks to Brandon Haynes for this idea!
Conclusion
If you are running a DotNetNuke based site, I strongly recommend taking a look at your site, its configuration and the overall size of the site. It is very possible that you have a lot of hidden disk usage. Obviously the best solution overall is to ensure that recurring errors are resolved, but those issues can take time and they shouldn’t cause your site to go down due to logging. I hope this helped, feel free to share comments below.