Quick Tip: Out-File and IOPs – Position matters
Take a look at the following two examples. Both commands gets a list of log files, selects a subset of each file properties and logs them to a file.
Get-ChildItem -Filter *.log -Recurse | ForEach-Object {
$_ | Select-Object Name,Length,FullName | Out-File -FilePath D:\LogFilesReport.txt -Append
}
Get-ChildItem -Filter *.log -Recurse | ForEach-Object {
$_ | Select-Object Name,Length,FullName
} | Out-File -FilePath D:\LogFilesReport.txt –Append
At first glance both commands looks very similar but under the surface one of them is very inefficient and is generating A LOT of IO activity. Let’s assume that the results from the Get-ChildItem cmdlet is 1000 objects. To test this I used Sysinternals Process Monitor, ran each command and exported the results to a csv file. Let’s look at the numbers.
PS > $cmd1 = Import-Csv .\cmd1.CSV
PS > $cmd2 = Import-Csv .\cmd2.CSV
PS > ($cmd1 | Measure-Object).Count
5276
PS > ($cmd2|
Measure-Object).Count
1203
The first command generates 5276 IO operations vs 1203 of the second one, that’s a HUGE difference. Here’s some more info from the csv files:
PS > $cmd1 | Group-Object Operation -NoElement | Sort-Object Count -Descending
Count Name
----- ----
1051 QueryOpen
1026 CreateFile
1023 CloseFile
1000 QueryStandardInformati...
1000 WriteFile
78 QuerySecurityFile
54 QueryDirectory
20 QueryBasicInformationFile
19 QueryNameInformationFile
2 QueryAttributeTagFile
2 SetDispositionInformat...
1 SetRenameInformationFile
PS > $cmd2 | Group-Object Operation –NoElement | Sort-Object Count -Descending
Count Name
----- ----
1078 WriteFile
42 QuerySecurityFile
18 CreateFile
15 CloseFile
14 QueryOpen
11 QueryBasicInformationFile
10 QueryNameInformationFile
9 QueryDirectory
2 QueryAttributeTagFile
2 SetDispositionInformat...
1 SetRenameInformationFile
1 QueryStandardInformati...
Based on the above you clearly want to pipe your result to the Out-File cmdlet (or any other file related cmdlets) after the ForEach-Object cmdlet. Doing so will improve the performance of your scripts and will give your disk spindle some rest :)