I often have the need to process large text files. PowerShell sucks at this. I was forced to create a PowerShell library that uses the default .net routines to handle large files. It is many times faster.
I have all the routines in a library file-library.ps1 which I dot source to use. This way all the routines are available.
ps> . ./file-library.ps1
The routines available in the libarary are:
- out-UTF8 -- outputs file in UTF8 no BOM -- always appends.
- append-file -- appends one file to another
- split-file -- splits a file by line into smaller sizes
- count-lines -- counts the lines in a file
- sort-file -- sort the lines in a file
- unique-file -- remove duplicates from a file
- sortunique-file -- sort and remove duplicate lines from a file
- get-fileEncoding -- reads the bom of a file to get the encoding