We recently had to migrate data in the form of some flat-files between two systems. As is often the case some transformation of the flat-file was required before loading into the destination system.
The transformed file would be moved into an output folder triggering a migration process, via a simple .exe, that wrote the flat-file contents to our destination system.
The first part was accepting a few parameters into the script of which the main thing was declaring a script-block parameter that would be the action we execute in response to finding a file matching our filter criteria.
For the script-block parameter which we named
$FileFoundAction you would then pass in something like this:
$args would be the Full Path to the file that has just been dumped in the folder we are watching.
A while block would then run until a break was detected. Inside the block we’d poll our directory for files matching
$FilePattern regex. Waiting for an equivalent of the
$PollingIntervalMin between checks to the directory.
The more complicated bit however was invoking the script-block and not blocking the polling operation. In this case, we settled for invoking the script-block As-Job and running it in another PSSession.
This meant though that we’d have to run the script as admin because of the
New-PSSession we had created and then check our jobs using the
Needless to say this created a wicked problem because we were going to dump a couple hundred files maybe a thousand thereby creating a lot of sessions and jobs that we would have to monitor.
We also observed that memory usage really shot up for a few sessions. As an aside I don’t think this is a PowerShell issue rather our script-block invoked code that loaded some large objects into memory.
In the end this would work for what it’s designed to do which is looking for new files added to a folder and triggering a simple action. But the solution wouldn’t scale well.
After some digging around we used another solution which came out the PowerShell box as far back as version 3.1 actually. Check Part 2 of this Post for this solution.
Full code for the script used here is available at this link: FileWatcher