Folder Watcher Or Hot Folders
Linux First - CenOS 5 and 6.
How many times have you wanted to execute an action or run a script or an application when a file appears in a folder?
Called folder watchers or hot folders, these are very useful for batch jobs or jobs that run in serial. When a file is uploaded to a folder, format that file in xxx fashion and deposit it into yyyy destination or any variation of that.
In Linux, I think it is called the inotify interace and a great implementation is of this is at http://inotify.aiken.cz/?section=common&page=home&lang=en
If you need to watch folders in Linux or other Unix variants, you must look at this.
We implemented a simple solution in our organisation to encode videos from one format to many others by using this.
It was a two step process.
Files would come into a folder over FTP.
Once they landed in the folder, the iNotify interface would run the a script that called ffmpeg to convert videos and would deposit the result into the respective folders - one folder per format - we had two different fomats in their separate format specific folders for the test.
We configured inotify for the destination (converted) video clip folders. When the files converted by ffmpeg would get saved in the new folders, another event would get fired by the inotify interface and would run an command line FTP command to send the files to other servers. The script would also make some entries in a MySql database that acted as an media management database for us.
We could easily drop more than 30 clips at a time and the event notification interface worked like a charm.
We tried with files of a few KB upto 300MBytes - everything worked like a dream.
Windows XP+
On Windows, there are many commercial products, but we chose to write our own folder watcher in VB.Net with the FileSystemWatcher class.
In Visual Studio, under the Windows section of the New Project Dialog box, you should find a Windows Service option. Select that, give your service a name and Visual Studio, helpfully, creates a skeleton where filling the blanks in is super easy.
OnStart
This Sub is called when the windows service starts - when you turn on the computer or restart the computer or service.
Here we register one folder to be "watched" by the operating system and then tell the service runtime which "sub"s to execute for two events - when a new file is created or when an existing file is updated. In our case, both events require the same function to be called but you could have different "sub"s to be called.
Protected Overrides Sub OnStart(ByVal args() As String)
' Add code here to start your service. This method should set things
' in motion so your service can do its work.
Try
watchfolder = New System.IO.FileSystemWatcher
watchfolder.Path = getRegString("FolderToWatch")
watchfolder.Filter = "*.htm"
watchfolder.NotifyFilter = IO.NotifyFilters.DirectoryName
watchfolder.NotifyFilter = watchfolder.NotifyFilter Or IO.NotifyFilters.FileName Or NotifyFilters.LastWrite
watchfolder.NotifyFilter = watchfolder.NotifyFilter Or IO.NotifyFilters.Attributes
AddHandler watchfolder.Changed, AddressOf weatherUpdate
AddHandler watchfolder.Created, AddressOf weatherUpdate
watchfolder.EnableRaisingEvents = True
Catch ex As Exception
WriteToEventLog("Could Not Start the service " & ex.Message, "Service1", EventLogEntryType.Error)
End Try
End Sub
Protected Overrides Sub OnStop()
watchfolder.EnableRaisingEvents = False
' Add code here to perform any tear-down necessary to stop your service.
End Sub
Linux First - CenOS 5 and 6.
Called folder watchers or hot folders, these are very useful for batch jobs or jobs that run in serial. When a file is uploaded to a folder, format that file in xxx fashion and deposit it into yyyy destination or any variation of that.
In Linux, I think it is called the inotify interace and a great implementation is of this is at http://inotify.aiken.cz/?section=common&page=home&lang=en
If you need to watch folders in Linux or other Unix variants, you must look at this.
We implemented a simple solution in our organisation to encode videos from one format to many others by using this.
It was a two step process.
Files would come into a folder over FTP.
Once they landed in the folder, the iNotify interface would run the a script that called ffmpeg to convert videos and would deposit the result into the respective folders - one folder per format - we had two different fomats in their separate format specific folders for the test.
We configured inotify for the destination (converted) video clip folders. When the files converted by ffmpeg would get saved in the new folders, another event would get fired by the inotify interface and would run an command line FTP command to send the files to other servers. The script would also make some entries in a MySql database that acted as an media management database for us.
We could easily drop more than 30 clips at a time and the event notification interface worked like a charm.
We tried with files of a few KB upto 300MBytes - everything worked like a dream.
Windows XP+
On Windows, there are many commercial products, but we chose to write our own folder watcher in VB.Net with the FileSystemWatcher class.
In Visual Studio, under the Windows section of the New Project Dialog box, you should find a Windows Service option. Select that, give your service a name and Visual Studio, helpfully, creates a skeleton where filling the blanks in is super easy.
OnStart
This Sub is called when the windows service starts - when you turn on the computer or restart the computer or service.
Here we register one folder to be "watched" by the operating system and then tell the service runtime which "sub"s to execute for two events - when a new file is created or when an existing file is updated. In our case, both events require the same function to be called but you could have different "sub"s to be called.
Protected Overrides Sub OnStart(ByVal args() As String)
' Add code here to start your service. This method should set things
' in motion so your service can do its work.
Try
watchfolder = New System.IO.FileSystemWatcher
watchfolder.Path = getRegString("FolderToWatch")
watchfolder.Filter = "*.htm"
watchfolder.NotifyFilter = IO.NotifyFilters.DirectoryName
watchfolder.NotifyFilter = watchfolder.NotifyFilter Or IO.NotifyFilters.FileName Or NotifyFilters.LastWrite
watchfolder.NotifyFilter = watchfolder.NotifyFilter Or IO.NotifyFilters.Attributes
AddHandler watchfolder.Changed, AddressOf weatherUpdate
AddHandler watchfolder.Created, AddressOf weatherUpdate
watchfolder.EnableRaisingEvents = True
Catch ex As Exception
WriteToEventLog("Could Not Start the service " & ex.Message, "Service1", EventLogEntryType.Error)
End Try
End Sub
OnStop
Magically, this is the Sub called when the service is stopped by you or the operating system. Put any clean up code in here.
watchfolder.EnableRaisingEvents = False
' Add code here to perform any tear-down necessary to stop your service.
End Sub
The Event Handler
This is the "sub" that actually does the work - as registered with the AddHandler command in the OnStart Sub above.
Private Sub weatherUpdate(ByVal source As Object, ByVal e As System.IO.FileSystemEventArgs)
'Update a database
'Convert the file
'Send the file by FTP to someplace else
'Send an e-mail
'Or whatever else you want to do with the file
End Sub
Hope this helps someone
Cheers
No comments:
Post a Comment