Monitor

Top  Previous  Next

 

The Monitor step adds local files to a job's file set.
 

Friendly Name

INI Variable

Required?

Description

File Filter

specification

optional

specification for source file to match

File Path

path

required

directory to monitor for a source file

Timeout

timeout

optional

Timeout in minutes looking for source file and send error notification

Leave Files on Host

persistent_source

optional

leave processed files in source. default: false

Include Directories

include_directories

optional

import directories matching specification. default: false

File Order

file_order_first

optional

Start processing files starting from the oldest or newest files. Especially significant when multiple_files is something other than all. Possible values are "oldest" or "newest", without the quotes.

Multiple Files

multiple_files

optional

Number of files matching spec to import. Any is zero or more making the absence of a file for this step to import not an error. Possible values are "#", "all", or "any", defaulting to "all", without the quotes.

Minimum Age

older_than

optional

A negative, relative value with a resolution of seconds, minutes, hours, days, or months; for example: "-1 month" or "-7 days", without the quotes.

N/A

newer_than

optional

A relative value with a resolution of seconds, minutes, hours, days, or months; for example: "1 month" or "7 days".

N/A

errors_directory

optional

put files that error out here and send notification

N/A

archive_directory

optional

put files that succeed here

 

The Monitor step defines both the beginning and ending processing for jobs that operate on local files. Most framework jobs begin with either a Monitor step or a Pull step.

 

Beginning Processing

The Monitor step searches the source path for file names that match the configured specification. Matching files are added to the job's file set. Subsequent job steps operate on this file set. If no matching files are found within the timeout period this steps returns an error and the remaining job steps are not processed.

 

If the value of the multiple_files option is a number then exactly that number of files are added to the file set. If the expected number of files are not available the step will error out and job processing will halt. If the value is "all" then all matching files are added to the file set. If no matching files are found the step reports an error and processing stops. If the value is "any" then all matching files are added and the absence of any matching files is not treated as an error and processing continues to the next step.

 

Ending Processing

After all the subsequent job steps have completed, control returns to the Monitor step and the files are removed from the current file set. If the job was successful the processed files are deleted unless the archive_directory and/or persistent_source options are configured. The archive_directory option is used to store successfully processed files and the persistent_source option causes the Framework to leave the source files in their original place and track their processing status in a database. The errors_directory option is used to keep files when job processing fails. A configuration error is returned if the end-of-processing move or delete operation fails.

 

The include_directories option does not work with the persistent_source option.

 

 

Example Usage:
step_monitor