Configuration

« Back to User Guide

$Id: configuration.html 23018 2015-04-29 17:50:20Z jmfee $
$URL: https://ghttrac.cr.usgs.gov/websvn/ProductDistribution/trunk/etc/documentation/userguide/configuration.html $

Navigation

Format

The configuration file uses an INI style syntax for properties. Any lines that begin with a semi-colon are treated as comments and ignored.

	globalproperty1 = globalvalue1
	globalproperty2 = globalvalue2

	[section1]
	section1property1 = section1value1

	[section2]
	section2property1 = section2value1

Configured Objects

Each INI section represents an Object that is loaded by the application. The section name is used as an object name

The type property in a section determines what type of object this section will be loaded as. The properties that each object requires differ, and are defined elsewhere on this page.

The properties in this discussion refers to configuration of objects such as listeners and receivers. This is not to be confused with the configuration of properties for products, such as the "magnitude-type" property of the origin product.

Configured Object Example

	listeners = listener1, listener2, listener3

	...

	[listener1]
	type = gov.usgs.earthquake.distribution.ExternalNotificationListener
	command = /path/to/executable
	storage = storageX

	[storageX]
	type = gov.usgs.earthquake.distribution.FileProductStorage
	directory = /path/to/storage/directory
Explanation

When ProductClient.jar is run with the --receive argument, it checks for the global property listeners . It finds the name listener1 in the list of listeners, and next looks for a matching configuration section named listener1 .

It finds the section listener1 , sees it is an ExternalNotificationListener, and continues loading. The ExternalNotificationListener type of object requires a command property, which is the path to a executable program to run, and a storage property which refers to another configuration object.

In this case, the storage property refers to a configuration object named storageX . The loading process finds the section storageX , sees it is a FileProductStorage, and continues loading. The FileProductStorage requires a directory property, which is a filesystem path to a directory.

At this point, the listener listener1 has been loaded, and the application loads any other configured objects before running.

Product Client Configuration

The product client includes an embedded configuration with defaults for sending and receiving products. User configuration is combined with the embedded configuration, and user settings override any defaults.

By default the product client tries to load the user configuration from a file named 'config.ini' in the current working directory. You can override this default user configuration location using the "--configFile=<configFile>" argument.

Once again, product client configurations are not to be confused with the properties of products, which is a different subject altogether.

Global Properties

logdirectory
A directory where log files are created. The default value is "log". Log files rotate once per day and are named "ProductClient_YYYYMMDD.log".
NOTE: this replaces the "logfile" property.
logformat
format for logs. Possible values are "pdl" (default), "simple", and "xml".
loglevel
How verbose of logging. Default is INFO. Possible values are SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST.
redirectconsole
Whether to redirect console output (STDOUT, STDERR) to the log file. Default is false.
senders
A comma delimited list of section names to load as senders. Items in this list must implement the ProductSender interface. This property is used with the --send command line argument of ProductClient.jar
receivers
A comma delimited list of section names to load as receivers. Items in this list must implement the NotificationReceiver interface. This property is used with the --receive command line argument of ProductClient.jar
listeners
Listeners are used by Hubs and Clients. A comma delimited list of section names to load as listeners. Items in this list must implement the NotificationListener interface. This property is used with the --receive command line argument of ProductClient.jar
enableTracker
Whether or not to send tracker updates for sent and received products. Default is disabled (false).
trackerURL
Default tracker URL assigned to products when sending. It is recommended this be http://ehppdl1.cr.usgs.gov/tracker/
enableZabbix
Whether or not to enable zabbix monitoring. Default is disabled (false).
zabbixPort
Override the default port (10052) when using zabbix monitoring.

Distribution Components

NotificationIndex

A notification index is used by Hubs and Clients to track available products.

gov.usgs.earthquake.distribution.JDBCNotificationIndex
indexfile
The SQLite database file. Default is "pd_index.db".

NotificationListener

Listeners are used by Hubs and Clients

includeTypes
(Optional) A comma delimited list of product types to process. When present, only types in this list are processed.
excludeTypes
(Optional) A comma delmited list of product types NOT to process. When present, types in this list are NOT processed.
includeSources
(Optional) A comma separated list of product types to include. If this list is defined it constitutes an "allow" list, and only types in this list are allowed.
excludeSources
(Optional) A comma separated list of product types to exclude. If this list is defined it constitutes a "block" list, and types in this list are not allowed.
attemptCount
maxTries
(Optional, Default 1) Number of times to attempt delivery of each notification, if the listener throws a ContinuableListenerException during onNotification(). A value <= 1 means do not re-attempt.
timeout
(Optional, Default 0) Number of milliseconds before thread running onNotification() is interrupted. A value <= 0 means never interrupt.
retryDelay
(Optional, Default 300000) Number of milliseconds to wait before attempting redelivery of a notification after a listener throws a ContinuableListenerException during onNotification(). A value < = 0 means no delay. Used with maxTries.
includeScenarios
(Optional, Default false) Flag to indicate scenario products should be accepted. Set to “true” or “yes” to enable.
includeActuals
(Optional, Default true) Flag to indicate actual products should be accepted. Set to “false” or “no” to disable.
includeTests
(Optional, Default false) Flag to indicate test products should be accepted. Set to “true” or “yes” to enable.
includeInternals
(Optional, Default false) Flag to indicate internal products should be accepted. Set to “true” or “yes” to enable.
includeDevelopments
(Optional, Default false) Flag to indicate development products should be accepted. Set to “true” or “yes” to enable.
listenerIndex
Name of a configured NotificationIndex object.

Alternatively, a property listenerIndexFile may be used to create a SQLite JDBCNotificationIndex without further configuration. This should not be used when the index is shared.

includePaths
NOTE: this property does not filter product downloads, only product processing. A comma separated list of content paths to include. Only products with a content path in this list will be processed.
excludePaths
NOTE: this property does not filter product downloads, only product processing. A comma separated list of content paths to exclude. Products with a content path in this list will not be processed.

Listener Types

gov.usgs.earthquake.distribution.ExternalNotificationListener
This represents a command/process to run each time a product is received.
storage
Name of a configured ProductStorage. Generally, a ProductStorage should not be shared between a receiver and listener. Products are extracted into this storage before executing command, with a --directory argument representing the stored location.

Alternatively, a property storageDirectory may be used to create a FileProductStorage without further configuration.

command
A path to an executable file and arguments
timeout
(Optional, default=100000) Timeout in milliseconds for an execution of command. If command doesn't finish executing within the timeout, the process is terminated. When timeout is set to zero (0), commands are never terminated.
gov.usgs.earthquake.distribution.EIDSNotificationSender
Stores products using a URL Product Storage, and sends EIDS notifications with links to the products.
listenerIndex
Name of a configured NotificationIndex. This is used to track notifications that are sent for later automatic cleanup. Generally, a NotificationIndex should not be shared by multiple EIDSNotificationSenders unless they are sending different types of products that do not overlap.

Alternatively, a property listenerIndexFile may be used to create a SQLite JDBCNotificationIndex without further configuration. This should not be use when the index is shared.

storage
Name of a configured URLProductStorage.
storageAge
(Optional, default=3600000=1 hour) Maximum time in milliseconds for products to be kept before automatic cleanup.
cleanupInterval
(Optional, default=3600000=1 hour) How often, in milliseconds, for automatic cleanup to run. When interval is set to zero (0), no automatic cleanup is run.
serverHost
EIDS server hostname, used to attempt sending notifications to EIDS via CORBA.
serverPort
EIDS server port, used to attempt sending notifications to EIDS via CORBA
serverPolldir
(Optional, default=polldir) EIDS server polldir, notifications are output as files when unable to send via CORBA.
gov.usgs.earthquake.distribution.RelayProductListener
Forward any received products to another SocketProductReceiver. For configuration purposes, a RelayProductListener is treated as a gov.usgs.earthquake.distribution.SocketProductSender.
gov.usgs.earthquake.indexer.Indexer
See Indexer Configuration
gov.usgs.earthquake.distribution.HeartbeatListener
Collects status information from various components in one client installation, and writes it to a file whenever a heartbeat product is received.

The primary purpose of this listener is monitoring. Products are usually event driven, and at times are far apart. Each PDL hub will generally send a heartbeat product every 5 minutes, so there is a steady flow of information and problems can be detected early.

heartbeatFilename
(Optional, default=heartbeat.dat) File where heartbeat information is written. The file contents are a JSON object. The object keys correspond to configuration file section names, and values are objects with one or more statuses that are timestamped.
heartbeatTimeout
(Optional, default=0) Milliseconds. When zero, disables automatic removal of status information (which is not usually needed). Used in combination with cleanupInterval to remove old heartbeat status information after timeout.
gov.usgs.earthquake.distribution.ContentListener
A listener that listens for a specific content path.

This is intended for users who wish to output specific pieces of product content, such as "quakeml.xml", for products that otherwise meet their configured NotificationListener criteria.

includePaths
Comma separate list of content paths to output when received. (reuses DefaultNotificationListener property).
outputDirectory
Directory where content is output.
tempDirectory
(Optional, default=system temp dir) Directory where content is written before being moved to outputDirectory
outputFormat
(Optional, default=SOURCE_TYPE_CODE_UPDATETIME_PATH) Format for generated filenames in output directory. This format string is filled in with these attributes using string replacement:
SOURCE
Product Source
TYPE
Product Type
CODE
Product Code
UPDATETIME
Product Update Time
PATH
Content Path

NotificationReceiver

Receivers are used by Hubs and Clients.

index
Name of a configured NotificationIndex. Generally one NotificationIndex may be shared by all receivers. If a NotificationIndex is shared, the ProductStorage must also be shared.

Alternatively, a property indexFile may be used to create a SQLite JDBCNotificationIndex without further configuration. This should not be use when the index is shared.

storage
Name of a configured ProductStorage. Generally one ProductStorage may be shared by all receivers. If a ProductStorage is shared, the NotificationIndex must also be shared.

Alternatively, a property storageDirectory may be used to create a FileProductStorage without further configuration. This should not be used when the storage is sharted.

storageage
How long to store products in milliseconds. Default is 300000 milliseconds = 5 minutes.
cleanupInterval
How often to check for products older than storageage milliseconds. Default is 300000 milliseconds = 5 minutes.
connectTimeout
Number of milliseconds allowed before connection attempts timeout, when downloading products.
readTimeout
Number of milliseconds allowed before read attempts timeout, when downloading products. And, for SocketProductReceiver, when accepting products via socket.
listenerNotifier
executor
(default) Executor uses one queue per listener and notifications are processed in the order they are received.
roundrobin
Round robin uses one queue per listener for each product source+type and takes one product from the front of each queue. This prevents a bulk sender from slowing processing of other products.

Receiver Types

gov.usgs.earthquake.distribution.EIDSNotificationReceiver
This is used primarily by clients to receive notifications from EIDS.
serverHost
The IP address or hostname where an EIDS Server with notifications is running.
serverPort
The port where an EIDS Server with notifications is running.
alternateServers
(Optional) A comma separated list of host:port that should be used as alternate servers when serverHost is unreachable.
gov.usgs.earthquake.distribution.SocketProductReceiver
This is used primarily by hubs to receive products directly.
port
(Optional) Default is 11235. The port used to accept connections.
sizeLimit
(Optional) Default is -1. When greater than zero, limit incoming products to this many bytes. NOTE: this limit is based on the number of bytes transferred instead of actual product content bytes, and also applies to any metadata or formatoverhead.
threads
(Optional) Default is 10. The size of the thread pool used to process accepted connections. Connections beyond this number are queued and processed in the order they are accepted.

ProductSender

Senders are used by Product Creators to send a product to a Hub.

gov.usgs.earthquake.distribution.SocketProductSender

See Sending products for suggestions on configuring compression and format settings

host
The IP address or hostname where a SocketProductReceiver is listening.
port
The port where a SocketProductReceiver is listening.
connectTimeout
(Optional) Default is 15000. Milliseconds to wait for a connection to succeed.
readTimeout
(Optional) Default is 15000. Milliseconds to wait for data before timing out.
writeTimeout
(Optional) Default is -1. Milliseconds to wait between writes before timing out.

NOTE: Java does not natively support write timeouts for sockets in blocking mode. This option uses a separate thread to monitor socket writes, and closes the socket if a write does not occur within this time.

binaryFormat
(Optional) Default is false. Send products using binary format (true) or xml format (false). Binary format avoids base64 encoding overhead, and reduces bytes tranferred by ~25%.
enableDeflate
(Optional) Default is true. Enable deflate compression during transfer (true) or disable (false). Deflate compression can reduce content size by 90+%, but is also CPU intensive.
deflateLevel
(Optional) Default is 1 (best speed). Only used when enableDeflate = true. A number between 1 (best speed) and 9 (best compression). -1 lets the library decide the best level, but currently implies 6.

ProductStorage

Storage is used by Hubs and Clients to store received products.

gov.usgs.earthquake.distribution.FileProductStorage
directory
The directory where products are stored. Default is "storage".
verifySignatures
Whether or not to verify signatures when storing products. Should be one of:
off
(Default) Do not verify signatures
test
Check signatures, but do not reject products with invalid signatures.
<anything else>
Verify signatures, and reject products with invalid signatures.
keychain
Used when verifying signatures. A comma delimited list of sections to load as ProductKeys.
keychainFile
An alternative to keychain where keychain is managed in a separate config file. File should contain a global (before any sections) property named "keychain", which is a comma separated list of sections to load as ProductKeys.
listeners
Comma-separated list of StorageListeners to notify when a product is added to or removed from the storage.
useHashes
A flag indicating if the generated directory structure for a product should use a simple hash of the product. Setting this to “true” will help on ext3 file systems. Default ”false”
legacyStorages
Optional. Comma separated list of "legacy" ProductStorages. Used when making incompatible changes to an existing storage. ProductStorages in this list are used when the primary storage (i.e. that has the legacyStorages property configured) does not contain a product, and are only used to access or remove products; never store newly added products.
gov.usgs.earthquake.distribution.URLProductStorage

A URLProductStorage is configured just like a FileProductStorage, with the following additional property

url
A fully qualified (including http://) base url corresponding to directory .

To map a directory onto a url in Apache, use an Alias.

StorageListener

Listeners are notified when products are added-to, or removed from the ProductStorage.

gov.usgs.earthquake.distribution.DefaultStorageListener
Provides simple dispatch functionality to allow sub-classes to listen optionally to only onProductStored or onProductRemoved if desired.
gov.usgs.earthquake.distribution.ReplicationStorageListener
A replication storage listener that will replicate the local ProductStorage to a directory on a remote server. Note, this listener will only respond to notifications from a gov.usgs.earthquake.distribution.ProductFileStorage.
archiveSync
“true” or “false” Flag indicating if the replication should be done using the archive flag on the replication command.
rsync
The name of the replication command to use. (Generally “rsync”). Should use the fully-qualified path name.
maxTries
Integer. Number of times to try the sync if it fails.
timeout
Integer. How long to wait (in milliseconds) before assuming the replication process is hung.
targetHosts
Comma-separated list of target hosts to which the storage should be replicated. Format for this value is “user@host:path”. Care should be taken when setting this value. The remote user must have a public key set up on the remote host. The remote path must already exist.

ProductKey

Keys are used to verify signed products, and more specifically by FileProductStorage.

gov.usgs.earthquake.distribution.ProductKey
key
An OpenSSH format public key (similar to .ssh/authorized_keys).
types
A comma delimited list of product types for which this key is valid. Default = all.
sources
A comma delimited list of product sources for which this key is valid. Default = all.

Indexer Components

Indexer

The indexer is a specialized NotificationListener that associates related products into events.

gov.usgs.earthquake.indexer.Indexer
Builds and index of received products. Associates groups of related products into events.
associator
Name of a configured Associator.
archivePolicy
A comma delimited list of configured ArchivePolicy objects.
archiveInterval
Interval, in milliseconds, between executing archive policies.
storage
Name of a configured ProductStorage.

Alternatively, a property storageDirectory may be used to create a FileProductStorage without further configuration.

index
Name of a configured gov.usgs.earthquake.indexer.ProductIndex.

Alternatively, a property indexFile may be used to create a SQLite JDBCNotificationIndex without further configuration.

modules
A comma delimited list of configured IndexerModule .
listeners
A comma delimited list of configured IndexerListener
enableSearch
(Default false). Whether or not to listen for searches using a socket.
searchPort
(Default 11236) Which port to bind to listen for searches. Only used when enableSearch=true.
searchThreads
(Default 5) How large a thread pool to use for searches. 1 thread is used for each connection, so 5 allows 5 concurrent searches.
associateUsingCurrentProducts
(Default false) Only consider "current" versions of products when associating new products to events. This improves association performance for events with many products.

ArchivePolicy

An archive policy is used to generate a ProductIndexQuery . The query is then used to search the ProductIndex . Products and/or events that are returned using this query are archived (generally meaning removed) from the index.

gov.usgs.earthquake.indexer.ArchivePolicy
Archive events in the index based on their preferred properties. All associated products are also archived when an event is archived.
minAge Deprecated See: minEventAge
Age in milliseconds. Products younger than (occurring since) this time will match the generated product index query. (You generally don't want to set this parameter).
maxAge Deprecated See: maxEventAge
Age in milliseconds. Products older than (occurring before) this time will match the generated product index query.
minEventAge
Event age in milliseconds. Events older than (occurring before) this time will match the generated product index query. If both the minEventAge and maxEventTime are specified, the minEventAge is ignored and the maxEventTime is used. What’s the difference between Age and Time?
maxEventAge
Event age in milliseconds. Events newer than (occurring since) this time will match the generated product index query. (You generally do not want to set this parameter without a corresponding minEventAge). If both the maxEventAge and minEventTime are specified, the maxEventAge is ignored and the minEventTime is used. What’s the difference between Age and Time?
minEventTime
Event time in milliseconds. Events occurring after this time will match the generated product index query. (You generally do not want to set this parameter without a corresponding maxEventTime). If both the maxEventAge and minEventTime are specified, the maxEventAge is ignored and the minEventTime is used. What’s the difference between Age and Time? .

minEventTime can also be configured using ISO8601 formatted dates, e.g. 2012-12-11T01:02:03.123Z

maxEventTime
Event time in milliseconds. Events occurring before this time will match the generated product index query. If both the minEventAge and maxEventTime are specified, the minEventAge is ignored and the maxEventTime is used. What’s the difference between Age and Time?

maxEventTime can also be configured using ISO8601 formatted dates, e.g. 2012-12-11T01:02:03.123Z

minDepth
Decimal depth value in kilometers. Positive values indicate positions below the earth's surface. Products with depth greater (deeper) than this value will match the generated product index query.
maxDepth
Decimal depth value in kilometers. Positive values indicate positions below the earth's surface. Products with depth less (shallower) than this value will match the generated product index query.
minLat
Decimal degrees minimum product latitude. Products with latitude north of this value will match the generated product index query.
maxLat
Decimal degrees maximum product latitude. Products with latitude south of this value will match the generated product index query.
minLng
Decimal degrees minimum product longitude. Products with latitude east of this value will match the generated product index query.
maxLng
Decimal degrees maximum product longitude. Products with latitude west of this value will match the generated product index query.
minMag
Decimal value for minimum magnitude. Products with magnitude greater than this value will match the generated product index query.
maxMag
Decimal value for maximum magnitude. Products with magnitude less than this value will match the generated product index query.
eventSource
The preferred source of the event to archive. When omitted all sources are included.
gov.usgs.earthquake.indexer.ProductArchivePolicy
Archive products in the index based on their properties. Any properties from regular ArchivePolicy may be used, and at least one must be used. When they don't seem meaningful it is recommended to use "maxAge=1".
minProductAge
Product age in milliseconds; computed using product update time. Products older (occurring before) this time will match the generated product index query. If both the minProductAge and maxProductTime are specified, the minProductAge is ignored and the maxProductTime is used. What’s the difference between Age and Time?
maxProductAge
Product age in milliseconds; computed using product update time. Products newer (occurring since) this time will match the generated product index query. You generally do not want to specify this parameter without also specifying a corresponding minProductAge. If both the maxProductAge and minProductTime are specified, the maxProductAge is ignored and the minProductTime is used. What’s the difference between Age and Time?
minProductTime
Product update time in milliseconds. Products updated after this time will match the generated product index query. You generally do not want to specify this parameter without also specifying a corresponding maxProductTime. If both the maxProductAge and minProductTime are specified, the maxProductAge is ignored and the minProductTime is used. What’s the difference between Age and Time?

minProductTime can also be configured using ISO8601 formatted dates, e.g. 2012-12-11T01:02:03.123Z

maxProductTime
Product update time in milliseconds. Products updated before this time will match the generated product index query. If both the minProductAge and maxProductTime are specified, the minProductAge is ignored and the maxProductTime is used. What’s the difference between Age and Time?

maxProductTime can also be configured using ISO8601 formatted dates, e.g. 2012-12-11T01:02:03.123Z

productType
The type of product to archive. When omitted all types are included.
productSource
The source of product to archive. When omitted all sources are included.
onlySuperseded
Only remove superseded versions of products. When omitted all versions are included.
onlyUnassociated
Only remove products not associated to an event.
productStatus
Only remove products whose status matches this status.

IndexerListener

gov.usgs.earthquake.indexer.ExternalIndexerListener

Uses the commandLine API to execute an external command when the index is updated. Provides additional arguments that describe how products are related to events in the index.

Supports the same configuration parameters as ExternalNotificationListener, and in addition:

processPreferredOnly
(Optional, default=false) Whether to ignore non-preferred products. All unassociated products are considered preferred, and may be filtered separately using processUnassociated.
processUnassociated
(Optional, default=true) Whether to ignore unassociated products. Products that are currently unassociated may later be associated as other information arrives.
processOnlyWhenEventChanged
(Optional, default=false) Whether to ignore updates that do not change an events preferred location (time, latitude, longitude, depth, magnitude).
ignoreArchive
(Optional, Default = true) Whether or not to ignore EVENT_ARCHIVED and PRODUCT_ARCHIVED indexer events. Value values are "true" and "false".
autoArchive
(Optional, Default = true) Whether or not to remove products from the configured ProductStorage in response to indexer PRODUCT_ARCHIVED and EVENT_ARCHIVED events. This is independent of ignoreArchive configuration.

IndexerModule

gov.usgs.earthquake.shakemap.ShakeMapIndexerModule
Provides support for shakemap type products. There are no configuration parameters for this module.
gov.usgs.earthquake.momenttensor.MTIndexerModule
Provides support for moment-tensor and adjusts the weight of a given product. There are no configuration parameters for this module.

ProductIndex

gov.usgs.earthquake.indexer.JDBCProductIndex
indexfile
The filename to use for a SQLite database. If the file does not exist, one is created. Only required if the driver parameter is set to the default value "org.sqlite.JDBC".
driver
The JDBC database driver class to use. This allows the JDBCProductIndex to use any type of relational database, provided a suitable driver is available. Default value is "org.sqlite.JDBC". For a MySQL database, you should use "com.mysql.jdbc.Driver". See Configuring ProductIndex to Use MySQL.
url
JDBC connection URL. Usually, this url is prefixed with the database type and includes a host, schema to use, username and password. For example to connect to a MySQL database called "productIndex" on localhost with username "foo" and password "bar", the URL would look like: "jdbc:mysql://127.0.0.1/productIndex?user=foo&password=bar". This parameter is ignored for SQLite databases.