Product Distribution User Guide

$Id: index.html 19873 2013-07-30 18:55:34Z jmfee $
$URL: https://ghttrac.cr.usgs.gov/websvn/ProductDistribution/trunk/etc/documentation/userguide/index.html $

Navigation


System Overview

Products and Notifications

Product

A Product is a group of files or content. Each product is uniquely identified by source, type, code, and update time. These key attributes are described as follows:

Versions

Products with the same source, type, and code, but different update times are considered different versions of the same product. Later updates to the same product replace previous versions, where older versions are retained in the database.

Status

A product status may have different meanings depending on product type. "DELETE" always has the same meaning: that a product has been cancelled. Deleting a product is the same as sending a product, except the status is now "DELETE" and contents are optional. This version of the product is distributed the same way as any other update.

Properties

Properties are optional name/value pairs that describe the product. They are used to quickly identify characteristics of a product without needing to parse its content. Examples include PAGER "alertlevel", and origin "magnitude-type".

Links

Links are optional and are a URI with a relation that describes how the resource is related to the product. Because the link is a URI, it can be either a URN or URL. One example would be including a "background" related link for a product. Another example would be identifying a related product that was used as an input.

Notification

A Notification is a small message with Product's source, type, code, update time, URL for download, and URL expiration date.

Distribution

Product Distribution provides a standard solution for data distribution.

Data producers and consumers are loosely coupled. Producers broadcast a product, via redundant hubs, and all clients are notified it is available. Only clients that are interested download the full content for processing. This simplifies both sending and receiving data because all producers and consumers only need to communicate with the hubs. Because producers and consumers initiate connections to the hubs, most users do not need to reconfigure firewalls.

Producers use a private key to sign products when they are sent. Hubs verify this signature using a public key before distributing products to ensure only authorized producers are able to send products. Clients may optionally verify these signatures as well.

Clients can be configured to trigger processing when new products arrive, either using command line arguments or a Java API, eliminating polling latency. In many cases this has reduced processing time from minutes to seconds.

Notification Receivers

A NotificationReceiver receives Notifications for products and keeps a NotificationIndex of all received notifications. Whenever a notification is received, it is queued for processing by each NotificationListener. Each listener has a separate queue, and uses a separate thread to process queued notifications one at a time. This prevents a slow listener from blocking a fast listener.

When a listener processes a notification, it may request the product from the receiver. Any number of listeners may request the same product, but the receiver will only download once. Once the receiver has retrieved the product, the listener processes its contents.

Periodically, the receiver will remove expired notifications from its index, and downloaded products that are no longer needed. By default a receiver only keeps downloaded products for 15 minutes, just long enough for listeners to process, to minimize disk usage.

Notification Listeners

ExternalNotificationListener

ExternalNotificationListeners process products by executing an external command line process for each received product. The ExternalNotificationListener does not associate related products together, for that functionality use the Indexer.

Example ExternalNotificationListener command line arguments
/path/to/executable --directory=/Users/jmfee/Desktop/ProductClient/data/listener_example_storage/origin/nc71742550/nc/1330975200000 --type=origin --code=nc71742550 --source=nc --updateTime=2012-03-05T19:20:00.000Z --status=UPDATE --trackerURL=http://ehppdl1.cr.usgs.gov/tracker/ "--property-eventtime=2012-03-05T19:18:22.500Z" "--property-cube-magnitude-type=D" "--property-magnitude-type=Md" "--property-eids-feeder=ncss3.ncss-mp.cisn.org" "--property-azimuthal-gap=46.8" "--property-magnitude=1.8" "--property-eventsource=nc" "--property-eventsourcecode=71742550" "--property-eids-feeder-sequence=208382" "--property-location-method-class=Unknown" "--property-depth=2.4" "--property-version=1" "--property-magnitude-error=0.2" "--property-horizontal-error=0.2" "--property-num-phases-used=37" "--property-magnitude-num-stations-used=14" "--property-vertical-error=0.3" "--property-minimum-distance=0.00898315" "--property-longitude=-122.8205" "--property-latitude=38.8177" "--property-location-method-algorithm=B" "--property-cube-location-method=B" "--property-standard-error=0.06" "--property-review-status=AUTOMATIC" --signature=MCwCFCT2On3fJ6dydk+MIoPp8zZ3ChbAAhQY01euDYqi6xaOD660dbYIML8qKQ==

See Receiving Products for more information

Indexer

The Indexer associates related products into events. The indexer uses authoritative regions to choose preferred products when multiple sources contribute the same type of product for the same event. Every time the Indexer changes the index, it notifies IndexerListeners about the change.

Indexer Listeners

IndexerListeners are similar to NotificationListeners but listen to Indexer changes (which are triggered by products) instead of listening to notifications for products. Because the indexer keeps track of previously processed products, duplicate and non-preferred information can be filtered. Additionally, updates that do not change an event's location or magnitude can also be filtered.

Example ExternalIndexerListener command line arguments
/path/to/executable --directory=/Users/jmfee/Desktop/ProductClient/data/indexerlistener_storage/origin/nc71742550/nc/1330975200000 --type=origin --code=nc71742550 --source=nc --updateTime=2012-03-05T19:20:00.000Z --status=UPDATE --trackerURL=http://ehppdl1.cr.usgs.gov/tracker/ "--property-magnitude-type=Md" "--property-eventsourcecode=71742550" "--property-location-method-class=Unknown" "--property-version=1" "--property-horizontal-error=0.2" "--property-magnitude-num-stations-used=14" "--property-minimum-distance=0.00898315" "--property-longitude=-122.8205" "--property-cube-location-method=B" "--property-standard-error=0.06" "--property-eventtime=2012-03-05T19:18:22.500Z" "--property-cube-magnitude-type=D" "--property-eids-feeder=ncss3.ncss-mp.cisn.org" "--property-azimuthal-gap=46.8" "--property-eventsource=nc" "--property-magnitude=1.8" "--property-eids-feeder-sequence=208382" "--property-depth=2.4" "--property-magnitude-error=0.2" "--property-num-phases-used=37" "--property-vertical-error=0.3" "--property-latitude=38.8177" "--property-location-method-algorithm=B" "--property-review-status=AUTOMATIC" --action=EVENT_UPDATED --preferred-eventid=nc71742550 --preferred-eventsource=nc --preferred-eventsourcecode=71742550 --eventids=nc71742550 --preferred-magnitude=1.8 --preferred-latitude=38.8177 --preferred-longitude=-122.8205 --preferred-depth=2.4 --preferred-eventtime=2012-03-05T19:18:22.500Z --signature=MCwCFCT2On3fJ6dydk+MIoPp8zZ3ChbAAhQY01euDYqi6xaOD660dbYIML8qKQ==

See Indexer for more information

Product Senders

A SocketProductSender sends a product directly to a ProductHub, which stores the product and sends a Notification to all connected NotificationReceivers. Product hubs register a public key for each sender, and verify product signatures to ensure only known senders may distribute products.

See Sending Products for more information

Java API

Custom Notification and Indexer Listeners can be integrated with PDL using the Java API. Other languages need to use the ExternalNotificationListener and ExternalIndexerListener.

NotificationListeners must implement the gov.usgs.earthquake.distribution.NotificationListener interface. Most users can extend the gov.usgs.earthquake.distribution.DefaultNotificationListener class.

IndexerListeners must implement the gov.usgs.earthquake.indexer.IndexerListener interface. Most users can extend the gov.usgs.earthquake.indexer.DefaultIndexerListener class.

See the Java API Documentation for more information.

System Requirements


Installation

Download and unzip either one of the following distribution files. Each includes shell scripts for unix and a batch file for windows.