Sending Products

« Back to User Guide

$Id: sending.html 22941 2015-04-06 14:11:05Z jmfee $
$URL: https://ghttrac.cr.usgs.gov/websvn/ProductDistribution/trunk/etc/documentation/userguide/sending.html $

Navigation

Getting Started

  1. The application comes with an embedded configuration that should be sufficient for most Product Senders in a production environment. Be default, this configuration sends to all Product Distribution Hubs. During development, or for the sake of being explicit, you can create a simple configuration file to send to the development Product Distribution Hub.
  2. Create a new Keypair.

    Senders must use their private key to sign products. Receivers may use a sender's public key to verify signatures.

    ssh-keygen -t dsa -b 1024 -C 'Comment describing key' -f <keyfilename>
    

    Do not assign a key passphrase. This creates two files: a private key named “<keyfilename>” and a public key named “<keyfilename>.pub”.

    As with all public/private keypairs, NEVER share your PRIVATE key.

    Give your PUBLIC key to everyone who needs to verify your product signatures.

  3. Register your public key

    Send your public key and information about your products to jmfee@usgs.gov. You will receive a source and type that can be used to send products.

  4. Generate product documentation.

    This will let receivers know what they can expect. View existing documentation for examples of other product documentation.

  5. Use the Command Line or Java API to Send Products.

    Send your products using the assigned source and type.

Designing New Products

What is a Product?

A product is a specific version of any content produced in response to an earthquake, information about an earthquake, or another product.

A product usually consists of a directory of files, but may also include a stream of bytes, links to internet resources, and key/value pairs.

Product IDs

The source, type, and code uniquely identify a product and its associated content. Update times indicate different versions of the same product, and products with more recent update times supersede products with less recent update times.

source
The product creator. Usually a network code.
type
The type of product. Multiple sources may create the same type of product.
code
A unique code, for a given source and type combination.
update time
When this version of content was created. Two products with the same source, type, and code, but different update times are different versions of the same product.

Status

A brief status code. Typically UPDATE or DELETE. Any value that is not DELETE is considered an update.

When a product producer deletes a product they send an update with a status DELETE, which is distributed like any other product.

Contents

Files

Contents are optional, but are usually present when the status is not DELETE. Each content has a path, which corresponds to a filename.

Non file content

Each product may have one piece of content that is a stream of bytes. It has an empty path and isn't stored in a file. Typically this content is read from standard input, and also delivered to consumers via standard input.

Metadata

Links

A product may include links to other products and resources. Links have a relation, which speci├×es how the resource is related to the product. There may be multiple links for each type of relation.

Properties

Key value pairs. Each property may only have one value. These can be used to determine whether further processing is required. "Standard" properties include:

eventsource
network code of related event
eventsourcecode
network event code of related event
eventtime
time of related event
latitude
latitude of related event
longitude
longitude of related event
depth
depth of related event
magnitude
magnitude of related event
version
producer product version. This is not used for ordering products, which use update time as a versions.

All of these properties are optional, but a combination of either eventsource and eventsourcecode OR eventtime, latitude, and longitude are recommended for event-related products for association purposes.

Signature

Producers generate a signature which may be verified by both hubs and consumers. The signature consists of a hash of all product contents and attributes, which is encrypted using a private key and verified using a public key.

Production hubs require a signature, and a keypair must be registered before products are sent.

Generating a “Good” Product

The information contained in this section is a recommendation, not a requirement however recommendations can be seen as a best practice when generating new products.

Product Properties

Properties should be defined for important attributes of your product. They allow recipients to quickly determine whether they should process a product, without extensive parsing.

For example, eventsource and eventsourcecode are used by the indexer to associate a product with an event. Other examples include dyfi number of responses, losspager alert level, and shakemap maximum MMI.

See the product documentation for other examples of product properties.

Contents.xml File

It is recommended you include with your product a “contents.xml” file. This file describes all of the contents you are sending with your product. It helps the web page to better display your product and ultimately determines what content users will be able to easily download from the web site.

The contents.xml file format is defined with a contents.xsd file. You must understand how to read XML schema (XSD) files in order to properly generate a contents.xml file.

Grouping Content

When mapping products into PDL, it is important to consider how content is generated and updated when deciding what individual products should contain.

For event-related products, the group of content is everything related to that event.

For non-event-related products, the group depends more on how content is generated and updated. For example, station related content might be broken up by station. It's possible that this content would need to be further divided based on update intervals.

Product Code Conventions

A product code must be unique forever, from the same source and type. Product creators also reuse the code to update or delete the product.

For event-related products, an eventid makes a good code because it also must be unique forever. Examples include : us2011abcd and nc12345678.

For non-event-related products another ID scheme is required. Timestamps are often useful in this situation when the products are related to a time window.

An internal database key can be used.

Using the Command Line to Send Products

java -jar ProductClient.jar --send [send arguments]

Send Arguments

The Command Line API is used to create new products, and by external listeners to receive products.

These arguments are mandatory.

--source=<source>
This is assigned when you register your KeyPair with the hubs.
Examples include us, nc, and ci.
--type=<type>
Product type. This is assigned when you register your KeyPair with the hubs.
Examples include shakemap, pager, and dyfi.
--code=<code>
A unique identifier for this product. This should only be reused when updating an existing product. Generally a full event id (2 character network code followed by up to 8 character network event code)
Examples include us2009abcd and nc12345678
--trackerURL=<trackerURL>
A location where tracking updates are sent.
Example: --trackerURL=http://ehppdl1.cr.usgs.gov/tracker/

These arguments are optional.

--updateTime=<updateTime>
Optional. Default is now. When this version of source+type+code was updated.
An example is 2010-01-14T14:11:28.691-07:00.
--status=<status>
Optional. Default is UPDATE. Product generators may use any status without spaces. However, the status must be used consistently for all products of that type.
Examples include UPDATE, and DELETE.
--delete
Optional. A synonym for --status=DELETE. This updates the status to delete. Product generators may send content along with a "deleted" product.
NOTE: ProductDistribution makes no attempt to remove earlier versions of the product, and only attempts to deliver this update.
--property-<name>=<value>
Optional. Product property. Each property may have only one value. Name and value must not contain an equals (=).
--eventtime=<eventtime>
Optional. When the event with which this product is associated occurred.
Examples include 2010-01-14T14:11:28.691-07:00 and 2010-01-14T21:11:28.691Z.
Synonym for --property-eventtime=<eventtime>.
--latitude=<latitude>
Optional. Latitude (in decimal degrees) of event with which this product is associated.
Synonym for --property-latitude=<latitude>.
--longitude=<longitude>
Optional. Longitude (in decimal degrees) of event with which this product is associated.
Synonym for --property-longitude=<longitude>.
--eventsource=<eventsource>
Network that generated event with which this product is associated.
Examples include us, nc, and ci.
--eventsourcecode=<eventsourcecode>
Code assigned by network that generated event with which this product is associated.
Examples include 2008abcd and 12345678.
--magnitude=<magnitude>
Optional. Magnitude of event with which this product is associated.
Synonym for --property-magnitude=<magnitude>.
--depth=<depth>
Optional. Depth (in kilometers) of event with which this product is associated.
Synonym for --property-depth=<depth>.
--version=<version>
Optional. Product generator version. This is independent of the Product updateTime, which is used by distribution for versioning.
Synonym for --property-version=<version>.
--eventid=<eventid>
Deprecated. Use the --eventsource and --eventsourcecode arguments instead.
Optional. Full 10 character eventid with which this product is associated.
An example is us2008abcd.
Synonym for --property-eventid=<eventid>.
--link-<relation>=<uri>
Optional. Represent a relationship between this product and another product or resource. Each relation may have many uris. <uri> must be a valid URI as defined by RFC2396 or RFC2732.
Example: --link-related=urn:usgs-product
--content
Optional. Non-file product content is available via standard input.
--contentType=<contentType>
Optional, used with --content. Content mime-type for content read from standard input.
--file=<file>
Optional. <file> is a path to a file that is product content. The file's name and modification date are preserved. The mime type is inferred from the file extension. The file is added at the root level of the product.
--directory=<directory>
Optional. <directory> is a path to a directory that contains product contents. Any directory hierarchy within <directory> is preserved.
--privateKey=<privateKey>
Optional. Path to a file containing a PKCS#8 encoded private key in PEM or DER format. When present, the key will be used to generate a signature for this product.
--configFile=<file>
Optional. Path to a file containing the configuration for your sender. If ommitted, the sender will check the current working directory for a file called “config.ini”. If that file does not exist the sender will use the default embedded configuration.

Send Examples

Sending or Updating a Product

Sending a new product is the same as sending an update to a product.

Line continuation characters (\) added for readability, all arguments would appear on one line.

% java -jar ProductClient.jar --send \
	--source=us \
	--type=dyfi \
	--code=EVENTID \
	--directory=DYFIPRODUCTCONTENTS \
	--privateKey=PRIVATEKEYFILE \
	--eventsource=EVENTNETWORK \
	--eventsourcecode=EVENTCODE \
	--eventtime=2010-10-25T15:10:24Z \
	--latitude=12.345 \
	--longitude=23.456 \
	--property-num-responses=17000

In the above example, a dyfi product is sent. Source, Type, and Code uniquely identify this product. Directory contains all the product files. The Private Key is used to sign the message. Event properties eventsource, eventsourcecode, eventtime, latitude, longitude are all included. In addition, a dyfi specific property "num-responses" is included that indicates how many people have filled out the dyfi questionaire.

Deleting a previously sent Product

To delete a product, reuse the same source, type, and code, and set the status to DELETE using the --delete argument.

Line continuation characters (\) added for readability, all arguments would appear on one line.

% java -jar ProductClient.jar --send \
	--source=us \
	--type=dyfi \
	--code=EVENTCODE \
	--eventid=EVENTCODE \
	--delete \
	--privateKey=PRIVATEKEYFILE

In the above example, a dyfi product is deleted. Source, Type and Code uniquely identify which product is being deleted. The --delete argument sets the status to DELETE. The Private Key is used to sign the message.

Product Signatures

Product Senders use an OpenSSH DSA keypair used to sign and verify their products.

Creating a new Keypair

ssh-keygen -t dsa -b 1024 -C 'Comment describing key' -f <keyfilename>

Do not assign a key passphrase. This creates two files: a private key named "<keyfilename>" and a public key named "<keyfilename>.pub"

As with all public/private keypairs, NEVER share your PRIVATE key.

Give your PUBLIC key to everyone who needs to verify your product signatures.

Using the Keypair

The private key is used by the sender when sending products, and the public key is used by receivers when verifying products.

Signing products with the private key

Use the "--privateKey=<keyfilename>" argument when --send ing products.

Configuring signature verification with the public key

This will check signatures before processing a product.

Update the ProductStorage configuration:

[my_storage]
...
verifySignatures = true
keychain = ..., key_<keyfilename>

[key_<keyfilename>]
type = gov.usgs.earthquake.distribution.ProductKey
types = <producttype>
sources = <productsource>
key = <public key contents>

How is the signature generated and verified?

Both the sender and receiver compute a SHA-1 digest of all product contents and metadata (except the signature).

A sender uses their private key to encrypt this digest, and sends the encrypted digest as the signature.

A receiver uses the senders public key to decrypt the signature (only the senders public key is able to decrypt the signature). The receiver then compares the computed digest to the sender's digest.

If the sender and receiver digests match, the signature is verified and the content is what was originally sent. If the sender and receiver digests DON'T match, the product is not what was originally sent, or was signed using a different key.

Compression

Depending on the type of content your product contains (text/binary/compressed binary), transfer compression can significantly decrease network usage. However, this is a trade off between available CPU and network bandwidth.

A general rule is text or uncompressed binary content should use compression; compressed binary content should not use additional compression. You can read how to configure your transfer on the ProductSender configuration.

You can use the DeflateComparison utility test content you expect to send:

java -cp ProductClient.jar gov.usgs.earthquake.distribution.DeflateComparison FILE [FILE ...]
Where FILE(s) are files or directories that are part of your product content.

Quakeml compression example

etc/test_products/quakeml/pde20100314080803960_32.xml, length = 611853 bytes
no compression
	0.000s, 0.0% reduction (0 fewer bytes)
default compression (-1)
	0.497s, 94.3% reduction (576916 fewer bytes)
best speed (1)
	0.444s, 93.4% reduction (571463 fewer bytes)
best compression (9)
	0.432s, 94.6% reduction (578911 fewer bytes)

ShakeMap compression example

etc/test_products/usa00040xz/us_shakemap_usa00040xz_1287260900624.xml, length = 2405991 bytes
no compression
	0.001s, 0.0% reduction (0 fewer bytes)
default compression (-1)
	1.819s, 39.2% reduction (943012 fewer bytes)
best speed (1)
	1.747s, 36.7% reduction (883394 fewer bytes)
best compression (9)
	1.502s, 39.3% reduction (944685 fewer bytes)

PAGER compression example

etc/test_products/usa00040xz/us_losspager_usa00040xz_1287260989064.xml, length = 1519786 bytes
no compression
	0.000s, 0.0% reduction (0 fewer bytes)
default compression (-1)
	0.934s, 43.8% reduction (665810 fewer bytes)
best speed (1)
	0.902s, 41.2% reduction (625532 fewer bytes)
best compression (9)
	0.966s, 43.9% reduction (667636 fewer bytes)