Network Working Group B. Gilbert
Internet-Draft HJ. Cho
Intended status: Standards Track A. Goode
Expires: March 06, 2013 J. Harkes
L. Huston
W. Richter
M. Satyanarayanan
R. Sukthankar
September 04, 2012

Diamond Protocol


The Diamond system is a framework for interactive search of non-indexed content such as medical images. The Diamond system is centered around an open source implementation named OpenDiamond. The Diamond protocol defines the communication mechanisms between the Diamond server and the client. The Diamond protocol carries data serialized according to the RFC 4506 format, and uses a custom Remote Procedure Call (RPC) protocol.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on March 06, 2013.

Copyright Notice

Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents ( in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction

1.1. Overview

The OpenDiamond system is a client-server search system that enables interactive search of Internet repositories that store vast amounts of complex and non-indexed data, such as webcam photographs and medical images. These data are referred to as objects, and a user can configure a search to return only the objects that meet some user-specified requirements. The user does this by requesting the OpenDiamond servers to run user-provided executables against the objects. For each executable the user also provides string parameters, a binary argument and various other configuration parameters (see Section 4.1.1 for a detailed list). The executable is called a filter, and the binary argument is called a blob argument; together, these are called blobs. At runtime, each filter produces a floating-point value called a score; for an object to pass the filter, its score must be within the range specified by the minimum and maximum threshold values in the configuration parameters.

Before initiating a search, a user must first define the scope: the set of objects that she desires to execute the search on. The user does this through the scopeserver, one of the main components of the OpenDiamond system. Using a web browser, the user connects to the scopeserver, authenticates, and selects a set of objects. Optionally, she can instruct the scopeserver to further restrict the set based on arbitrary object metadata. Once a scope has been selected, the user downloads a set of scope cookies, which are the entities that define the scope of objects to be searched.

Once the user obtains the scope cookies, she can invoke a client application to send the scope cookies to the servers, and the servers use the cookies to define the search scope. The client application also sends the filter configurations, which include a list of blob signatures that the servers use to determine whether they have all the blobs that are necessary to execute the search. After the client application has sent the blobs that are missing, the servers are ready for the search.

To initiate the search, the client application specifies the properties of objects that a user desires by sending the servers the names of a set of desired attributes. Attributes are named binary property values associated with an object, and the servers can use them to return properties of objects, such as thumbnail images, to the user. Attributes are typically generated by the filters while an object is being examined.

Upon the search request, the servers query the dataretrievers, which are essentially proxy servers that retrieve data from Internet repositories, to return the objects specified by the scope cookies. Once the servers obtain the objects, they carry out the selection process using the filters, and they return to the user the requested attributes for the objects that pass the filters. Typically in a search, a user will only request a subset of all the attributes, such as a small thumbnail image and the object name. Users can then request a more detailed set of information including more attributes. This process is referred to as reexecution.

The servers maintain double-precision floating-point values called session variables that are specific to a particular search. A filter can use these variables to keep track of typical parameter values for objects encountered during the search. This allows the filter to detect significant differences between the object it is currently examining and the objects that have already been examined. The OpenDiamond system is a distributed system, and by nature, the states across all servers can differ. To prevent the servers from becoming too far out of sync, the client can periodically retrieve the session variables, merge their values together, and send the combined values to all of the servers.

The remainder of this RFC is organized as follows. Section 1.2 briefly describes the OpenDiamond wire protocol. Section 1.3 describes the basics of requesting object attributes. Section 1.4 describes scope cookies. Section 2 describes the nonce exchange process that is required to begin a search. Section 3 describes the structure of an OpenDiamond RPC. Section 4 describes the protocol RPCs. Section 5 discusses system integrity and issues associated with security. Appendix Appendix A provides a reference guide of common Diamond terminology. Appendix Appendix B lists commonly-used Diamond attributes and attribute types. Appendix Appendix C lists server and filter-specific statistics values.

1.2. Diamond Protocol Overview

OpenDiamond uses a custom XDR-based Remote Procedure Call (RPC) protocol in a client-server model. Similar to a regular function or a procedure call, all operations are synchronous; when a client makes a request, it waits until the results of the remote procedure it requests are returned.

To perform a search, a client establishes two TCP connections with the server: the control connection and the blast connection. As the server can perform multiple simultaneous searches for multiple users, there must be a way to associate the control and blast connections of the same search. The two connections are therefore paired using an exchange of random nonces. The blast channel is used for transferring objects found during a search, and the control channel is used for all other client-server interactions.

Through the control channel, a client can use a set of RPCs to request searches, manipulate session variables or retrieve server statistics. The client typically begins a search by using the setup RPC (Section 4.1.1) to send the scope cookies and filter configurations to the server. The filter configuration does not include a copy of the filter code or blob arguments, as these are typically quite large; instead, a SHA-256 signature of this data is sent. The server replies with a list of filters and blob arguments it does not have. The client then uses the send_blobs RPC (Section 4.1.2) to send the missing filters and blob arguments. After these initial steps, the server is ready to perform a search. The client can then call the start RPC (Section 4.1.3) to initiate the search, and/or the reexecute RPC (Section 4.1.4) to obtain more information on a particular object. The client can also use the statistics RPC (Section 4.1.5) to obtain statistical information regarding the server, such as the average processing time of objects. If session variables are supported by the filters being used, the client can retrieve and update session variables with the get_session_variables (Section 4.1.6) and set_session_variables (Section 4.1.7) RPCs.

The blast channel is used for transferring search result objects. A client can call the get_object RPC (Section 4.2.1) to request an object. The get_object RPC does not have to be used synchronously, and the client will typically pipeline multiple get_object RPCs to minimize the effect of round-trip latency.

When the search has completed the server returns an object containing no attributes. The client MAY terminate the search at any time by closing the control and blast connections.

Below is a typical protocol interaction in a regex-like notation:

    setup send_blobs? ((reexecute (send_blobs reexecute)?)+ |
        (start (statistics | (get_session_variables set_session_variables))*))

1.3. Requesting and Retrieving Attributes

A client may specify a list of attributes when starting a search or reexecution. This list is called the push attribute list and it serves to restrict the set of attribute values that will be returned: the server returns the keys of all object attributes, but only returns values for those attributes included in the push attributes list. If no push attribute list is specified, the server returns values for all attributes. As a special case, the server MUST always return a value for the _ObjectID attribute.

1.4. Scope Cookies

OpenDiamond servers use cookies for two purposes: for authorization and to specify the objects to search. A scope cookie includes a cryptographic signature, and the servers MUST verify that this signature is from a trusted public key before executing a search. Scope cookies also contain the URLs of scope lists, which enumerate the objects to be searched.

Scope cookies are Base64-encoded, and the beginning and the end of a scope cookie are delimited by the strings:



The Base64-decoded contents of a scope cookie can be represented as follows:

    scope-cookie = signature LF data

    signature = 1*(DIGIT / "a" / "b" / "c" / "d" / "e" / "f")

    data = header LF body

    headers = *(header LF)

    (* Each header must appear once only *)
    header = "Version:" version |
             "Serial:" random-uuid |
             "Expires:" expiration-time |
             "Servers:" servers-list |
             "Blaster:" URL

    servers-list = server-address *((";" | ",") server-address)

    body = scopelist-url *(LF scopelist-url) [LF]

Scope cookie headers are defined as follows:

Version (mandatory) :
Cookie format version. The current version is 1.
Serial (mandatory) :
Random UUID that uniquely identifies the scope cookie.
Expires (mandatory) :
The expiration time of the scope cookie in ISO8601 format. The server MUST NOT accept an expired cookie.
Servers (mandatory) :
A list of hostnames or IP addresses for Diamond servers to be contacted for this search, separated by commas or semicolons. The server MUST verify that its name appears in this list.
Blaster (optional) :
The URL for the JSON Blaster that can communicate with the Diamond servers specified by the Servers header.

2. Nonce Exchange

A control connection and a blast connection are paired with a handshake using an exchange of random nonces. As soon as the socket connection is established for the control connection, the client MUST send a sequence of 16 null bytes as a nonce. The server MUST respond with a random nonce of 16 characters. The client uses this random nonce to establish the blast connection. As soon as the socket connection is established for the blast connection, the client MUST send this random nonce and the server MUST respond with the same random nonce.

2.1. Nonce

A nonce is a sequence of 16 random bytes. Below is the XDR representation of a nonce, serialized according to the RFC 4506 format:

    typedef opaque nonce[16];

3. Protocol Message

3.1. Messages

Diamond client-server messages consist of requests from client to server and responses from server to client. The requests and responses consist of a header and an optional body depending on the RPC. Request and response messages are serialized according to the RFC 4506 format. Below is the generic protocol message in the XDR data description language.

    struct message
        int sequence_number;
        int status;             /* See Section 3.3 for status code definitions */
        int command;            /* See Section 3.4 for command code definitions */
        opaque message_body<>;

3.1.1. Request Message

Request messages are signified by a status code of MINIRPC_PENDING (Section 3.3).

3.1.2. Response Message

A successful response message MUST have the status code of MINIRPC_OK (Section 3.3). A non-zero status code indicates an error. Negative codes indicate errors at the RPC protocol layer, and positive codes indicate errors in the application layer. The status code MINIRPC_PENDING MUST NOT be used in a response message. Error responses MUST have an empty message body.

3.2. Sequence Number

Each request MUST have a unique sequence number. The sequence number of the server response MUST match the sequence number of the request.

3.3. Status

The status element signifies the status of a particular RPC. Below is the list of status codes:

    enum status_code
        MINIRPC_OK                      = 0,   /* request processed successfully */
        MINIRPC_PENDING                 = -1,  /* denotes that a message is a client request */
        MINIRPC_ENCODING_ERR            = -2,  /* encoding error due to bad XDR structure */
        MINIRPC_PROCEDURE_UNAVAIL       = -3,  /* requested remote procedure not available */
        MINIRPC_INVALID_ARGUMENT        = -4,  /* request message body contains invalid value */
        DIAMOND_FAILURE                 = 500, /* no search scope, filters configured; cookie invalid */
        DIAMOND_FCACHEMISS              = 501, /* requested object is not cached */
        DIAMOND_COOKIE_EXPIRED          = 504, /* scope cookies have expired */
        DIAMOND_SCHEME_NOT_SUPPORTED    = 505  /* server does not support blob uri scheme */

3.4. Command

The command element indicates the remote procedure to execute. Below is the list of command codes:

    enum control_command_code
        setup                   = 25, /* used for search configuration (Section 4.1.1) */
        send_blobs              = 26, /* used to send filters and blob arguments (Section 4.1.2) */
        start                   = 28, /* used to initiate search (Section 4.1.3) */
        reexecute               = 30, /* used to request reexecution (Section 4.1.4) */
        statistics              = 29, /* used to request server statistics (Section 4.1.5) */
        get_session_variables   = 18, /* used to request session variables (Section 4.1.6) */
        set_session_variables   = 19  /* used to update session variables (Section 4.1.7) */

    enum blast_command_code
        get_object              = 2 /* used to request objects (Section 4.2.1) */

4. RPC Definitions

4.1. Control Connection RPC Definitions

4.1.1. setup

The setup RPC is used to configure the server for a search. In this RPC, a client sends the server the following entities to be used for search configuration:

The server uses the scope cookies to define the set of objects to be searched.

Each filter configuration contains the following:

The filter and blob argument URIs have the following format: sha256:<hexadecimal SHA-256 sum>.

The server MUST respond with a list of blob URIs that are not present in its cache. setup Request Body Encoding

The setup request body uses the following format:

    typedef string cookie<>;
    typedef string argument<>;
    typedef string dependency<>;

    struct filter_config
        string name<>;                /* filter name */
        argument arguments<>;         /* arguments to the filter */
        dependency dependencies<>;    /* list of other filters that the filter is dependent on */
        double min_score;             /* lower threshold to pass filter execution */
        double max_score;             /* upper threshold to pass filter execution */
        string code<>;                /* filter uri */
        string blob<>;                /* blob argument uri */

    struct setup_request_body
        cookie cookies<>;
        filter_config filters<>;
    }; setup Response Body Encoding

The setup response body uses the following format:

    typedef string uri<>;

    struct setup_response_body
        uri uris<>;

4.1.2. send_blobs

The send_blobs RPC is used to transmit filters and blob arguments to be added to the server's cache.

The response from the server MUST have an empty body.

The purpose of send_blobs is to send filters and blob arguments that are missing in the server. Therefore send_blobs RPC SHOULD only be used after the setup RPC, and only if the server indicates that there are missing filters and blobs. send_blobs Request Body Encoding

The send_blobs request body uses the following format:

    typedef opaque blob<>;

    struct send_blobs_request_body
        blob blobs<>;
    }; send_blobs Response Body Encoding

The send_blobs response body MUST be empty.

4.1.3. start

The start RPC is used to request that the server initiate a search. In this RPC, a client sends to the server the following entities to be used for the search.

The search ID is used to correlate searches across multiple servers; each server used in a search MUST receive the same search ID. The search ID MUST be a UUID in canonical string representation, and SHOULD be unique to the requested search over the lifetime of the scope cookies used.

When the start RPC is invoked, the server begins evaluating the objects in the scope and returning passed objects via the get_object RPC. The start RPC MUST NOT be called until necessary configuration has been performed using the setup and send_blobs RPCs. The response from the server MUST have an empty body. start Request Body Encoding

The start request body uses the following format:

    typedef string attribute_name<>;

    struct start_request_body
        opaque search_id[36];
        attribute_name *attribute_list;
    }; start Response Body Encoding

The start response body MUST be empty.

4.1.4. reexecute

A search invoked by the start RPC evaluates every object in the scope. The reexecute RPC, in contrast, allows a Diamond client to request that the filters be executed against one particular object, to obtain additional information about it. In this RPC, a client sends to the server the following:

The object ID identifies the object to be processed, and is obtained from the _ObjectID attribute from a previous search.

A client MAY specify an object ID consisting of the string "sha256:" followed by an SHA-256 hash value in hexadecimal. If an object matching this checksum is stored in the server's cache, the server MUST perform reexecution on this object. Otherwise, the server MUST respond with a DIAMOND_FCACHEMISS error. The client MAY then call send_blobs to send the object and then retry the reexecute RPC.

If the requested object is dropped by the filters, the server MUST return only the _ObjectID attribute. reexecute Request Body Encoding

The reexecute request body uses the following format:

    struct reexecute_request_body
        string object_id<>;
        attribute_name *attributes_list; /* see Section for attribute_name */
    }; reexecute Response Body Encoding

The reexecute response body uses the following format:

    struct attribute
        string name<>;
        opaque data<>;

    struct reexecute_response_body
        attribute attributes<>;

4.1.5. statistics

The statistics RPC is used to request that the server return statistical information about the search. statistics Request Body Encoding

The statistics request body MUST be empty. statistics Response Body Encoding

The statistics response body uses the following format:

    struct stat
        string name<>;    /* name of statistic */
        hyper value;

    struct filter_stat
        string name<>;    /* name of filter */
        stat stats<>;     /* list of statistics of this filter */

    struct statistics_response_body
        stat stats<>;  /* list of statistics associated with search */
        filter_stat filter_stats<>;   /* filter-specific statistics */

4.1.6. get_session_variables

A client can retrieve the server's session variables by calling the get_session_variables procedure.

The get_session_variables RPC MUST have an empty request body.

The client MUST NOT call get_session_variables twice without an intervening set_session_variables call. get_session_variables Request Body Encoding

The get_session_variables request body MUST be empty. get_session_variables Response Body Encoding

The get_session_variables response body uses the following format:

    struct session_variable
        string name<>;
        double value;

    struct get_session_variables_response_body
        session_variable session_variables<>;

4.1.7. set_session_variables

A client can use the set_session_variables RPC to modify the server's session variables.

The server MUST update each specified session variable to contain the specified value plus the difference between the current value of the variable and its value at the time of the most recent get_session_variables RPC.

The response to this RPC MUST have an empty message body. set_session_variables Request Body Encoding

The set_session_variables request body uses the following format:

    struct set_session_variables_request_body
        /* See Section for session_variable definition */
        session_variable session_variables<>;
    }; set_session_variables Response Body Encoding

The set_session_variables response body MUST be empty.

4.2. Blast Connection RPC Definitions

4.2.1. get_object

The get_object RPC is called on the blast connection to request an object that has passed the filters. This RPC will block until an object is ready. If no get_object RPCs are pending and the server produces a search result, the server MAY block, MAY queue the object and continue executing the search, but MUST NOT drop the search result. Multiple get_object requests MAY be pipelined so that the effect of round-trip latency between object requests can be minimized.

The client MAY perform the get_object RPC at any time after the blast connection is established. The server MUST indicate that the search has completed by returning an object containing no attributes on the blast channel. If the client performs further get_object RPCs after the completion of the search, the server's response is undefined.

The attributes included in the object struct MUST comply with the rules specified in Section 1.3. The list of push attributes is taken from the start RPC. get_object Request Body Encoding

The get_object request body MUST be empty. object Encoding

The XDR representation of a Diamond object is shown below:

    struct object
        attribute attributes<>; /* See Section for attribute definition */

5. Security Considerations

The OpenDiamond system allows users to run arbitrary x86 executable filter code on the servers. Although this feature makes the OpenDiamond platform a versatile search system, it also poses a major threat to server integrity. Server implementations must take care to execute filter code in a restricted sandbox to minimize the damage that can be caused by rogue filters.

6. References

Appendix A. Terminology

attribute :
A named binary value associated with an object during search execution. As an object is retrieved and filters are executed against it, attributes are associated with the object which can store result values, image thumbnails data, and so on.
blob argument :
A binary argument to a filter.
dataretriever :
The dataretriever is a simple HTTP server that emits an object list and objects in the format that an OpenDiamond server expects. In a simple Diamond setup, the scopeserver would be configured to produce a URL that points to a dataretriever local to the system that the OpenDiamond server is running on. The dataretriever will read objects locally and feed them to the OpenDiamond server.
filter :
A single program to be included in a search. Responsible for a single task, such as face detection or texture recognition. A filter is started when the search starts and killed when the search completes. Filters accept zero or more string arguments and exactly one blob (binary) argument. Filters can have dependencies on other filters; for example, a face detection filter can depend on another filter that decodes JPEG image to an RGB pixel array.
object :
A unit of data to be searched, such as a single image or text file. Each object to be examined is processed by one or more filters.
scope :
The set of objects to be examined during a particular search, typically computed via user interaction with a scopeserver. The scope is encoded in a scope cookie which is downloaded to the Diamond client, and then uploaded to one or more Diamond servers.
scope cookie :
OpenDiamond uses scope cookies to define what objects are going to be searched. The scope cookie contains one or more URLs, which the OpenDiamond server program uses to contact a dataretriever, which returns a list of objects. Each object in this list is again identified by a URL, which is used to retrieve the object before it is passed to the filters.
session variable :
The servers maintain double-precision floating-point values called session variables that are specific to a particular search. A filter can use these variables to keep track of typical parameter values for objects encountered during the search. This allows the filter to detect significant differences between the object it is currently examining and the objects that have already been examined. The OpenDiamond system is a distributed system, and by nature, the states across all servers can differ. To prevent the servers from becoming too far out of sync, the client can periodically retrieve the session variables, merge their values together, and send the combined values to all of the servers.

Appendix B. Attributes

Appendix B.1. Attributes that are Handled Directly by the Server

Most attributes are generated by filters while an object is being examined, but some attributes are generated directly by the server. These include:

"" (the zero-length string):
object data
_ObjectID :
URI uniquely identifying an object
Display-Name :
user-friendly name of an object
Device-Name :
string uniquely identifying the server that processed this object
_filter.%s_score (%s is the name of the filter) :
the score assigned to the object by the named filter, as a string

Appendix B.2. Common Diamond Attributes

thumbnail.jpeg :
small thumbnail image for the object :
height of an object in pixels :
width of an object in pixels
_rgb_image.rgbimage :
decoded pixel data for the object.
_filter.%s.patches :
regions of interest located by the named filter. %s is the name of the filter.
_filter.%s.heatmap.png :
a grayscale heat map of regions deemed interesting by the filter; brighter pixels are more interesting. %s is the name of the filter.

Appendix B.3. Attribute Name Suffixes

By convention, Diamond attribute names include a suffix that indicates the type of data stored in the value. This allows the client to implement generic display handlers for different types of data.

.int :
32-bit binary-encoded little-endian integer
.float :
32-bit binary-encoded little-endian floating-point value
.double :
64-bit binary-encoded little-endian floating-point value
.jpeg :
JPEG image data
.png :
PNG image data
.rgbimage :
Uncompressed pixel data, encoded as a C structure with little-endian members (see below). Since these attributes can be large, filters that produce them will typically request that the Diamond server not return them to the client.
.patches :
Regions of interest in an image. The value is encoded as a C structure with little-endian members (see below). The distance value signifies how dissimilar the regions of interest are from the example patches provided by a client. For example, if there is a 90% resemblance, the distance is 0.1. The minimum and maximum x/y values specify the upper-left and lower-right pixel coordinates of a region within the image.
.binary :
Arbitrary binary data such as object data.

    struct RGBImage
        uint32_t type;      /* RGBImageType */
        uint32_t nbytes;    /* size of this struct */
        int32_t height;
        int32_t width;
        struct RGBPixel data[0];

    struct RGBPixel
        uint8_t r;
        uint8_t g;
        uint8_t b;
        uint8_t a;          /* mainly just padding */

    enum RGBImageType
        IMAGE_UNKNOWN = 0,
        IMAGE_PBM = 1,
        IMAGE_PGM = 2,
        IMAGE_PPM = 3,
        IMAGE_TIFF = 4,
        IMAGE_JPEG = 5,
        IMAGE_PNG = 6

    struct __attribute__((packed)) patches
        int32_t num_patches;
        double distance;
        struct patch patches[0];

    struct patch
        int32_t min_x;
        int32_t min_y;
        int32_t max_x;
        int32_t max_y;

Appendix C. List of Statistics

Appendix C.1. List of Server Statistics

objs_total :
Total number of objects in the scope.
objs_processed :
Total number of processed objects.
objs_dropped :
Total number of objects dropped while being processed.
objs_passed :
Total number of objects passed after being processed.
objs_unloadable :
Total number of objects that could not be fetched.
avg_obj_time_us :
Average processing time per object in microseconds.

Appendix C.2. List of Filter Statistics

objs_processed :
Number of objects considered.
objs_dropped :
Number of objects dropped.
objs_cache_dropped :
Number of objects dropped by the cache.
objs_cache_passed :
Number of objects accepted by the cache.
objs_computed :
Number of objects examined by the filter.
objs_terminate :
Number of objects causing the filter to crash.
avg_exec_time_us :
Average filter execution time per object.

Authors' Addresses

Benjamin Gilbert EMail: bgilbert AT cs DOT cmu DOT edu
HongJai Cho EMail: ahjcho.tb AT gmail DOT com
Adam Goode
Jan Harkes EMail: jaharkes AT andrew DOT cmu DOT edu
Larry Huston
Wolfgang Richter EMail: wolf AT cs DOT cmu DOT edu
Mahadev Satyanarayanan EMail: satya AT cs DOT cmu DOT edu
Rahul Sukthankar EMail: rahul AT cs DOT cmu DOT edu