Welcome
-------------------

In this file, ">>" indicates a line to execute. The intent is that reading through this file while executing these lines provides a reasonable way to install and build OpenDiamond.

Notes
-------------------
-- OpenDiamond does NOT work with the latest version of gcc. Downgrade gcc until you stop getting "can't compile with both -static and -shared".
-- adiskd can't find data -> check the number of zeros in GIDIDX0000...000...


Building the system
-------------------

If you are building from CVS, run autoconf to generate the configure script:
>>(1) cd diamond/src; autoconf

Now run the configure script. The OpenDiamond source trees allows compilation output to be in a separate tree, so the sources can be read-only. Here the tree starts in a separate directory named opendiamond-OBJS.

Thus either do:
>>(2a) cd opendiamond/src; ./configure

OR

>>(2b) mkdir opendiamond-OBJS; cd opendiamond-OBJS/
>>(2b) /fullpath..../opendiamond/src/configure

The configure script may alert you to missing libraries. Note that
OpenDiamond requires Kerberos 5 and OpenSSL. Install the development
packages on your system for any missing libraries.

Then build OpenDiamond. As the system builds, all of the important header files,
libraries and binaries will be put in target/{include, lib, bin}.

>>(3) make

One make has completed, you can either run OpenDiamond programs  out of the 
build directory or you can install it in the system.  Running
"make install" will install all the files in 
/usr/local/diamond/{include,lib,bin}  (this requires root access).  Options
to make install allow data to be install into other locations.

>>(4 - optional) make install

OpenDiamond Configuration
-------------------------

The OpenDiamond platform has two components.  The first runs on the back end
where the data is stored and the other is the runtime running
on the host where the application is executed.  Both
require special configuration files that describe
how the system is configured and where data is located.

On the back-end the daemon `adiskd` needs to be running.  This
handles incoming requests and creates new search contexts by forking
off new processes.  To simplify deployment, we currently run this
as a specific user instead of as a service.  Any files that
are to be processed during the search, need to be read-accessible
to the user starting the daemon.

The front-end software is included in the Diamond applications
through the libraries. 

In both cases (the front and back end), the necessary configuration files live
in a configuration directory.  The system will search the following
locations for the configurations files in this order.  First
it looks for an environmental variable "DIAMOND_CONFIG"  if
this is set, it uses the value as the potential directory
for the configuration files.  Next is searches for a directory
called ".diamond" in the current working directory.  The last
location is "${HOME}/.diamond".  


Back-end configuration
---------------------

On each of the back-ends there should be a config file
called "diamond_config".  This is a text file
with lines that describe where the data is stored.  Currently
there are two types of storage available.  The first uses
the native file-system, the second uses an object store
built on the native file system.  We will only consider
the native file system for now.   The text of 
the file should be something like

	DATAROOT /dataroot_directory
	INDEXDIR /index_file_directory
	CACHEDIR /cache_storage
	DATATYPE NATIVE

The line DATAROOT gives a prefix relative
to all the files.  Each file being searched is opened
with the concatenation of the DATAROOT and the local file
name.  The INDEXDIR gives the absolute pathname to a directory
where index files are stored (see below for more on index
files).  CACHEDIR give a directory that the system can
use to store cached state.  Some place in /tmp is usually
a good location.  DATATYPE NATIVE, tells the system to use
the native files instead of the object store.

Data in Diamond is organized into groups.  Each group contains files
that are related to each other.  For examples, one group can be images,
another group can be sounds, etc.   Each group is assigned a 64-bit
number that is used to identify the collection.  (In many config
files we write the number as 8 8-bit values separate by colons:
group 1 -> 00:00:00:00:00:00:00:01, etc.)  Each group has
an index file that is a list of the files belonging to that group,
for the NATIVE file system this index file is a text file where each
line is a file.   The index files are stored in INDEXDIR with a name
GIDIDX????????????????, where the ? are the hex values of the 64-bit
number.  A simple way to generate an index file is using find.  For
example to generate a index of all ppm files into group 1 run

	find . -name \*.ppm > indexdir/GIDIDX0000000000000001



Here is a reasonable way execute all this. Assume your data, .ppm images, are in the directory /home/you/diamond/data. We will create a diamond_idx directory in your home directory.

Create the index file:
>>(5) cd ~; mkdir diamond_idx
>>(6) cd ~/diamond/data; find . -name \*.ppm > ~/diamond_idx/GIDIDX0000000000000001

And a location for cached data:
>>(7) mkdir /tmp/cachejunk

>>(8) cd ~; mkdir .diamond

Put all the pieces together:
>>(9) Create the file diamond_config in .diamond/
DATAROOT        /home/you/diamond/data
INDEXDIR        /home/you/diamond_idx
DATATYPE        NATIVE
CACHEDIR        /tmp/cachejunk


Front-end configuration
-----------------------

On the front-end there are several configuration files that describe 
where the data resides.  The first is the gid_map file
that maps groups to one or more machines that holds data in that
group.  The lines in this file are composed of a gid (colon seperated) 
followed by a list of machines that have data for that group.  
The other file is name_map that takes a symbolic name 
(e.g.  "photos" and maps it to one or more groups).  

Name the group, e.g. search1
>>(10) In .diamond/, create the file name_map
search1 00:00:00:00:00:00:00:01

We have to map the GID to its location on the network. E.g. for localhost:
>>(11) In .diamond/, create the file gid_map
00:00:00:00:00:00:00:01 127.0.0.1


Running adisk
------------

To start the adiskd, just run it without any arguments.  This will
create a daemon that runs in the background.  Running adiskd -h
will show other arguments.


Run adiskd in debug mode:
>>(12a) diamond/src/target/bin/adiskd -d -n
OR
>>(13a) diamond-OBJS/target/bin/adiskd -d -n

Remote setup
------------

Manging several machines can be a challenge, to simplify the task
we provide a shell scripts in the target/bin directory.  This
shell script assumes the use of ssh and that the user the ability
to login into the other machines.  This works better when
ssh-agent is used to avoid many login prompts.

First edit the file remote.config to match the system configuration.
Also edit the diamond_config.sample file to match the desired
configuration.  

Then you can perform tasks by running dsetup.  

	./dsetup install 
Will shutdown the remote servers and load the new diamond_config and binaries
onto them.

	./dsetup start

will start the daemons on the remote machines.

	./dsetup stop

will stop the daemons on the remote machine's.



Building Debian packages
------------------------

Debian packaging files are in the debian subdirectory.  To build the
debian packages from scratch, run the following from the OpenDiamond
source directory, which must be named opendiamond-<version>.

	cp -pr src opendiamond-<version>
	cd opendiamond-<version>
	fakeroot debian/rules clean

This will do a make clean.  Then build the source package.
	cd ..
	dpkg-source -ICVS -b -sn opendiamond-<version>

This will create the source package, which consists of:
	opendiamond_<version>.tar.gz (source tarball)
	opendiamond_<version>.dsc	(source summary)

Build the binary package:
	cd opendiamond-<version>
	fakeroot debian/rules binary

This will create opendiamond_<version>_<arch>.deb in the directory above.



Installing Debian packages
--------------------------
To install the binary package:
	dpkg -i opendiamond_<version>_<arch>.deb

To unpack the source package:
	dpkg-source -x opendiamond-<version>.dsc


Building RPMs
-------------

Use the alien package to build RPMs from debian files.
To build the binary package:
	alien -k -r opendiamond_<version>_<arch>.deb

This will generate opendiamond-<version>.<arch>.rpm

To build the source package:
	alien -k -r opendiamond_<version>.tar.gz

This will generate opendiamond_<version>.noarch.rpm
