root/trunk/src/client/usrint/README_UCACHE @ 9223

OrangeFS User Cache Documentation
-------------------------------------------------------------------------------

1. Overview
        1.1     The user cache (ucache) for OrangeFS is an optional layer of 
                functionality that can be used to improve I/O latency and 
                throughput of OrangeFS/PVFS files. Users can use the familiar 
                stdio library functions and POSIX system calls by statically 
                linking with our static library or LD_PRELOAD the shared 
                library.

2. Installation
        2.0 Shared Memory Adjustments
                Since the ucache relies on SYSV shared memory, the SYSV 
                shared memory limits on your system probably need to be 
                increased.

                For your convenience, a Linux compatible script has been 
                included to adjust these values for you. Though, in order 
                to make these changes permanent, appropriate steps must be taken:

        Linux users should run and follow the instructions as the script 
        executes:

            By default, the script is located in /src/apps/user.
            The script depends upon the file ucache.conf to supply the script
            with the necessary info to compute the total shared memory size
            required by the ucache. Run as root:

            ./checkset_shmall_shmmax.linux

            At the end of the script there's a note about what to do to make 
            the shared memory limit changes persistent.
        
        2.1 Configure ucache enabled 
                ./configure --enable-ucache <other config options>

        2.2 Configure ucache disabled
                ./configure --disable-ucache <other config options>

        Note: By default, currently, the ucache is not compiled automatically,
        so if you wish to use it, make sure to perform the configure step in
        in section 2.1.

        2.3 Finish
                make
                make install

3. ucached Daemon capabilities
    3.0 The ucache daemon (ucached) is a daemon that polls FIFO descriptors
                to perform commands sent to it by the ucached daemon 
                client (ucached_cmd). When the daemon recieves a new command, 
                it attempts to execute the command and returns a response 
                through another FIFO back to ucached_cmd. The ucached must be 
                properly started before the ucache can be used. 

        Following installation, by default, ucached and ucached_cmd are both
        located in /usr/local/sbin. 

        3.1 Starting the ucache:
                ucached_cmd s

                Note: this command effectively calls "ucached_cmd c" as a part of 
        starting the ucache, so the only time the user should call 
                "ucached_cmd c" themselves is only after they've destroyed the 
                shared memory segments with "ucached_cmd d" and would like them
                recreated. 

        3.2 Exiting the ucache:
                ucached_cmd x

        Note: this command effectivley calls "ucached_cmd d" as a part of exiting the 
        ucache, so the only time the user should call "ucached_cmd d" 
        themselves is if they want to destroy the shared memory segments 
        used by the ucache.

        3.3 Remove the ucache related SYSV shared memory segments:
                ucached_cmd d

        3.4 Create and Initialize the SYSV shared memory segments:
                ucached_cmd c

        3.5 Dump info pertaining to the ucache to stdout:
                ucached_cmd i

                *Note: info options can be specified that control the verbosity
                of this output. The ordering of these options doesn't matter. 
                Multiple options can be used by simply appending additional 
                options to the previous ones. 

                ex:     ucached_cmd i spcf (note 'spcf' is equivalent to 'a') 

                Options:
                ---------------------------------------------------------------
                        Default - no info options (same as 'c')

                a   Show All Info

                c       Show ucache contents (append 'f' to see the free ucache elements,
            see below)

                s       Show Statistics Summary

                p       Show ucache parameters

                f       Show free ucache elements (must be used in conjunction with with 
            'c' to affect output)

                Combo examples:
                ---------------------------------------------------------------
                i sc    Show stats and ucache contents  

                i cf    Show ucache contents (including free ucache elements)

                i spc   Show stats, parameters, and contents (but not free ucache 
                elements)

                ...etc

4. Examples
    4.1 Simple Test Program
===============================================================================
        Compiled with:
gcc -g -Wall -O0 -o 14 14.c \
-L/usr/local/lib -lofs -lpvfs2 -rdynamic -lssl -lcrypto -lpthread -ldl
===============================================================================
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

#define B_SIZE (1024 * 256)
#define N_BLKS 679

int main(int argc, char **argv)
{
    /* Write buffer */
    size_t write_buff_size = N_BLKS * B_SIZE;
    char *write_buff = (char *) malloc(write_buff_size);
    assert(write_buff);
    memset(write_buff, '0', write_buff_size);

    /* Open file */
    char fullpath[]= "/mnt/pvfs2/myfile";
    int file = open(fullpath, O_RDWR | O_CREAT,
        S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
    if(file < 0)
    {
        perror("open");
        return -1;
    }

    /* Perform the write */
    int howmany = write(file, write_buff, write_buff_size);
    if(howmany < write_buff_size)
    {
        perror("write");
        close(file);
        return -1;
    }

    /* Seek back to beginning of file */
    off_t lseek_rc = lseek(file, 0, SEEK_SET);
    if(lseek_rc == -1)
    {
        perror("lseek");
        close(file);
        return -1;
    }

    /* Read Buffer */
    int read_count = B_SIZE;
    char read_buff[N_BLKS];

    /* Perform many reads on small segments to demonstrate caching benefit */
    int i;
    for(i = 0; i < read_count; i++)
    {
        howmany = read(file, read_buff, N_BLKS);
        if(howmany != N_BLKS)
        {
            perror("read");
            close(file);
            return -1;
        }
    }

    /* Close File */
    if(close(file) != 0)
    {
        perror("file close");
        return -1;
    }
    return 1;
}
===============================================================================

    4.2 Timed Test w/o Cache
        time sudo ./14

        real    1m35.229s
        user    0m31.900s
        sys 0m11.110s

    4.3 Timed Test w/ Cache
        time sudo ./14

        real    0m2.365s
        user    0m0.860s
        sys 0m0.690s

    4.4 Test w/ Cache Stats Summary
        SUCCESS
            user cache statistics:
            hits=   262822
            misses= 679
            hit percentage= 99.742317
            pseudo_misses=  0
            block_count=    0
            file_count=     0

5. Limits
    5.1 Block Limit
        The ucache is composed of blocks which store both hash table data to 
        facilitate ucache functionality and the actual data itself. Currently 
        the ucache is limited to 1024 blocks.

    5.2 File Entry Limit
        The ucache can keep up to 682 files in memory at once. Any files beyond
        that will not use the ucache. Also, files are flushed from the ucache 
        when they are closed.

    5.3 Memory Entry Limit
        Memory entries represent the data block cached and some associated 
        cache meta-data. The ucache can keep up to 682 memory entries per 
        file at once. Any memory entries inserted beyond that limit will cause
        eviction of the least recently used (LRU) memory entry of that file. 
        For the file being read/written, if no LRU memory entry is available 
        to be evicted, the LRU memory entry of the file in ucache with the most
        memory entries is evicted.
 
6. Issues
    6.1 Gossip

    6.2 Multithreading

    6.3 Multiprocessing