[McIDAS #XRR-599672]: Configuring scouring of McIDAS data

Hi Samuel,

Sorry for the late reply.  It was a busy day...

re:
> I'm running into a similar type problem in /data/ldm/mcidas/grib, where
> there are files named *2008081*, *2008080*, *2008074*, and *2008073* files.
> Should I delete them too?

Hes, all of these are very old.

> There are also some files in /data/ldm/mcidas/bufr that have access times
> greater than 20 days ago (*2008072*,*2000001*, *1999001*, etc...) should I
> delete these types of files also?

Actually, McIDAS does not do anything with the BUFR files yet, so you are free
to delete all of them.  They are filed so that sites that have written their
own BUFR access routines/decoders have them accessible.  Their existence is
_not_ recorded in the MySQL database.

> One possible solution I can think of for removing files that have access
> times greater than expected, is setup a crontab entry that would have 
> something
> like the following entry:
> find /data/ldm/mcidas/grib -name "*" -atime +10 -print -exec rm -f {} \;

Looks like the scouring approach that is used in the LDM utility 'scour'.

> I'm not certain if this would cleanup the sql database, however.

There is no MySQL impact for the BUFR files.

I don't believe that there is anything you need to do for GRIB files that
have names that indicate old data in the MySQL database.  I say this after
reviewing the method used in the XCD scouring utility, xcdscour:

GRIB|grib)
        i=$max_days_to_delete

        # MySQL
        # Do this before deleting files so client doesnt get false positives
        LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
        export LD_LIBRARY_PATH
        daynum=`days_ago_year $days`
        MBIN=`which mysql |grep -v no`
        if [ -n "$MBIN" ]; then
                strSQL="DELETE FROM file_list WHERE j_day <= \"$daynum\";"
                echo "$strSQL" |$MBIN --user=gribwrite --password=gribwrite 
mcrtgrib >/dev/null 2>&1
                strSQL="OPTIMIZE table file_list;"
                echo "$strSQL" |$MBIN --user=gribwrite --password=gribwrite 
mcrtgrib >/dev/null 2>&1
        fi

        while [[ $i -ge $days ]]; do
                daynum=`days_ago_year $i`
                hournum=0
                while [[ $hournum -le 23 ]]; do
                        rm -f *.*.$daynum.$hournum??.*.*.grib
                        rm -f *.*.$daynum.$hournum??.*.grb2
                        hournum=$((hournum+1))
                done 
                rm -f *.*.$daynum.*.*.*.grib
                rm -f *.*.$daynum.*.*.grb2
                i=$((i-1))
        done
        ;;

The operative portion of this snippit is:

        daynum=`days_ago_year $days`
        MBIN=`which mysql |grep -v no`
        if [ -n "$MBIN" ]; then
                strSQL="DELETE FROM file_list WHERE j_day <= \"$daynum\";"
                echo "$strSQL" |$MBIN --user=gribwrite --password=gribwrite 
mcrtgrib >/dev/null 2>&1
                strSQL="OPTIMIZE table file_list;"
                echo "$strSQL" |$MBIN --user=gribwrite --password=gribwrite 
mcrtgrib >/dev/null 2>&1
        fi

which says to delete entries in the MySQL database whose dates are <= $daynum.  
To me
this means that all files older than $daynum will be deleted.  Note, however, 
that
this does not account for files that get named with dates in the future.  They
should be scoured eventually, however.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: XRR-599672
Department: Support McIDAS
Priority: Normal
Status: Closed