[IDD #TTR-495389]: Slow access to local datasets through ADDE

Hi Samuel,

re:
> O.K. I believe that the \CURDAY construct, by itself, has nothing to do with 
> loading
> images in IDV.

Good, because it simply can't.  The way ADDE works is as follows:

- there are 'dir' (listings) and 'get' (transfer data) requests.  'dir' requests
  return back lists of directory information for one or more images in a 
dataset.

- the more images one has in a dataset, the more directories there are to 
transfer
  back when asking for information on all images in a dataset

- 'get' requests, on the other hand, always send back information for one image

So, cutting down the number of images in a dataset using the \CURDAY construct
will speed up the 'dir' requests since there will be less image directories to
send back to the client (e.g., IDV or McIDAS).  The 'get' request (i.e., display
an image) will not change at all, so it is hard for me to understand how the
display of an image in the IDV would change.

Question: we are discussing displaying an image in the IDV, true?

> The datset list for our "\CURDAY" replaceable set looks fine using the 
> "Absolute"
> tab to view files.

Good.  It should show a shorter list than when including all images in the 
dataset.

> It seems as though something about the new dataset I've created with the 
> \CURDAY
> "replaceable", causes IDV to use up more than 500MB of memory, even on a 
> single image.

Again, we are talking about displaying the image aren't we?

> The exact same image, with the exact same timestamp displays fine in the 
> dataset
> with * as part of the directory path, but fails with \CURDAY as part of the 
> path.

This seems to me to be impossible, but I am willing to consider the possibility 
of
something unexpected.

I just logged onto shu and ran some tests using the full set of images in the
GINIEAST/GE1KVIS dataset (using '*') and the same set of tests using \CURDAY
to limit the set of images searched to those from today.  The listing and when
using \CURDAY was MUCH faster than when using '*'.  The display of the same 
image
was faster when using \CURDAY since it takes less time for the server to 
determine
which image to send back.

> It looks like it might have to do with the default Image Size and 
> Magnification of
> the dataset.

If this is true, it has nothing to do with the ADDE serving of the data.  Rather
it has something to do with the defaults the IDV will use in the display.

> Is there some way I can set these default values so it doesn't blow IDV up?

In the IDV image selection widget, there should be some controls that allow
you to load a portion of the image and/or specify a stride (blowdown in McIDAS
terms) for the data (i.e., every other pixel, every third pixel, etc.).

> Please see enclosed images.

I have to believe that in the case where all images are defined, the IDV
somehow knows to get only every nth pixel from the server while in the
case where you are using \CURDAY, its default is to ask for all pixels.
In the latter case, you would be dealing with a LOT of data to render.
If you want to display the entire GINI Vis images, you will need to increase
the amount of memory that the IDV has available to it.  This is done by
modifying the startup script:

EXAMPLE

change:

-Xmx512m

to:

-Xmx1024m

N.B.: The amount of data you can tell the IDV to use will be dependent on how 
much
memory you have on the machine running the IDV.

> The output from a dsserve.k LIST GINIEAST looks like this:
> -bash-2.05b$ dsserve.k LIST GINIEAST
> 
> Group/Descriptor         Type  Format & Range     RT Comment
> ------------------------ ----- ------------------ -- --------------------
> GINIEAST/GE1KVIS         IMAGE GINI                  GINI 1km VIS East CONUS
> DIRFILE=/data/ldm/mcidas/images/sat/EAST-CONUS/1k
> m/V
> IS/*/VIS_*
> GINIEAST/GE1KVIST        IMAGE GINI                  GINI 1km VIS East CONUS 
> - T
> oday
> DIRFILE=/data/ldm/mcidas/images/sat/EAST-CONUS/1k
> m/V
> IS/\CURDAY/VIS_*
 ...
> dsserve.k: done


Ah Ha!  Now I understand... You created a new dataset, GINIEAST/GE1KVISST, and 
the IDV
does not have an entry in its defaults file to specify how to load the image 
(stride,
etc.).  Because it is an unknown dataset, the default will be to load the entire
image.  This is why you are running out of memory!

Try this test:

<as 'mcidas'>
cd $MCDATA
-- edit RESOLV.SRV and modify your ADDE dataset definition for GINIEAST/GE1KVIS 
to
look exactly like the one for GINIEAST/GE1KVIST:

change:

N1=GINIEAST,N2=GE1KVIS,TYPE=IMAGE,RT=N,K=GINI,R1=1,R2=999999,MASK=/data/ldm/mcidas/images/sat/EAST-CONUS/1km/VIS/*/VIS_*,C=GINI
 1km VIS East CONUS,

to:
N1=GINIEAST,N2=GE1KVIS,TYPE=IMAGE,RT=N,K=GINI,R1=1,R2=999999,MASK=/data/ldm/mcidas/images/sat/EAST-CONUS/1km/VIS/\CURDAY/VIS_*,C=GINI
 1km VIS East CONUS,

Exit out of the editor saving your changes.

Next, load the GINIEAST/GE1KVIS image from the IDV.  Notice that the defaults 
for how
to load the image are now in effect, and your IDV session does not run out of 
memory.

Bottom line: the \CURDAY construct has _nothing_ to do with loading of images 
in the IDV
other than the time it takes to start sending back the data will be shorter.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed