Access to Data other than the Unidata-Wisconsin Datastream
In Data File Access we described what McIDAS data files you can expect to find on your system and how you can access them. In this section we describe how you can alter the types and numbers of data files on your system.
McIDAS data products can be classified into four general types:
Control of which files are written to end user's file systems is handled by the McIDAS File Routing System. How many files are maintained on end user's file systems is controlled by both the File Routing System and by Data Scouring.
The McIDAS file routing system, as used by Unidata, is composed of two subsystems:
Data in the Unidata-Wisconsin channel are transmitted in identifiable units knows as products. Data products are uniquely identified by a two-character Product Code.
The Product Codes assigned to data objects in the Unidata-Wisconsin datastream are:
Product Code Product Description McIDAS SYSKEY.TAB Vals --------------+---------------------------------------+---------------------- CA CIMSS Cloud Top Pressure 2173 2174 2175 CB CIMSS Precipitable Water 2176 2177 2178 CC CIMSS Sea Surface Temperature 2179 2180 2181 CD CIMSS Lifted Index 2182 2183 2184 CE CIMSS CAPE 2185 2186 2187 CF CIMSS Ozone 2188 2189 2190 CG CIMSS Wildfire ABBA Nth Hemishere 2191 2192 2193 CH CIMSS Wildfire ABBA Sth Hemishere 2194 2195 2196 CI GOES-E/W Infrared Composite 2161 2162 2163 CV GOES-E/W Visible Composite 2164 2165 2166 CW GOES-E/W Water Vapor Composite 2167 2168 2169 N1 GOES-E IR/Topography composite 2125 2126 2127 N2 GOES-E VIS/Topography composite 2128 2129 2130 N3 GOES-W IR/Topography composite 2131 2132 2133 N4 GOES-W VIS/Topography composite 2134 2135 2136 N5 MDR/Topography composite 2137 2138 2139 N6 Mollweide/Topography composite 2140 2141 2142 N7 GOES-E/W IR/Topography composite 2155 2156 2157 N8 GOES-E/W VIS/Topography composite 2158 2159 2160 RL NEXRCOMP: 6 km Nat. Base Refl. Comp. 2361 2362 2363 RN NEXRCOMP: 10 km RCM Composite 2364 2365 2366 RO NEXRCOMP: 1 km Flt. Base Refl. Comp. 2367 2368 2369 UA Educational floater I 2101 2102 2103 UB GOES-W Western US Water Vapor 2104 2105 2106 UC Educational floater II 2170 2171 2172 UI GOES-E North America Infrared 2143 2144 2145 UR Research floater 2107 2108 2109 UV GOES-E North America Visible 2146 2147 2148 UW GOES-E North America Water Vapor 2149 2150 2151 UX Global Mollweide Infrared Composite 2122 2123 2124 UY Global Mollweide Water Vapor Composite 2152 2153 2154 U1 Antarctic composite 2119 2120 2121 U3 Manually Digitized Radar 2110 2111 2112 U5 GOES-W Western US Infrared 2113 2114 2115 U9 GOES-W Western US Visible 2116 2117 2118Decoders convert the image objects in the datastream to data files of the type AREA. On Unix, the decoders for doing this are provided in the LDM-McIDAS package.
The image decoder will use the McIDAS File Routing Facility, if it is told to do so, and if it exists and is writable by the user running the decoder, to determine the output name space for the files that it creates. If the File Routing Facility does not exist (i.e. it has not been created or is not in the directory in which the decoders write their output), the decoders will create products based on the default name that each product is tagged with in the datastream.
The default names for products in the datastream are presented in Unidata-Wisconsin Data Stream Products
The result of having a file routing table is conceptualized in the following diagram:
The McIDAS file redirection facility can be used in combination with the file routing table to locate files ingested and decoded from the Unidata-Wisconsin datastream. This is schematically represented in:
The File Routing Facility is created/updated/listed with the McIDAS ROUTE command. The routing facility configuration is embodied in the file ROUTE.SYS.
ROUTE is used to:
Adding and Deleting Routing Entries
The syntax for using ROUTE to ADD or DELete products from the routing table are:
ROUTE ADD pcode type begin end file keywords "description ROUTE DEL pcode1 pcode2 ... pcodenwhere:
type - product type: AREA, GRID, MD, or TEXT begin - beginning file number for types: AREA, GRID, and MD end - ending file number for types: AREA, GRID, and MD file - file name for type: TEXTThe keywords supported by ROUTE are:
FORM=ALL - for LIST option, List expanded form of product entryThe control of the output name space comes from the ability to control the output file numbers for AREA, GRID, and MD files and from file for TEXT files. Since all non-image products were removed from the Unidata-Wisconsin datastream in summer 1999, ROUTE is mainly used for products of type AREA.CC=code - decoder condition code, range 1 to 7 (default=1) 1 - decoder ended normally -| If condition is met: 2 - partial failure during decoding | 3 - either 1 or 2 above | * - Increment cylinder 4 - total failure (nothing decoded) |-> * - Execute Post Proce> 5 - either 1 or 4 above | (if specified) 6 - either 2 or 4 above | 7 - all conditions -|
PP=file xcute | execute a post process batch file where file = name of batch file to execute, up to 12 characters xcute = NO -> execute batch file independent of condition code = YES -> execute batch file based on condition code (default)
SYS=bword eword cword | set file number bounds in SYSKEY table where bword = SYSKEY word for file number beginning value eword = SYSKEY word for file number ending value cword = SYSKEY word for file number current value
By setting the range of file numbers that, say, AREA files can occupy, one directly controls how many of them are to be filed on disk. The same holds true for GRID and MD files, but MD files required some further considerations.
The file routing system was designed to allow the user to configure it to automatically run McIDAS processes upon successful receipt of data products. This is where the CC= and PP= keywords are used. PP= specifies the name of a McIDAS BATCH file that is to be run upon successful receipt of a product. CC= defines what successful actually means.
BATCH files run automatically by the routing system are referred to as ROUTE PostProcess BATCH files.
On Unix systems, ldm-mcidas decoders run from the Unidata LDM to create data files. Since the LDM is typically not run by the user mcidas, system administrators must perform extra configuration steps to enable running of ROUTE PostProcess BATCH files. For more information, the user is advised to review Route Postprocess Batch Invocation on Unix.
The invocation syntax for these BATCH file is always:
BATCH pcode fname DAY TIME ccode "bnameHere:
pcode -> the product code of the product received
fname -> the number of the file just written (AREA, GRID, and MD)
or the file name (TEXT)
DAY -> the date the file was received in YYDDD
TIME -> the time the file was received in HHMMSS
ccode -> the decoder status code
bname -> the name of the BATCH file to run
The SYS= keyword, if specified, tells the data decoder to update
entries in another McIDAS facility, the System Key Table.
The System Key Table is embodied in the file
SYSKEY.TAB. Unidata uses SYSKEY.TAB as a storage location for
information on what data is currently available. SYSKEY.TAB is
sharable among all versions of McIDAS.
The number tuple managed by the SYS= keyword represents the beginning, ending, and current file numbers for products of type AREA and the file number, time, and day for products of type GRID and MD. Unidata does not use SYSKEY.TAB entries for files of type TEXT.
On Unix systems, ldm-mcidas decoders run from the Unidata LDM are responsible for creating data files. Since the LDM is not run by the user mcidas, system administrators must take care to insure that the files ROUTE.SYS, SYSKEY.TAB, and SCHEMA are readable and writable by the user running the LDM. Furthermore, the system administrator must insure that all three of these files are placed in the directory in which the ldm-mcidas decoders create their output data files.
Examples of using ROUTE to configure data file ingestion:
ROUTE ADD U1 AREA 190 199 CC=3 SYS=2119 2120 2121 "Antarctic IR Composite ROUTE ADD U5 AREA 130 139 CC=3 SYS=2113 2114 2115 PP=GW-IR.BAT "GOES-West US IR Band 4 ROUTE ADD U9 AREA 120 129 CC=3 SYS=2116 2117 2118 PP=GW-VIS.BAT "GOES-West US VisibleUnidata provides a McIDAS BATCH file, DROUTE.BAT, that is designed to aid the user in configuring their site's routing table for all of the products available by the Unidata IDD.
ROUTE LIST presents the user with a snapshot at the routing table at the time it runs:
ROUTE LIST S Pd Description Range Last Received Post Process C - -- ------------------------- --------- ------------ ---------- ------------ - CA Cloud Top Pressure 1100-1109 AREA1109 02356 1715 CTP.BAT 3 CB Precipitable Water 1110-1119 AREA1111 02356 1616 PW.BAT 3 CC Sea Sfc. Temperature 1120-1129 AREA1121 02356 1532 SST.BAT 3 CD Lifted Index 1130-1139 AREA1137 02356 1633 LI.BAT 3 CE CAPE 1140-1149 AREA1141 02356 1634 CAPE.BAT 3 CF Ozone 1150-1159 AREA1152 02356 1651 OZONE.BAT 3 CG NH Wildfire ABBA 1190-1199 AREA1192 02356 1536 WFABBA.BAT 3 CH SH Wildfire ABBA 1200-1209 AREA1203 02356 1552 WFABBA.BAT 3 CI GOES-E/W IR Composite 80-89 AREA0081 03202 2221 none 3 CV GOES-E/W VIS Composite 90-99 AREA0096 03202 2221 none 3 CW GOES-E/W H2O Composite 70-79 AREA0076 03202 2220 none 3 LD NLDN Lightning Flashes 71-71 MDXX0072 03202 2231 none 3 s MA Surface MD data default none none SFC.BAT 3 N1 GOES-East IR/TOPO Composi 220-229 AREA0228 03202 2131 none 3 N2 GOES-East VIS/TOPO Compos 230-239 AREA0230 03202 2130 none 3 N3 GOES-West IR/TOPO Composi 240-249 AREA0240 03202 2221 none 3 N4 GOES-West VIS/TOPO Compos 250-259 AREA0250 03202 2221 none 3 N5 MDR/TOPO Composite 260-269 AREA0265 03202 2206 none 3 N6 Mollweide IR/TOPO Composi 270-279 AREA0272 03202 2237 none 3 N7 GOES-E/W IR/TOPO Composit 280-289 AREA0287 03202 2221 none 3 N8 GOES-E/W VIS/TOPO Composi 290-299 AREA0293 03202 2221 none 3 NF Global Initialization Gri 101-102 none none GLOBAL.BAT 3 NG Early Domestic Products 1-2 none none ADDGRID.BAT 3 R1 Base Reflectivity Tilt 1 300-339 none none none 3 R2 Base Reflectivity Tilt 2 340-379 none none none 3 R3 Base Reflectivity Tilt 3 380-419 none none none 3 R4 Base Reflectivity Tilt 4 420-459 none none none 3 R5 Composite Reflectivity 460-499 none none none 3 R6 Layer Reflect SFC-24 K ft 500-539 none none none 3 R7 Layer Reflect 24-33 K ft 540-579 none none none 3 R8 Layer Reflect 33-60 K ft 580-619 none none none 3 R9 Echo Tops 620-659 none none none 3 RA Vertical Liquid H2O 660-699 none none none 3 RB 1-hour Surface Rain Total 700-739 none none none 3 RC 3-hour Surface Rain Total 740-779 none none none 3 RD Storm Total Rainfall 780-819 none none none 3 RE Radial Velocity Tilt 1 820-859 none none none 3 RF Radial Velocity Tilt 2 860-899 none none none 3 RG Radial Velocity Tilt 3 900-939 none none none 3 RH Radial Velocity Tilt 4 940-979 none none none 3 RI 248 nm Base Reflectivity 980-1019 none none none 3 RJ Storm-Rel Mean Vel Tilt 1 1020-1059 none none none 3 RK Storm-Rel Mean Vel Tilt 2 1060-1099 none none none 3 RL 6 km Nat. Base Refl. Comp 1160-1169 none none none 3 s RM Mandatory Upper Air MD da default none none MAN.BAT 3 RN 10 km RCM Composite 1170-1179 none none none 3 RO 1 km Flt. Base Refl. Comp 1180-1189 none none none 3 s RS Significant Upper Air MD default none none SIG.BAT 3 U1 Antarctic IR Composite 190-199 AREA0192 02356 1512 none 3 U2 FSL2 hourly wind profiler default MDXX0082 02356 1617 none 3 U3 Manually Digitized Radar 200-209 AREA0201 02356 1705 MDR.BAT 3 U5 GOES-West US IR Band 4 130-139 AREA0130 02356 1635 GW-IR.BAT 3 U6 FSL2 6-minute Wind profil default MDXX0092 02356 1704 none 3 U9 GOES-West US Visible 120-129 AREA0120 02356 1635 GW-VIS.BAT 3 UA Educational Floater I 160-169 AREA0160 02356 1636 none 3 UB GOES-West US Water Vapor 170-179 AREA0174 02356 1635 GW-WV.BAT 3 UC Educational Floater II 60-69 AREA0063 02356 1637 none 3 UI GOES-East US IR Band 4 150-159 AREA0158 02356 1630 GE-IR.BAT 3 UR Research Floater 180-189 none none none 3 s US Undecoded SAO Data default none none none 1 UV GOES-East US Visible 140-149 AREA0140 02356 1630 GE-VIS.BAT 3 UW GOES-East US Water Vapor 210-219 AREA0218 02356 1630 GE-WV.BAT 3 UX Mollweide Composite IR 100-109 AREA0102 02356 1330 MOLL.BAT 3 UY Mollweide Composite H2O 110-119 AREA0112 02356 435 none 3In addition to defining the correspondence between product codes and file names, the routing table contains information on:
ROUTE LIST UB FORM=ALL Product: UB Type: AREA Status: Active Condition Code: 3 Cylinder Begin: 170 Cylinder End: 179 Current Position: 175 Product Description: GOES-West Western US H2O Creation Day: 100301 Creation Time: 20:00:00 Received Day: 2000301 Received Time: 20:55:27 Post Processing Batch File: GW-WV.BAT Condition Code Testing: Yes SYSKEY word | Cylinder Begin: 2104 Cylinder End: 2105 Current Position: 2106 route.k: Done
In order for the LDM-McIDAS decoders to be able to use ROUTE.SYS, it has to exist in the directory in which the decoders will create their output data files. In order for SYSKEY.TAB to be updated by the LDM-McIDAS decoders, it also has to be located in the in the directory in which the decoders will create their output data files. Finally, both ROUTE.SYS and SYSKEY.TAB must exist together and both must be readable and writable by the user running the LDM.
McIDAS provides a utility that is used to store system-wide defaults for a variety of McIDAS parameters, the System Key Table. Unidata uses the System Key Table to store information about the most recently decoded data from LDM-McIDAS and XCD processes. The System Key Table entries used by Unidata are listed in the file PRODUCT.DAT that is located in the data directory of the McIDAS distributions.
The System Key Table, realized in the file SYSKEY.TAB, is created and manipulated with a pair of McIDAS commands:
Since SYSKEY creates a new copy of SYSKEY.TAB, it is a dangerous command in the hands of inexperienced or malicious users. Unidata sites typically run systems of McIDAS machines/sessions that all get information on current file availability from a shared SYSKEY.TAB. If a user that has write permission for the shared copy of SYSKEY.TAB runs SYSKEY, then all information on data availablility stored in SYSKEY.TAB will be lost to ALL users. Unidata strongly recommends that system administrators set file permissions on the shared version of SYSKEY.TAB so that it is writable only by the user running the LDM and by the user mcidas. All other users should have read-only access to SYSKEY.TAB.
Since the corruption of SYSKEY.TAB entries is unfortunately possible, Unidata provides two McBASI scripts that are used to save and restore SYSKEY.TAB values:
Whenever you consider doing some system maintenance that may affect SYSKEY.TAB, you should first save entries by running:
GETSYS
If the work you were doing did damage SYSKEY.TAB, simply restore the values saved by GETSYS by running:
PUTSYS
To list SYSKEY.TAB entries, one uses the SYSVAL LIST option:
SYSVAL LIST bword ewordHere:
bword - the beginning SYSKEY.TAB word to list (default=1) eword - the ending SYSKEY.TAB word to list (default=bword)SYSVAL can also modify entries in SYSKEY.TAB:
SYSVAL CHANGE word valueHere:
word - the SYSKEY.TAB word to modify (no default) value - the new value for word (no default)
SYSVAL CHANGE is as dangerous as SYSKEY in the hands of inexperienced or malicious users.
SPC T NA #SYS(2002) MDF=#SYS(2001) DAY=#SYS(2003)The SYSKEY.TAB entries used in this example refer to:
#SYS(2002) - current SAO/METAR data time [HH] #SYS(2001) - current SAO/METAR MD file number #SYS(2003) - current SAO/METAR date [YYDDD]
In File Routing System, you were introduced to the facility that allows you to control how many of each kind of data file is allowed to be filed on your system. In this section, we discuss how you control the numbers of files that you keep on your system.
McIDAS contains a built-in Scheduler facility that allows for the execution of McIDAS and non-McIDAS applications based on the system clock. McIDAS also provides data scouring routines that, when run, will trim files to user-specified limits. The combination of the scheduler running the data scouring routines can be used to control the numbers of files on your system.
The scheduler facility is maintained by a set of three routines:
SKU provides five basic functions:
SKE day time repeat int <keywords> "commandHere:
day - Julian day to initiate command [CCYYDDD]
time - time to initiate command [HH:MM:SS]
repeat - number of times the command should execution; if you enter 999999 the
word MANY replaces it in the header (default=1)
int - time interval between command executions [DDDHH:MM:SS]
(def=1:00:00, meaning 1 hour)
command - command to execute
The keywords supported by SKE are:
FILE= - scheduler file name; valid in McIDAS-X only; see the McIDAS-X
Users Guide for details (default=SKEDFILE)
ID= - 4-digit number to identify the command in the scheduler
(default=system assigned number)
NAME= - user initials (user id) to use for local/host commands
PROJ= - project number command is executed under (default=current)
TOL= - late tolerance, HH:MM:SS; if the command cannot execute within
the designated time, it is ignored (default=1 hour)
In practice, one typically does not specify any keyword except TOL=
when creating scheduler entries. The TOL= keyword specifies the
time interval, which starts at the time a command is supposed
to be run by the scheduler, in which a command can run. If for some reason
a command can not be run within this time interval (for instance, when
the scheduler is off), the scheduler will not try to run it.
For data scouring, one specifies that a souring command is to be run at least once-per-day. The time that the command runs is user-selectable, but Unidata recommends that it be sometime close to, but after 0 Z.
SKL allows one to list out the entries in the user's scheduler easily. For instance, the following shows a listing of our recommended entries for the mcidas supervisory user:
SKL FORM=ALL
*** SCHEDULER IS OFF *** MESSAGE DEVICE IS: N
T# ID XS NEXT EXECUTN # REM INTERVAL TOL NAME PROJ COMMAND TEXT...
-- ---- -- ------------ ----- -------- ----- ---- ---- ------------
1 1 96314 13000 MANY 1000000 BIG USER 0 DOQTL 1 30 3
1 2 96314 13500 MANY 1000000 BIG USER 0 DOQTL 31 60 3
1 3 96314 14000 MANY 1000000 BIG USER 0 DOQTL 71 80 3
1 4 96314 14500 MANY 1000000 BIG USER 0 DOQTL 81 90 3
1 5 96314 15000 MANY 1000000 BIG USER 0 IGU DEL #+GFILE
1 6 96314 15500 MANY 1000000 BIG USER 0 LWU DELETE VIRT9001
1 7 96314 15600 MANY 1000000 BIG USER 0 LWU DELETE VIRT9002
1 8 S 96314 10000 MANY 30000 259 USER 0 RUN UX 998 DEBUG FILE=G
ETBYFTP.MCB
--END OF LIST
For sites running McIDAS-XCD, the recommended scheduler entries
look like:
SKL FORM=ALL
*** SCHEDULER IS OFF *** MESSAGE DEVICE IS: N
T# ID XS NEXT EXECUTN # REM INTERVAL TOL NAME PROJ COMMAND TEXT...
-- ---- -- ------------ ----- -------- ----- ---- ---- ------------
1 1 96314 13500 MANY 1000000 BIG USER 0 QRTMDG MD 1 90 3
1 2 96314 14000 MANY 1000000 BIG USER 0 QRTMDG GRID 1 10 2
1 3 96314 14000 MANY 1000000 BIG USER 0 QRTMDG GRID 101 102 2
1 4 96314 14000 MANY 1000000 BIG USER 0 QRTMDG GRID 5100 5400 1
1 5 96314 15000 MANY 1000000 BIG USER 0 IGU DEL #+GFILE
1 6 96314 15500 MANY 1000000 BIG USER 0 LWU DELETE VIRT9001
1 7 96314 15600 MANY 1000000 BIG USER 0 LWU DELETE VIRT9002
1 8 S 96314 10000 MANY 30000 259 USER 0 RUN UX 998 DEBUG FILE=G
ETBYFTP.MCB
--END OF LIST
You will note that from the listings above that the scheduler facility is
not ON or active. In order to activate scheduler entries, one
has to run SKU as follows:
SKU ON
Since the scouring routines are McIDAS applications, they need McIDAS environment information in order to run. This is easily setup in a shell script by defining the environment variables MCDATA, MCPATH, and PATH.
Unidata provides an example shell script, mcscour.sh that includes all of the scouring functionality in the scheduler listing above. The example scouring contained in mcscour.sh is intended to be run once-per-day, typically in the middle of the night.
The Unidata Internet Data Distribution system (IDD) gives the Unidata user community free access to all of the Family Of Services (FOS)/NOAAPORT datastreams that originate from the National Weather Service:
The twenty Level III NEXRAD products include:
All of the above data services are described in more detail at:
Data Available Through Unidata
For many years, Unidata McIDAS users were unable to take advantage of this wealth of data instead having to live with only the products available in the Unidata-Wisconsin datastream. With the LDM-McIDAS package developed at Unidata and SSEC's McIDAS-XCD package, this situation has changed dramatically. Unidata McIDAS users can now decode all the data available in these streams. Unidata McIDAS users' limitation to data is now simply a matter of their departmental processing capabilities and storage capacities.
The Unidata McIDAS distribution also provides routines that allow for easy conversion of non-McIDAS data into McIDAS compatible formats.
The McIDAS Abstract Data Distribution Eenvironment (ADDE) perhaps provides for the simplest access to the vast stores of data contained in NOQAPORT. Users allowed by cooperating institutions can display and analyze data without having to transfer it to local storage (although copying the data for local storage is also provided by ADDE).
Users can also get access to archived Unidata-Wisconsin datastream products. Interested users should consult the Data Recovery page maintained by Unidata.