Merging and Archiving Network Data

Manual URL:

http://www.eas.slu.edu/New_Earthquake/Data_retrieve/

BLO
FVM
MPH
OLIL*
PBMO
PVMO
SIUC*
SLM
UALR
USIN*
UTMT
*18 channels - BH, BL, HH, HL, LH, LL

BH and BL are 40 samples per second data.
1 24-hour day = 3456000 samples (40x60x60x24).

HH and HL are 100 samples per second data.
1 24-hour day = 8640000 samples (100x60x60x24).

LH and LL are 1 samples per second data.
1 24-hour day = 86400 samples (1x60x60x24).

See output from *err files or qmerge -n:

SIUC.NM.HHZ. rate=100 (2005.256 00:00:00.0083 to 2005.257 00:00:00.0083) : 8640000 points, 0.0 msec correction, (min,max,max_step = -0.0,0.0,0.0 msec)

SIUC.NM.BLZ. rate=40 (2005.256 00:00:00.0195 to 2005.257 00:00:00.0195) : 3456000 points, 0.0 msec correction, (min,max,max_step = -0.0,0.0,0.0 msec)

SIUC.NM.LLZ. rate=1 (2005.256 00:00:00.0695 to 2005.257 00:00:00.0695) : 86400 points, 0.0 msec correction, (min,max,max_step = -0.0,0.0,0.0 msec)

rlogin wiggles: ssh -X wiggles
passwd: xxxxxxxx

cd /seis/rbh/STAGING

NMakeDay ??? (??? is yesterday's julian day).

cd BLO/WORK.BLO.???

ls -l; check file sizes of BH channel data at LH channel data

more *err

If all files are complete move to next directory. cd ../../MPH/WORK.MPH.???

Repeat the process for all stations.

If qmerge can't read the header. Go to station's parent directory, run msfix on the seed file, read past the bad header.

cd /cnmsn/SLM/BHE.D
msfix SLM.NM.BHE.D.2004.142.0001
qmerge -n SLM.NM.BHE.D.2004.142.0001

If you have a data gap, run

qlog /cnmsn/STA/LOG.L/*2005.???.* | pg

to check for system errors (#ERR or #MSG) occuring at data gap.

If no system error messages appear, run necessary data retrieval processes.

Merge data files.

cp /seis/rbh/STAGING/sta_merge to station data directory:
cp /seis/rbh/STAGING/sta_merge BLO/WORK.BLO.250
cd /seis/rbh/STAGING/sta_merge BLO/WORK.BLO.250
vi sta_merge
change the previous julian day {PDOY}
change the current julian day {DOY}
change the station name {STA}
sta_merge
check data

Archive data (/seis/rbh/STAGING/netmove JDAY).

To retrieve missing day files from systems with remote Sparcs storage systems:

on wiggles, get the machine addresses:
pg /etc/hosts (prints out the machine addresses)

su - seismic
password: xxxxxxxx

OR (if no day files are present, or if there are time gaps)
kermit; con; retrieve archive files; scp files to wiggles
logout

On wiggles:
cd /cnmsn/BLO
sdrsplit CBYYMMDD.???
mv BLO*BHE* BHE.D; mv BLO*BHN* BHN.D...

move to work directory and merge data
(vi /eas/seis/work/rbh/STAGING/sta_merge; run sta_merge)
move files to /net/taumsauk/r/2005/???

OR (if no day files are present, or if there are time gaps)
kermit; con; retrieve archive files; scp files to wiggles
logout

On wiggles:
cd /cnmsn/MPH
sdrsplit CBYYMMDD.???
mv MPH*BHE* BHE.D; mv MPH*BHN* BHN.D...
Move to work directory and merge data; archive data to taumsauk

If no day files are present, or if there are time gaps, there is nothing more you can do at this point. Logout and notify Eric of station outage.

On wiggles:
mv OLIL.NM.BHE.D.2005.256.0000 BHE.D, etc.
Move to work directory and merge data; archive data to taumsauk

Retrieving SIUC data:
ssh siucbbs.geo.siu.edu
password: xxxxxxxx (same as seismic's)
cd SIUC
ls BH?.D
If channel day files are there, run sendalltowiggles to send all the files over
or you can send one channel at a time by:
scp BH?.D/*2005.???.* wiggles.eas.slu.edu:/cnmsn/SIUC
repeat process for BL, HH, HL, LH and LL channels

If no day files are present, or if there are time gaps, there is nothing more you can do at this point. Logout and notify Eric of station outage.

On wiggles: follow same procedure as for OLIL and SIUC

Retrieving USIN data:
ssh tgpusin.usi.edu
password: xxxxxxxx (same as seismic's)
cd USIN
ls BH?.D
If channel day files are there, run sendalltowiggles to send all the files over
or you can send one channel at a time by:
scp BH?.D/*2005.???.* wiggles.eas.slu.edu:/cnmsn/USIN
repeat process for BL, HH, HL, LH and LL channels

If no day files are present, or if there are time gaps, there is nothing more you can do at this point. Logout and notify Eric of station outage.

On wiggles: follow same procedure as for OLIL

Move to work directory and merge data; archive data to taumsauk

Retrieving UALR* data:
ssh ualrbbs.ualr.edu
password: xxxxxxxx (same as seismic's)
cd UALR
ls BH?.D
scp BH?.D/*2005.???.* wiggles.eas.slu.edu:/cnmsn/UALR
repeat process for LH channels
logout
OR (if no day files are present, or if there are time gaps)

telnet to Quanterra using "grab" script:

grab tgpualr.uasys.edu bh\? YY/MM/DD,HH:MM:SS YY/MM/DD,HH:MM:SS /cnmsn/UALR/grab.log /cnmsn/UALR

To retrieve data from systems without a remote Sparc storage system:

Retrieving data from PVMO:
cd /cnmsn/PVMO
kermit; setisdn; con; atdt 3168292; k bh?; retrieve archive files

sdrsplit CBYYMMDD.???
mv PVMO*BHE* BHE.D; mv PVMO*BHN* BHN.D...

Retrieving data from FVM:
cd /cnmsn/FVM
kermit; setisdn; con; atdt 2146563; k bh?; retrieve archive files

sdrsplit CBYYMMDD.???
mv FVM*BHE* BHE.D; mv FVM*BHN* BHN.D...

Retrieving data from PBMO:
cd /cnmsn/PBMO
kermit; setpbmo; connnect; k bh?; retrieve archive files

sdrsplit CBYYMMDD.???
mv FVM*BHE* BHE.D; mv FVM*BHN* BHN.D...

Retrieving data from UTMT:

grab tgputmt.utm.edu bh\? YY/MM/DD,HH:MM:SS YY/MM/DD,HH:MM:SS /cnmsn/UTMT/grab.log /cnmsn/UTMT

To retrieve data from SLM:
cd /cnmsn/SLM
login as spyder (password same as seismic's)
kermit; setmanual2; con; atdt 9773127; retrieve archive files

sdrsplit CBYYMMDD.???
mv SLM*BHE* BHE.D; mv SLM*BHN* BHN.D...
OR
Go to: http://www.eas.slu.edu/New_Earthquake/Data_retrieve/ and click on the SLM section.

"Splitting" seed files:

cd /cnmsn/{STA}
sdrsplit CBYYMMDD.???
output:

{STA}.NM.BHZ.D.2005.254.120005
{STA}.NM.BHN.D.2005.254.120005
{STA}.NM.BHE.D.2005.254.120005

Merging data:

Once you have retrieved data from the various sources, you will want to merge the retrieved data with the rest of the data stream to make the complete seed file.

There is a shell script in your work directory sta_merge. Edit the variables: YEAR, PDOY, DOY, STA and COMP. This will take all the day files and the previous day files for a particular station and merge them into one complete day file. The file contents:

#!/bin/sh

#       Merge the dayfile (DOY) with the previous day file (PDOY) for
#       complete day file.
#       Change the variables DOY (Day OF YEAR) and PDOY (Previous Day Of Year).

YEAR=2005
PDOY=211
DOY=212
STA=SLM

for COMP in BHZ BHN BHE
#LHZ LHN LHE
do

qmerge -T -f 2005.${DOY},00:00:00.0000 \
-s 1d \
-o ${STA}.NM.${COMP}.D.2005.${DOY} \
./${STA}.NM.${COMP}.D.${YEAR}.${DOY} \
/cnmsn/${STA}/${COMP}.D/${STA}.NM.${COMP}.D.${YEAR}.${PDOY}.* \
/cnmsn/${STA}/${COMP}.D/${STA}.NM.${COMP}.D.${YEAR}.${DOY}.*

qmerge -n ${STA}.NM.${COMP}.D.2005.${DOY} > ${STA}.NM.${COMP}.D.2005.${DOY}.

Or you can edit an executable file, doit, found in the staging directory. Copy the file to {STA}/WORK.{STA}.???. Edit the file. The contents of the file runs the programs qmerge with options that let you trim the data stream to exactly 1 day (see manual). This example then tells the program to merge yesterday's day file with the day before and then with the retrieved data file that starts at 12:00, and then to name the output file {STA}.NM.BHZ.D.2005.254.

cd /seis/rbh/STAGING/{STA}/WORK.{STA}.254
cp ../../doit .
vi doit
change dates, times, etc.
run doit and merge files

qmerge -T -f 2005.254,00:00:00.0000 \
-s 1d \
-o UTMT.NM.BHZ.D.2005.254 \
/cnmsn/UTMT/BHZ.D/UTMT.NM.BHZ.D.2005.253.???? \
/cnmsn/UTMT/BHZ.D/UTMT.NM.BHZ.D.2005.254.???? \
/cnmsn/UTMT/BHZ.D/UTMT.NM.BHZ.D.2005.254.12????

qmerge -n UTMT.NM.BHZ.D.2005.254 > UTMT.NM.BHZ.D.2005.254.err

Archiving data:

Finally, when all data is accounted for:

su - seismic
password: xxxxxxxx
cd /seis/rbh/STAGING
netmove jday This script moves all data to /net/taumsauk/r/2005/jday

or you can move 1 station at a time:

cd BLO/WORK.BLO.????
mv * /net/taumsauk/r/2005/???
cd ../
rmdir WORK.BLO.???
Repeat process for remaining stations

ls -l /net/taumsauk/r/2005/??? | pg
to make sure all data has been moved

Last updated May 5, 2008