CNMSN Station Data Processing

Quick Reference

Site Map

Last changed June 14, 2008

Broadband Stations:

SLU, CERI and CAPE Networks: Locating Earthquakes

Background

Currently, the operating broadband stations are SLM, BLO, FVM, MPH, OLIL, PBMO, PVMO, SIUC, UALR, USIN and UTMT. Each station consists of the seismometers, Quanterra digitizers, power supplies and telemetry systems. More information on the individual station components can be obtained from the web site:
http://www.eas.slu.edu/Earthquake_Center/NM/
Data is recorded continuously by the Quanterra digitizer and in some cases, sent to the SLU machine wiggles in real time. Some stations have an extra component in the form of a Sparc storage system at the local site. In these cases, the data is sent to the Sparc and then at the end of a day (UTC time) the data is sent over the network to the SLU machine wiggles. On wiggles, and the remote Sparcs, all data is stored in the parent directory /cnmsn (CNMSN - Cooperative New Madrid Seismic Network), containing subdirectories named for each station, which in turn contain subdirectories of SEED data files for each component or channel recorded by the stations. For example:
/cnmsn 
| 
SLM 
| 
| | | | | |
BHZ.D  BHN.D  BHE.D  LHZ.D  LHN.D  LHE.D
| 
SLM.NM.BHZ.D.2008.142.0000, etc.

The data files in /cnmsn/SLM/BHZ.D are in SEED format. SEED names are given as:
/cnmsn/SLM/BHZ.D/{STA}.{NETWORK}.{COMP}.D.{YEAR}.{J-DAY} , i.e.,

SLM.NM.BHZ.D.2008.142.0000.
NM reflects the name of the New Madrid network.

The real-time data is written to an active file on wiggles or the remote Sparc. At the end of the day the active file is renamed for the station, network, channel and date, as shown above.

The same directory structure applies to the other stations. The files remain in this directory for about a month, and then removed after they are archived to /net/taumsauk/r/2008/{J-DAY}.

We want to merge Quanterra data streams into a 24 hour miniseed file. We will archive the BH and LH channels for all stations, with the exception of OLIL, SIUC, USIN and UALR, hi-sample stations that record addtional HH and HL channels. UALR is unique to the other hi-sample stations in that the additional channels are named HH and HN. If there are data gaps, we will need to determine whether or not we can retrieve the the data. In some cases, we will need to login to the remote Sparcs and retrieve the day files from that system. And in other cases, we will need to go directly to the Quanterra digitizer to retrieve the data. There will be times when we cannot retrieve the missiing data. In the case of OLIL, SIUC UALR and USIN, we will only be able to retrieve day files if they rest on the Sparc at the site.

Programs such as sdrsplit and qmerge are used to manipulate the data stream further. Scripts automate the process of retrieving data and later, merging and renaming the data files.

Several tasks are performed daily for each stations' data. For archiving purposes, a complete 24-hour day file is created. If there are any gaps in the data, steps are taken to retrieve the missing gaps, if possible, so as to have the most complete day file as possible. These files are then moved to /net/taumsauk/r/2008/{J-DAY}, where the files are archived and will eventually be forwarded to IRIS.

Programs and scripts used to manipulate the SEED data will be discussed in detail.

Back to top

Data Quality Control

To check the data acquisitions:

login to wiggles
ssh wiggles;
move to /seis/rbh/STAGING and run the script NMakeDay : This script invokes the program qmerge (see manual) which merges yesterday's and the previous day's data streams stored in /cnmsn/{STA}/{COMP.D} and merges them into a 24-hour seed file.

To get a complete day for May 22, i.e., May 22, 2008 from 00:00:00 to 23:59:99, run:

NMakeDay 142
The script will merge
/cnmsn/SLM/BHZ.D/SLM.NM.BHZ.D.2008.142.0000 with
/cnmsn/SLM/BHZ.D/SLM.NM.BHZ.D.2008.141.0005,
renames the day files and moves them to the sub-directory SLM/WORK.SLM.142 .
The directory structure:
/seis/rbh/STAGING
| 
| | | | |
SLM  BLO  MPH  UTMT  OLIL
| | | | |
WORK.SLM.142 WORK.BLO.142 WORK.MPH.142  WORK.UTMT.142 WORK.OLIL.142
| | | | |
SLM.NM.BHZ.D.2008.142
SLM.NM.BHZ.D.2008.142.err ...

Change directory to SLM/WORK.SLM.142 to view the output files.
The seed file SLM.NM.BHZ.D.2008.142 and the *err file, containing the output from qmerge -n SLM.NM.BHZ.D.2008.142 are found here. The same directory structure applies to the other stations. Check the *err files to make sure the data is complete, that there are no gaps. For example:

more SLM.NM.BHN.D.2008.142.err

produces:
SLM BHN: rate=20 (2008.142 00:00:00.0398 to 2008.143 00:00:00.0405) : 1728000 points, 0.6 msec correction, (min,max,max_step = 0.0,0.6,0.0 msec)

This is a good way to check if the day's data is complete. It reads that the BHN file contains data from 2008.142 00:00:00.0398 to 2008.143 00:00:00.0405, a 24 hour day. It contains 1728000 points: 864000 seconds (24 hours) x 20 samples/seconds. It also contains clock error information.

A gap in data stream may look something like:

SLM.NM.BHE. rate=20 (2008.142 00:00:00.0104 to 2008.142 07:18:17.1608) : 525943 points, 0.4 msec correction, (min,max,max_step = 0.0,0.4,0.2 msec)
SLM.NM.BHE. rate=20 (2008.142 11:33:49.6608 to 2008.143 00:00:00.0110) : 895407 points, 0.2 msec correction, (min,max,max_step = 0.0,0.2,0.2 msec)

(this is the output from qmerge -n SLM.NM.BHE.D.2008.142). This indicates that there is a data gap for this channel from approximately 07:00 to 12:00. You will need to retrieve this data, if possible.

If you have a data gap, run

qlog /cnmsn/STA/LOG.L/*2008.???.* | pg

This lets you look at the log files and check for information regarding time gaps. Page through the file to the time the outage is indicated. #ERR or #MSG flags will indicate a system error and no data will be retrievable during that time. If there is no idication of system errors, then we should be able to retrieve the data and merge the data streams to complete the day files. Refer to the Data Retrieval Sections for a particular station.

Sometimes, you will find that qmerge cannot open a data file. The program complains that it can't read the header. Go to the station's parent directory (/cnmsn/SLM/BHE.D) and run the program msfix on the seed file. Now you can run qmerge on the seed file, though there will now be a gap in the data where msfix read past the bad header. Run qlog and proceed as above.

cd /cnmsn/SLM/BHE.D
msfix SLM.NM.BHE.D.2004.142.0001
qmerge -n SLM.NM.BHE.D.2004.142.0001

If all the SLM.NM.{COMP}.D.2008.142.err files show complete days, i.e., day 142, 00:00:00 thru 143 00:00:00, then the files are ready to be archived.

Next, move all the *err files and all the seed files to /net/taumsauk/r/2008/{JDAY}
/seis/rbh/STAGING/netmove ??? {JDAY}
netmove is a script that will move the data, or you can move the data manually:

cd /seis/rbh/STAGING/{STA}/WORK.{STA}.{JDAY}
mv * /net/taumsauk/r/2008/142
cd ../
rmdir WORK.SLM.142
Repeat the above steps for the BLO, FVM, MPH, UTMT, PVMO, PBMO, SIUC*, UALR*, USIN*and OLIL* .

* These stations will contain additional channel data. When you run the script NMakeDay it will produce BH, HH, HL, HN and LH channels. These channels get moved to the archive directory along with the others.

Back to top

Data Retrieval: SLM (Kermit)

The broadband stations sometimes experience gaps in the data due to network problems, or the system disk fills up. If you have time gaps in your seed data, first check the station logs.
Run: qlog /cnmsn/SLM/LOG.L/SLM.NM.LOG.L.2008.???.* | pg
This will tell you if there is an ERR# or MSG# flag at the time of the data gap. If there is, then the missing data cannot be retrieved. If not, or if only part of a day comes over, then you need to retrieve the data. Kermit is a way we can retrieve the SLM data.

To retrieve SLM data:

logon to wiggles
login as spyder
su - spyder
passwd: xxxxxxxx (same as seismic)
cd to directory that you want to store the data in:
cd /cnmsn/SLM
type: kermit

C-Kermit> setmanual2
C-Kermit> con
C-Kermit> at (OK)
C-Kermit> atdt 9773127
login as seed
passwd: xxxx

You are now directly connected to the Quanterra. Type ? to display menu options. Type K BH? or K LH? and proceed to retrieve the data channels you need. (Depending on how big the files are, this could take a while.) This retrieves the CB* or CL* files and stores them directly to the local directory on wiggles. If you have large gaps in the data, request 4 hours worth of the BH channel data at a time. You can merge the data streams later. Anything larger usually results in hang ups. LH channels produce smaller files, so you can retrieve an entire day at one session.

..............................................................

QUANTERRA SCREEN

Command? K BH?

UTC starting date for VBB data:
yy/mm/dd hh:mm:ss
? 02/11/09 00:00:00

UTC ending date for VBB data:
yy/mm/dd hh:mm:ss
? 02/11/09 04:00:00

...search interval 2008/11/09 00:00:00 - 2008/11/09 04:00:00

1431 total records selected.

...creating local file

....writing local file

...creating archive file "CB021109.A00"
BHN-11/08 23:59+ , BHE-11/08 23:59+ , BHN-11/09 00:01....

"will start sending in...escape back to kermit ( CTRL\C ) and type receive "

CTRL\C

..............................................................

C-Kermit> receive


Data transmission begins. This could take a while, so if running overnite, lock the terminal that you're working on.

When finished, return to kermit :

CNTRL\C
C-Kermit> quit
Process the retrieved data using sdrsplit and qmerge (see Merging and Archive Files )

Back to top

Data Retrieval: MPH, BLO

When network problems occur that cause gaps in the data files or missing data files, you'll need to retrieve it from either the station's remote Sparc or the Quanterra. MPH and BLO have Sparc drives at the site. You need to login to the Sparc and scp the day files to wiggles, or use kermit to send the data from the Quanterra to the Sparc, where it can then be scp'd over to wiggles.

First, on wiggles , login to the remote machine as seismic ; use a secure shell login:

MPH: ssh tgpmph.ceri.memphis.edu -l seismic or
BLO (tgpblo.geology.indiana.edu)
passwd: xxxxxxxx
When the day files are not sent to wiggles. You can look for these files on the remote Sparc by:
cd /cnmsn/MPH/BHZ.D (same directory structure as wiggles )

ls will list active file (current day file) and maybe the missing day file, i.e.,
MPH.NM.BHZ.D.2008.142.0000

scp MPH.NM.BHZ.D.2008.142.0000 wiggles.eas.slu.edu:/cnmsn/MPH/BHZ.D

Repeat this process for all BH and LH data channels and logout.

If the day file is not there, or you need to retrieve small gaps, you can now use kermit to retrieve data from the Quanterra and send to the remote Sparc. From there it will be scp'd to wiggles.

Move to /cnmsn/MPH, where you'll want the files written. Type:

kermit (wait for prompt)
C-Kermit> con
(there is a .kermitrc file that has the port and speed information. This should tell you: port /dev/cum03; speed 9600 baud) You are now in the Quanterra menu. If you find yourself in some other screen, type V and that should take you back to the archive screen. From the remote Sparc, when retrieving an entire day's worth of data, you will want to divide the time retrievals into 2 or more files. Type:
k MPH.BH? (kermit transmission and channel name)
from: YY/MM/DD 00:00:00
to: YY/MM/DD 12:00:00 (repeat for the next 12 hours worth of data)
Machine will display available files and write files. Machine will then prompt, "will start sending in..."

Type:

CTRL\C (return to kermit)
C-Kermit> receive
Machine will start sending the files to the Sparc: cb020522.a00 cb020522.m00 ...

Return to kermit

CTRL\C
C-Kermit> con
and repeat as above for as many channels or gaps you need to retrieve.

When finished:

CTRL\C (return to kermit)
C-Kermit> quit
Back at Sparc,
ls
to make sure you're in the data directory.
scp cb020522.a00 wiggles.eas.slu.edu:/cnmsn/MPH
scp cb020522.m00 wiggles.eas.slu.edu:/cnmsn/MPH ...
Once they're on wiggles , run sdrsplit on the files and qmerge the seed files to make sure the data is there.

On the remote Sparc:

Remove the cb020522.a00 cb020522.m00 ... files from the Sparc

CNTRL D (logout)

Merge and archive the files.

Back to top

Data Retrieval: UTMT

If you have time gaps in your seed data, first check the station logs (run qlog). If the logs indicate no problem, then a network error occurred and you need to retrieve the data. LH data can be retrieved in 1 day records. BH data should be retrieved in records no larger than 6 hours.

There is a script that telnets to UTMT and then ftp's the archive files over automatically. The script name is grab. A copy of the contents can be viewed here.

The directory /cnsmn/STA is where the archive files will be written to. You may get all the bh(bh\?) channels or the lh(lh\?) channels.

logon to wiggles (as seismic):

su - seismic
passwd: xxxxxxxx
Run the script:
grab tgputmt.utm.edu lh\? 01/09/05,00:00:00 01/09/05,05:00:00 /cnmsn/UTMT/grab.log /cnmsn/UTMT

*************************** 

 SYNTAX
    grab [seed mask] [date1] [date2] [logfile] [ftp directory]

 ARGUMENTS:
   (1) seednames mask for RETRIEVE
   (2) date1   /*according to ultra-sheer software convention */
   (3) date2
   (4) complete path name of log file
   (5) complete path name of directory to place file (ftp)
The archive file will be ftp'd to wiggles where further processing will take place. See Merging and Archive Files.

The automation process may stop working at some point. For a description of the procedure to manually retrieve UTMT data, GO.

Back to top

Data Retrieval: PVMO, FVM, PBMO (kermit)

To retrieve PVMO data:
logon to wiggles
login as seismic
su - seismic
passwd: xxxxxxxx
cd to directory that you want to store the data in:
cd /cnmsn/PVMO
type: kermit

C-Kermit> setpvmo
C-Kermit> connect

To retrieve FVM data:
logon to wiggles
login as seismic
su - seismic
passwd: xxxxxxxx
cd to directory that you want to store the data in:
cd /cnmsn/FVM
type: kermit

C-Kermit> setisdn
C-Kermit> connect

To retrieve PBMO data:
logon to wiggles
login as seismic
su - seismic
passwd: xxxxxxxx
cd to directory that you want to store the data in:
cd /cnmsn/PBMO
type: kermit

C-Kermit> setpbmo
C-Kermit> connect

You are now directly connected to the Quanterra. Type ? to display menu options. Type K BH? or K LH? and proceed to retrieve the data channels you need. (Depending on how big the files are, this could take a while.) This retrieves the CB* or CL* files and stores them directly to the local directory on wiggles. If you have large gaps in the data, request 6 hours worth of the BH channel data at a time. You can merge the data streams later. Anything larger usually results in problems. LH channels produce smaller files, so you can retrieve an entire day at one session.

..............................................................

QUANTERRA SCREEN

Command? K BH?

UTC starting date for VBB data:
yy/mm/dd hh:mm:ss
? 02/11/09 00:00:00

UTC ending date for VBB data:
yy/mm/dd hh:mm:ss
? 02/11/09 04:00:00

...search interval 2008/11/09 00:00:00 - 2008/11/09 04:00:00

1431 total records selected.

...creating local file

....writing local file

...creating archive file "CB021109.A00"
BHN-11/08 23:59+ , BHE-11/08 23:59+ , BHN-11/09 00:01....

"will start sending in...escape back to kermit (CTRL\C) and type receive "

CTRL\C

..............................................................

C-Kermit> receive


Data transmission begins.

When finished, return to kermit :

CNTRL\C
C-Kermit> quit
Process the retrieved data using sdrsplit and qmerge (see Merging and Archive Files )

If data transmission stops:

logon to wiggles
login as seismic
su - seismic
passwd: xxxxxxxx
netmon -t FVM (or PVMO)
This stops data logger and comserve
ps -ef | grep "FVM" (PVMO)
checks that the processes are stopped
netmon -s FVM (or PVMO)
Should restart datalogger and comserve
ps -ef | grep "FVM" (PVMO)
to check that processes are running

Back to top

Retrieving OLIL, SIUC, USIN and UALR Data

There shouldn't be any gaps in the data, but if there are, there is no way to retrieve the missing gap. We can only talk to the Sparc and retrieve day files for these stations.

If an entire day file did not come over, first log in to the remote Sparc, as seismic; use a secure shell login:

OLIL:
ssh occrumble.iecc.edu -l seismic
psswd: xxxxxxxx
cd /cnmsn/OLIL
 run the script sendalltowiggles to send the data to wiggles:
sendalltowiggles ??? {JDAY}
CNTRL D

SIUC:
ssh siucbbs.geo.siu.edu -l seismic
psswd: xxxxxxxx
cd /cnmsn/SIUC
 run the script sendalltowiggles to send the data to wiggles:
sendalltowiggles ??? {JDAY}
CNTRL D

USIN:
ssh tgpusin.usi.edu -l seismic
psswd: xxxxxxxx
cd /cnmsn/USIN
 run the script sendalltowiggles to send the data to wiggles:
sendalltowiggles ??? {JDAY}
CNTRL D

UALR:
Run the script sendonedaytowiggles to send ONLY the files from that day to wiggles, since the day files are not removed from this machine.
ssh ualrbbs.ualr.edu -l seismic
cd /cnmsn/UALR
sendonedaytowiggles ??? {JDAY}
CNTRL D

Back to top

Last updated September 4, 2008