Weekly, Monthly, and Quarterly update data sets are also available in both formats. Visitors can conserve bandwidth by downloading updates on a regular basis rather than downloading the full data set repeatedly.
The easiest way to deal with the data is to use an
integrated package, such as my new Trademan
for updates, displays, and technical analysis charts. If you are not interested
in Trademan for some reason, the following discussion may help you deal
with the data your own way.
The data sets are compressed archive files with
the ".zip" extension. They may be uncompressed using "pkunzip" from
PKWARE,
Inc., "unzip" from Info-Zip,
and doubtless others that are compatible. The historical data set
format and the daily update format offer different possibilities and problems.
Files obtained by selecting from the alphabetical and sectorized tables are presented as plain text. The file for a single stock contains one record per line of text corresponding to the data for that date. The record is arranged into fields representing respectively the Date, Ticker, Open, High, Low, Close, Volume for that day. The fields are delimited by commas, and the records are delimited by <cr><lf>. The text files may be loaded directly into Excel and similar spreadsheet programs.
If you have a stock analysis program that uses the Metastock/Computrac data format, the historical data files can be converted for use with your program. Two shareware utilities that convert from ASCII to Metastock format are "makems," which I found via shareware.com, and DataManager. These mentions are intended as (hopefully) helpful suggestions rather than as endorsements.
With the full data set and updates, the historical data format is probably best handled by leaving it in its compressed form and using simple utilities to extract data as needed. These data may then be loaded into spreadsheets or converted into another format suitable for processing by stock analysis software. For example, a MSDOS (Windows3.x, Window95/NT) user could extract all of the data for 31 July 1997 into a file named "up970731.txt" by typing:
pkunzip -c sp500.zip | find "970731" > up970731.txt
(Info-Zip users would type "unzip -p" instead of "pkunzip -c.") As another simple example, to extract a history data file for Motorola, the same pkunzip user would type:
pkunzip -c sp500.zip | find ",MOT," > mot.txt
Note that the commas on either side of the ticker are generally necessary so that tickers that share consecutive letters are rendered unambiguous.
It is relatively difficult to update a historical data set, especially in MSDOS/Windows. If you have perl for Windows or wish to get it, you may use a little program I have written to update a "base" data set, such as the set of yearlong histories, with an "update" data set. This program will merge the two files, throwing away duplicate records and returning the result sorted with respect to date and ticker. The invocation is as follows:
perl -w upd_hist.pl <base>.zip <update>.zip <new_base>.zip
Replace the bracketed file names with your base, update, and output file names, respectively. The output file can be the same as one of the input files, although this may be not advisable.
If you are interested in a powerful and friendly language for manipulating data, perl is a good choice. However, there are options that do not involve installing an entire language package. Several years ago, a number of generous and able people decided to port Unix tools to MSDOS. One of these tools, a program to replace the MSDOS sort (which can handle files up to 64kB!), is a part of the textutils package written as part of this effort. This is freeware, covered by the Gnu Copyleft license. Please be sure to unzip this file in its own directory or use the -d option, since it will unpack about 50 files. I have archived with the textutils package a script named "merge" to perform the actual update. If you save nothing else, at least retain sort.exe and csplit.exe, since these are used by the merge script.
Merge a base file with an update file as follows:
merge <base>.zip <update>.zip <new_base>.zip
Those with a Unix OS can probably think of a number of ways to accomplish the historical file update without using perl. Feel free to use my program anyway, since perl is commonly available on Unix systems. You will probably want to modify the script to use pipelining so that there is no need for unzipped files on disk at any time.
The data are also available as daily update sets. These data sets consist of files which are archived together, one for each day. The individual files contain a record for each stock, organized as Ticker, Open, High, Low, Close, Volume, delimited by commas. The file name itself carries the date. Once unpacked with pkunzip or unzip, the individual files may be loaded directly for use with StockWiz or converted for use with MetaStock and other programs that use the same format.
The great advantage to the daily update format is that it is easy to update. Use pkzip or zip to add an update file to the base:
pkunzip -@updates <update>.zip
pkunzip <update>.zip
pkzip -m <base>.zip @updates
Replace the bracketed file names with your update and base file names, respectively. The preceding commands unpack the update file into its constituent files and then add them to the base file. The unpacked files are then deleted. Use the "-a" option instead of the "-m" option if you do not want to delete the constituent files.
Feedback
Please send comments, suggestions, and questions to stocks@swcp.com.
Disclaimers