Sloppy data presentation

I have a client for whom I massage U.S. government supplied weather data. There is a bewildering amount of data available; I haven't a clue what most of it means: I just go get what they want and slice and dice it to their specs.

The folks who produce this data of course think nothing of changing locations, formats, or anything else. That's not unexpected and there's no point griping about it - it is what it is.

What does cause me to make unpleasant faces is sloppiness in data representation. For example, here's a data line from something I was working on yesterday:

DUAN6 200609191320 65.4 0 -996 9 42.03 0 0.00 0 -996.0 9 -996 9 -996.00 9 -996.00 9 -996 0 -996 9 -996 9

It looks like space separated values, easily parsed in Perl with "split /\s+/", but is it? Here's another sample from the same data source.

ABGV1 200609221150 -999.0 1 -996 1 -996 1 0.00-9999 1-996-9999 1-996-9999 1 -996 1 -996 1 -996 1 -996 1 -996 1

What the heck is that? Values are run together. Let's align the two lines to make it easier to see:

DUAN6 200609191320 65.4 0 -996 9 42.03 0 0.00 0 -996.0 9 -996 9 -996.00 9 -996.00 9 -996 0 -996 9 -996 9
ABGV1 200609221150 -999.0 1 -996 1 -996 1 0.00-9999 1-996-9999 1-996-9999 1 -996 1 -996 1 -996 1 -996 1 -996 1

Hmmm. So it's not fixed length.. but it's not space delimited either. It's just a mess.

This particular mess is easy enough to fix. The only time we get this overlap is with negative numbers, so:

s/-/ -/g;
@line=split /\s+/;

That works. It's annoying, but hardly a "big deal". It is however, in my opinion, real sloppiness in the original programming. If there was no care for being human readable, why space anything out at all? Why not just run it together in pure record form? DUAN6200609191320000065.4 etc.? Apparently there was some intent at presentation, so why not make ALL data lines consistent?

Yeah, I know: grumpy old Unix guys. Ayup, that's me.

Comments?

More Articles by Anthony Lawrence - Find me on Google+

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.