[wxqc] APRS Solar Radiation Data
Leon A Benjamin
Leon.A.Benjamin at noaa.gov
Wed Oct 26 14:51:59 CDT 2011
MADIS has changed it tune. It is interested in Solar data. I'll talk
to you and Randy about this.
Russ Chadwick wrote:
> I forwarded some of the recent discussion on APRS Solar Radiation data to
> Steve Dimse and his comments are below. This is before I told him that
> about a year ago, I asked MADIS if they wanted to get APRS Solar Radiation
> data and was told "No, MADIS is not interested in receiving APRS Solar
> Radiation data."
> Steve's comments below
> No, I'm not on the list, would you be so kind as to forward this for me?
> When Russ forwarded this message to me tonight it was the first time that
> anyone mentioned to me that stations were collecting solar data and that
> MADIS could accept it. The most certain way to be sure that you will not get
> something is not to ask!
> The dollar cost of adding this is not an issue (though there will soon be a
> collection to raise the remaining funds for a new server for findU2, we only
> got halfway there last year and the findU2 server is aging fast). The time
> cost is significant only because there are hundreds of things I'd like to
> fix and add to findU and I have no free time for hobbies at this time.
> Still, this sounds like something I could put at the top of my list and find
> the time (at least the forwarding to MADIS part of it; web pages that
> display the data will probably need to wait until after my retirement August
> 2, 2012).
> There is one more major problem though. Weather data is stored in a table
> for each year, wx2011 currently holds more than 140 million records. This
> table is the the most heavily accessed table and the bottleneck of findU. To
> make it work at all requires multiple indices so all queries hit only one
> index. Adding a new field means that every single one of those 140M records
> must be read, expanded, and rewritten to disk, then each of the seven
> indices must be reconstructed. With a 200 GB MySQL table that doesn't happen
> in a few seconds. Just rebuilding the indices would take more than a day,
> and the server cannot be accepting new data during that time.
> findU is vicious on disk IO. It requires top end hardware with 6 15,000 RPM
> drives in a RAID 1+0 configuration to work at all. findU1 is the newer
> machine and it is still (barely) capable of running the full load (over 1
> million web pages a day, most of them requiring dozens or even hundreds of
> hits to the database). findU2 is older and not capable of handling the full
> load. Therefore I cannot take findU1 down for a day or two to rebuild and
> index the weather table without denying service to my users.
> The only time this is a reasonable possibility would be shortly after
> January 1 when I move a few days of data into a new file for 2012.
> Rebuilding a table with 2 million records is a lot faster than one with 140!
> I don't know how long this has been a discussion issue, but I'm only hearing
> about it today and I think a 2 month turnaround isn't so bad. Even if you do
> think it is bad, it is the best I can do.
> Steve K4HG
> wxqc mailing list
> Post messages to wxqc at lists.gladstonefamily.net
> To unsubcribe or change delivery options, please go to:
> To search the archives: http://www.google.com/coop/cse?cx=008314629403309390388%3Aknlfnptih9u
> The contents of this message are the responsibility of the author.
More information about the wxqc