[wxqc] APRS Solar Radiation Data

Leon A Benjamin Leon.A.Benjamin at noaa.gov
Wed Oct 26 14:51:59 CDT 2011


Russ,
MADIS has changed it tune.  It is interested in Solar data.  I'll talk
to you and Randy about this.
-- Leon

Russ Chadwick wrote:
> I forwarded some of the recent discussion on APRS Solar Radiation data to
> Steve Dimse and his comments are below.  This is before I told him that
> about a year ago, I asked MADIS if they wanted to get APRS Solar Radiation
> data and was told "No, MADIS is not interested in receiving APRS Solar
> Radiation data."
>
> Russ
>
> Steve's comments below
>
> No, I'm not on the list, would you be so kind as to forward this for me?
> -----
> When Russ forwarded this message to me tonight it was the first time that
> anyone mentioned to me that stations were collecting solar data and that
> MADIS could accept it. The most certain way to be sure that you will not get
> something is not to ask!
>
> The dollar cost of adding this is not an issue (though there will soon be a
> collection to raise the remaining funds for a new server for findU2, we only
> got halfway there last year and the findU2 server is aging fast). The time
> cost is significant only because there are hundreds of things I'd like to
> fix and add to findU and I have no free time for hobbies at this time.
> Still, this sounds like something I could put at the top of my list and find
> the time (at least the forwarding to MADIS part of it; web pages that
> display the data will probably need to wait until after my retirement August
> 2, 2012).
>
> There is one more major problem though. Weather data is stored in a table
> for each year, wx2011 currently holds more than 140 million records. This
> table is the the most heavily accessed table and the bottleneck of findU. To
> make it work at all requires multiple indices so all queries hit only one
> index. Adding a new field means that every single one of those 140M records
> must be read, expanded, and rewritten to disk, then each of the seven
> indices must be reconstructed. With a 200 GB MySQL table that doesn't happen
> in a few seconds. Just rebuilding the indices would take more than a day,
> and the server cannot be accepting new data during that time.
>
> findU is vicious on disk IO. It requires top end hardware with 6 15,000 RPM
> drives in a RAID 1+0 configuration to work at all. findU1 is the newer
> machine and it is still (barely) capable of running the full load (over 1
> million web pages a day, most of them requiring dozens or even hundreds of
> hits to the database). findU2 is older and not capable of handling the full
> load. Therefore I cannot take findU1 down for a day or two to rebuild and
> index the weather table without denying service to my users.
>
> The only time this is a reasonable possibility would be shortly after
> January 1 when I move a few days of data into a new file for 2012.
> Rebuilding a table with 2 million records is a lot faster than one with 140!
>
> I don't know how long this has been a discussion issue, but I'm only hearing
> about it today and I think a 2 month turnaround isn't so bad. Even if you do
> think it is bad, it is the best I can do.
>
> Steve K4HG
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> wxqc mailing list
> Post messages to wxqc at lists.gladstonefamily.net
> To unsubcribe or change delivery options, please go to:
> http://server.gladstonefamily.net/mailman/listinfo/wxqc
> To search the archives: http://www.google.com/coop/cse?cx=008314629403309390388%3Aknlfnptih9u
>
> The contents of this message are the responsibility of the author.



More information about the wxqc mailing list