[wxqc] APRS Solar Radiation Data

Mike scn at mvtel.net
Wed Oct 26 12:59:59 CDT 2011

This all sounds very promising to me.   I will contribute a little again
next time my SS check comes in again.   With all the interest in solar PV
and HW systems going on,  I think more solar data could be useful for a lot
of reasons.   And as for turnaround, where else but here could such a
project be put in place within a few months without mullti-billion dollar
grants from the government or corporations!   My solar is already in place
so I will be happy to share that data with everyone.   There a only a few
official solar reporting site around the country.

-----Original Message-----
From: wxqc-bounces at lists.gladstonefamily.net
[mailto:wxqc-bounces at lists.gladstonefamily.net] On Behalf Of Russ Chadwick
Sent: Wednesday, October 26, 2011 12:42 PM
To: wxqc at lists.gladstonefamily.net
Subject: [wxqc] APRS Solar Radiation Data

I forwarded some of the recent discussion on APRS Solar Radiation data to
Steve Dimse and his comments are below.  This is before I told him that
about a year ago, I asked MADIS if they wanted to get APRS Solar Radiation
data and was told "No, MADIS is not interested in receiving APRS Solar
Radiation data."


Steve's comments below

No, I'm not on the list, would you be so kind as to forward this for me?
When Russ forwarded this message to me tonight it was the first time that
anyone mentioned to me that stations were collecting solar data and that
MADIS could accept it. The most certain way to be sure that you will not get
something is not to ask!

The dollar cost of adding this is not an issue (though there will soon be a
collection to raise the remaining funds for a new server for findU2, we only
got halfway there last year and the findU2 server is aging fast). The time
cost is significant only because there are hundreds of things I'd like to
fix and add to findU and I have no free time for hobbies at this time.
Still, this sounds like something I could put at the top of my list and find
the time (at least the forwarding to MADIS part of it; web pages that
display the data will probably need to wait until after my retirement August
2, 2012).

There is one more major problem though. Weather data is stored in a table
for each year, wx2011 currently holds more than 140 million records. This
table is the the most heavily accessed table and the bottleneck of findU. To
make it work at all requires multiple indices so all queries hit only one
index. Adding a new field means that every single one of those 140M records
must be read, expanded, and rewritten to disk, then each of the seven
indices must be reconstructed. With a 200 GB MySQL table that doesn't happen
in a few seconds. Just rebuilding the indices would take more than a day,
and the server cannot be accepting new data during that time.

findU is vicious on disk IO. It requires top end hardware with 6 15,000 RPM
drives in a RAID 1+0 configuration to work at all. findU1 is the newer
machine and it is still (barely) capable of running the full load (over 1
million web pages a day, most of them requiring dozens or even hundreds of
hits to the database). findU2 is older and not capable of handling the full
load. Therefore I cannot take findU1 down for a day or two to rebuild and
index the weather table without denying service to my users.

The only time this is a reasonable possibility would be shortly after
January 1 when I move a few days of data into a new file for 2012.
Rebuilding a table with 2 million records is a lot faster than one with 140!

I don't know how long this has been a discussion issue, but I'm only hearing
about it today and I think a 2 month turnaround isn't so bad. Even if you do
think it is bad, it is the best I can do.

Steve K4HG

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://server.gladstonefamily.net/pipermail/wxqc/attachments/20111026/069df6ba/attachment.html>

More information about the wxqc mailing list