Web Cam Madness

Sharing my data with Weather Underground includes sharing my Webcam images for them to display alongside my weather data.  Recently I checked on their site to verify the Webcam images were being uploaded and kept seeing that my camera was disabled.  They were supposed to be grabbing the image using a URL I supplied them with.  I would check the URL I had given them for my camera image to make sure it was working.  Since it is the same URL used to display the image on my own pages I felt pretty confident that the image was available.  But every time I enabled my camera on their Web site it would become disabled within a few hours to a day.

In order to remedy this problem I chose to change from having them retrieve my camera image to me uploading it via FTP.  At least that way I would have some degree of control over when the images were being delivered.  That led to a couple of issues I needed to address.  The first was that the Webcam software I was using, Yawcam, could only do FTP to one site.  I was already using that capability to upload images to my own Web site.  That required that I create an FTP script to upload the image to Weather Underground.  After creating the script, I set Yawcam to save off the current Webcam image to a folder on my server’s hard drive.  Then I set up Windows task scheduler to run the FTP script every two minutes.  Everything looked good so I left it to run.

When I checked back later that day I noticed that the image had stopped changing on Weather Underground.  I checked my logs and saw that the FTP script was still working.  Checking my server I discovered that Yawcam had stopped updating the still image it was supposed to be capturing from the webcam.  The image being FTP’d to my Web site was fine, and the streaming video was also fine.  I restarted Yawcam and monitored the image being saved.  After several hours it stopped updating again.  This went on for the next few hours before I decided on a different approach to work around this apparent bug in Yawcam.

Yawcam has a feature where it can act as a Web server and serve up the latest image from the Webcam as a JPEG file.  I enabled this feature and set out to find a way to retrieve the image from Yawcam directly.  I ended up writing a Powershell script to retrieve the image and FTP it to the Weather Underground site.  I let it run for a day to make sure it was stable.  The image updated correctly and I was satisfied with how the script was running.

Now I started to get carried away with myself, like I tend to do.  I decided it would be nice to have a time-lapse video of the day’s webcam images.  I modified the script to create a folder using the date and save the images in that folder using a sequence number (the count of images in the folder + 1).  That permitted me to use ffmpeg to build an MPEG video of the images.  I set the script to run once per minute and save off the images in sequence.  I then wrote another script that runs ffmpeg to create the time lapse video and set it to run in the middle of the night on yesterday’s image sequence.  Finally, I had the script FTP the MPEG file it had just created to my weather Web site.  A link on my Web page permits viewing that video.  The Web site only retains one day of time lapse video.  The nightly upload overwrites the video from the prior day.  Everything appears to be running smoothly.  I still plan to write a script to clean up the sequence of images off of my hard drive after an expiration period has passed.

I ordered a new humidity sensor, as mine appears to be suspect.  I probably won’t have an opportunity to install the sensor chip on the humidity circuit board until the weather warms up a bit.  Always something to do.  I also have been preparing a Raspberry Pi to use as the new station computer.  That will come when I get some free time and motivation, or my current station computer quits working for me.  We’ll see which occurs first.

 

Web Site Software Updated

I just posted another update to the Web site software.  Several things have changed, hopefully for the better.

1) I added a configuration file for the JavaScript side of the house.  It is generated by the PHP code, which makes it a lot easier to keep the two in sync.  The new file is config/config.js.php.

2) I did some testing in other browsers and learned that the Knockout “foreach” construct was failing in some browsers.  I tracked it down to incompatibilities between knockout and jQuery.  The site now uses jQuery 1.11.0 and knockout 3.0.  I also had to add jQuery migrate 1.2.1 to the mix to keep some of the browsers happy.  Unfortunately the slide effect and a few other special effects I was using doesn’t work correctly with knockout in all browsers so I stopped using them.  All the browsers I have tested now work, although my browser list and testing is not comprehensive.

3) I revised the data model for the weather forecast to make it simpler to work with.  I updated the forecast display in both PHP and JavaScript, as well as the page that returns the JSON representation of the data.

Enjoy.  I may do some more work with the weather statistics JSON representation to only pass back the data that is actually being displayed on the We site.  That will take some analysis and isn’t a high priority at the moment.  It also looks like my humidity sensor may have an issue as it has been reporting 100% for at least a day now.  It’s too cold to do much about it right now, and the NWS is predicting 6-10 inches of snow in the next 24 hours.  I’ll keep an eye on the sensor and order parts if it keeps looking like I have a problem.

Latest Web Site Software Posted

I made a few more changes to the Web site software.  The NWS has a system of color codes for all of their watches, warnings, and alerts.  So I decided to implement that system in my Web site.  Now the banner at the top of the page will display any active alerts in the color chosen by the NWS.  This isn’t critical, and sometimes the colors don’t even look that great.  But I chose to conform, at least for now.  I tested it during our most recent winter weather and the banner did display several different colors as the alerts changed from watch to advisory and warning.

I also made some modifications to the configuration file for the PHP code.  I added more comments to explain what the different parameters are for.  I also added parameters for several items that were hard coded in the application.  I’m still thinking about having a similar configuration file for the JavaScript side of things.  The best thing to do would be to have the PHP code generate the JavaScript code.  I will probably work on that in the next week or so, depending on when I can get some free time.  I also want to do some work on the mobile page as it is pretty bare bones and doesn’t really look all that great.

I posted the updated Web site code as it stands today.  If anybody does download it let me know by email.

Back To Work On The Web Site

I must have gotten my energy back as I’ve made several more updates to the Web site.  I put the newest advisories Web page into production.  I also did some minor tweaking to make some of the transitions look a little smoother.  Alerts slide up and down rather than just popping in and out of view.  I also toyed around with ways to highlight changed values when the weather data updates dynamically.  I tried making the text fade in and out.  I also tried having it appear in a different color for a few seconds and then change color to match the rest of the page.  That particular effect is used by Weather Underground and by several other personal weather pages.  But in the end I decided I didn’t really like it and put things back so that updated values just change to the new value without drawing any attention.

The next thing I did was test my pages on a browser that doesn’t have JavaScript to see how they behave.  I had to make a few changes to make sure there is static text displayed in those browsers since they can’t update the page dynamically through JavaScript code.  So things like alerts don’t update when they change, but they do display whatever was in effect when the page is first loaded.  Some of these changes required having static and dynamic sections on a page.  The static sections display by default.  JavaScript code hides the static sections and displays the dynamic portions.  Browsers that do not have JavaScript will display the static portion as they can’t run the code to display the dynamic portions.  So now I have some new browsers to test with when I update my pages.  And yes, I do still need to zip and post the updated code to the download area of the Web site.

Finally, I have updated my Raspberry Pi in preparation for using it as my newest weather station computer.  My current one works fine.  My only complaints are that it draws more power than the Pi and won’t run as long on battery backup.  I am adding an external hard drive so that the constant updates to the weather data don’t “wear out” the SD card.  Hopefully time will allow me to complete that project in the next few weeks.

More Bug Squashing

Now that I was paying more attention to the Web site, I decided to make the alerts page more dynamic in order to address another nagging issue.  If an NWS alert went into effect or expired while a user had the page open, the alert would display or clear itself dynamically on the banner at the top of the page.  All of my pages have this capability.  The issue is that the alert details displayed below on the weather advisories page do not update dynamically to match the banner.  Also, the weather discussion and weather outlook were static and didn’t update over time if the page was left open.  So I set out to make the entire page update dynamically.  Using Knockout made short work of that item.  I just needed to build server side pages to retrieve and return the data in JSON format.  On the client side I modified the page to periodically retrieve the alerts, details, and discussion information and update the display.  I got that working well and decided I should confirm that everything was working in other browsers.

Firefox and Chrome worked fine.  Then I tried IE11.  Not only did this page not display correctly, but none of my other pages were updating either.  This puzzled me because I was sure it was working previously.  I must not have ever tested with IE11.  So now it was time for some frantic debugging to take place.  I eventually narrowed it down to a custom event I was using that wasn’t firing when the data was retrieved.  This was the event that caused the page to display any updated data.  If I refreshed the page the event would fire correctly.  It only happened on the initial load of the page.  So none of my dynamic pages were updating dynamically in IE11.  Not good.

After I pondered this for a while I wondered why I was using an event in the first place.  I’m pretty sure it had to do with the fact that there were different schedules for retrieving the different types of data, and the different functions that got called to refresh the page.  At one point in the past I had consolidated all of those functions, so now i had the ability to just call it directly without triggering an event.  I made that change and all the pages worked in all the browsers I tested.  Success!

During this process I also decided to update the versions of Minify, jQuery, and Knockout I was using to the latest versions.  I didn’t really see a difference from that change, but I hadn’t updated them for a long time so it was overdue.  I also took the opportunity to add a bit more error checking and console logging to my JavaScript code.  That should help debugging in the future.I also added a local alerts page that displays alerts created by our local county.  I ended up moving the Government status updates off of the traffic page and adding it to the local alerts page as well.

I still haven’t updated the source code zip file on the software Web page.  Hopefully that will come soon.  I also haven’t put my new and more dynamic weather advisories page into production yet.  Testing has been fine so far, but i may want to tweak a few things before I release it.

Software Update

I haven’t added a blog entry for quite a few months now.  The weather station has been performing well on “auto-pilot” for all of this time and I have no complaints.  But there were some nagging issues with the software I finally got around to correcting.

The first issue was that it didn’t handle cases where it was unable to retrieve a valid forecast from the NWS very gracefully.  At least not in all cases that were occurring.  I had some error checking in the code, but I was able to improve on it quite a bit.  Now when the software is either unable to retrieve the forecast or it determines that the forecast XML might not be valid it retains the prior forecast, saves the error information in an error file, and does a retry the next time the forecast is requested.  Prior to making this update it was possible for the software to fail parsing the forecast XML.

The second issue was not really a problem, just an enhancement.  When the software retrieved weather advisories sometimes there were additional details available.  The software now retrieves those details when available and formats the display a bit more cleanly.

The third issue was that there was no password protection on my page that received the weather data from my weather station server.  It previously did some minor error checking in an attempt to prevent malicious people from uploading bogus data.  The page is now password protected which adds a bit more protection from receiving bogus data from an unknown source.  This is certainly not iron-clad security, but it is quite a bit better than the security scheme I had in place before now.

I will get around to uploading the newer files to the site as soon as I can.  I see that a few people have downloaded the software.  Hopefully you found it useful.

Software Released

I chose to make the software that powers my weather station as well as the web site available for others to use if they so desire.  The software is available at http://www.cloppermillweather.org/software.php.  If anybody decides to try it let me know how it went.  It isn’t very polished, so if you aren’t interested in creating or modifying configuration files by hand it may not be the best choice for you.

Favicon

Just for grins, I added a favicon.ico to my weather web site so that browsers can show a little icon when the a user bookmarks a page on that site.  The icon also shows up in the browser’s tab for that page or in the address bar, depending on the browser.  That was all simple enough, but I wanted to attempt to take it a step further.  So I added some code to the class that retrieves the weather forecast that is displayed on the web site.  The new code converts the image for the current weather into an icon and stores it as favicon.ico.  I also added some code to the Web page that refreshes the icon every time the forecast is refreshed.  So now the favicon shows the current weather forecast.  This works really well in Firefox and Chrome. IE doesn’t always refresh when I would expect it to.  So not a total success.  But it was entertaining.  I may come back to it sometime and see if I can get IE to refresh a bit better.

Webcam Update

The weather station has been mostly on auto-pilot for several months now.  I do still check the logs and keep an eye on things, but it hasn’t required any real intervention on my part.  I’ll consider that a good thing as my attention has been focused elsewhere for the last few months.  The only real issue I was having with the weather station was the webcam software.  I had begun experiencing periodic lockups of the software where the image was no longer being captured or uploaded to my web site.  I had been using a trial version of Conquercam, which is not free software, so it was time to either pay up or find an alternative.

In the past I had looked at several free webcam packages.  My favorite had been Yawcam.  I don’t remember all the details, but I remember I abandoned that software because I was having to unplug and plug in the webcam fairly often to reset the driver.  My server had a fairly fresh install of Windows Server 2012 Essentials, so when Conquercam began having issues I decided to give Yawcam another chance.  I’m happy to say that Yawcam has performed quite well for me since I switched back to it.  It also has the advantage of being able to provide better streaming support.  For Conquercam I was using a page that refreshed the image every second rather than doing actual streaming.  Yawcam provides a Web page with fairly decent streaming support. So for now I’m a Yawcam user again.  It’s been running for several weeks now without a hitch.  I’ll continue to keep an eye on it and report back here if I have any problems with the software.

New Data Upload Method

I’ve been using FTP to transfer my weather data, statistics, and history to the cloppermillweather Web site.  This has worked fairly well, but I didn’t want to be dependent on my ISP providing FTP.  I knew I could create a PHP Web page to accept data.  I thought I might be able to use wget to do an HTTP POST with the data files I wanted to upload.  So I did some researching and discovered that curl was fully capable of doing what I wanted.  So I created a simple PHP page to accept a file and modified my FTP scripts on my weather server to use curl instead of FTP.  Everything is working great with a little less overhead than my prior FTP scripts.  I also added some simple validity checks on the upload page to make sure the data is good.  Not all that secure (at least not yet), but the concept works just fine.  Here’s a generic curl comand to post a file:

curl -F”operation=upload” -F”file=@fileToUpload” http://URLToReceiveHTTPPost

One thing to watch out for is if the destination site sends back an HTTP 301 redirect.  Curl has an option to follow the redirect and do a subsequent POST.  Most browsers will follow the redirect but refuse to do the HTTP POST after the redirect.  This is actually a security feature, so make sure you know where you are being redirected to before POSTing anything you want to keep private.  Weather data probably isn’t something where privacy is a concern, but the warning applies if you are using this technique for other types of data.