Raspberry Pi and Davis up and Running Again

It took me a few days, but I finally relocated my Raspberry Pi and set up the revised weather station software.  Everything appears to be working just fine.  I plan to just let things run as they are for now and see what develops.  Hopefully the weather station is all set for the winter.  I also need to post the updated software to my Web site in case somebody plans to use it.  I’ll revise this post once it has been uploaded.

Hardware Failure

Well, I sort of predicted there would be some kind of hardware failure in my future.  The predicted failure finally occurred.  After some diagnosis it looked like my 1-wire hub failed.  Also, my barometer became intermittent, occasionally returning a reading of zero.  Some research proved hat the hub was no longer available, and support for 1-wire in general wasn’t very good anymore.  Hobby Boards, which was the source of most of my hardware was even preparing to go out of business.  It was becoming clear to me that 1-wire was probably not the best choice to purchase as new hardware.

After some investigating, I decided that my best choice was a Davis weather station, either the Vantage Pro or Vantage Vue.  They both are solar powered and wireless, which was a big step up from my previous weather station.  The Pro would be my preference, but cost drove me to the Vue.  The two features I wish the Vue had are a fan aspirated solar shield and the ability to separate the anemometer from the rest of the station for better siting.  But when I looked at the cost I realized that my current station has all the sensors mounted together, even though I could separate them.  And I could figure out a way to add a fan if solar heating became an issue.  So I placed my order for a Vantage Vue and the Weatherlink hardware/software.

Once the new hardware arrived, I set it up indoors for testing.  After I verified that it was working I mounted the hardware sensors outside and made sure the console was receiving a good signal.  I had no issues and everything worked perfectly.  Then I installed the Weatherlink module and connected it to my computer in order to test that interface and modify my weather software to work with the Davis hardware.  After a bit of experimentation and reading the Davis documentation I got the software talking to the Weatherlink module.

At the moment the revised software is running in my development environment, which is a Windows 10 desktop computer.  I plan to migrate everything back to the Raspberry Pi in the next few days.  Since the Pi isn’t running my weather station software at the moment I took this opportunity to use apt-get to upgrade any software running on the Pi.  It hadn’t been upgraded, powered down, or rebooted in well over a year, so it was long overdue.  I plan to relocate the Pi to another room in my house, so it will take me some time to make room for everything.  I hope to have everything running back on the Pi within the next few days.

User Agent

I noticed yesterday that the weather forecast on my Web site was almost 24 hours out of date.  I looked through my log files and saw that I was getting a 403 error which means access denied or a permissions problem.  When I browsed to the same pages using a browser I was able to access them just fine.  I was also able to use wget to retrieve the pages. I decided it might be a problem where PHP was not sending a user agent header and the NWS Web server was looking for one.  I added a user agent header to all of my pages that retrieve weather data from the NWS and everything is back up and running again.

I’ve also noticed, but nor done anything about two of my other pages,  The RSS feeds I was using for local alerts and traffic information no longer operate as the feeds have been discontinued.  I put a message on the pages and will need to search for a new source of traffic and alert information.

Other than that the station has been running on auto-pilot for quite a few months now.,  I’m probably overdue for some type of hardware failure but I’m keeping my fingers crossed.

Solar Heating

In my last post I mentioned that I might be getting some solar heating.  I had checked my fan to make sure it was turning and found that it was stuck.  A small push got it turning again.  However my QC checks still showed signs of solar heating during the daytime.  Then I started seeing elevated temperature readings at night as well.  I suspected the fan bearing had seized causing the fan to get hot and be the source of the erroneous temperature readings.  So I made some time to go outside, take the solar shield apart and see what was going on.

The first thing I did was listen for the fan running.  No fan was heard.  Then I put my hand up inside the solar shield to check the fan wiring to see if it had come loose.  My hand brushed up against something that I didn’t recognize.  I bent down and looked up inside the solar shield to find a small wasp next.  Luckily it appeared it be deserted, otherwise I would most likely have been stung.  I knocked down the nest and started taking the solar shield apart.  A few wasps flew out and i jumped back.  When the coast was clear I carefully started opening up the solar shield and peeked inside.  I saw a larger wasp nest and several wasps flew out toward my face.  I backed up as fast as I could move in preparation to run away.  I ended up falling over backwards and slightly injuring myself.  Luckily the wasps let me escape without stinging me.  I got a can of bug spray put of the garage and that was the end of the wasp nest.

I was now able to complete my dis assembly and confirm that the fan had failed.  I ordered a new fan, which took about a week to receive.  Then I removed the old fan and wired in the replacement.  The old fan was pretty hot when I pulled it out, so I feel pretty confident that ti was the source of my heating.  The station appears to be working so I plan to let it run for a day or two and keep an eye on my QC checks.  I also have a spare fan (I bought two) so I can make repairs faster when I have this problem again.

 

Smooth Sailing

Just wanted to post an update after my fixes from last month.  The QC page from CWOP shows all green thumbs up for my weather station quality checking now.  That means that the comparison to whichever neighboring stations it uses shows that my station is at least in the ballpark.  That’s a whole lot better than it was before I installed the new humidity sensor.  The wind speed statistics also look good since I made the code change.  I need to go back and clean up an erroneous reading that was recorded prior to making the code change.  I’m not sure if it was caused by the clock bug or not, but it is definitely an error.  I also need to take a look at the fan I’m using to make sure it is still working.  The temperature readings look like I could be getting some unwanted solar heating.  So some maintenance on the solar radiation shield could be called for.  I’ll check it out and post back here if anything needed to be corrected.

Humidity Fix

My weather data has been flagged as having an issue on the quality checking from CWOP.  Specifically, the humidity readings have been too high compared to neighboring stations.  I had purchased a replacement humidity sensor chip several months ago, but never had the opportunity to install it.  Yesterday that opportunity presented itself.  I shut down the station, took it apart enough to get to the old sensor and brought it inside to the workbench.  It wasn’t very difficult to unsolder the old sensor and solder the new one in its’ place.  Then I reassembled the station and started it back up.  The humidity readings appear to be more reasonable.  I’ll have to wait a few days and see how the quality checking goes.

While i was at it, I modified the code to use System.nanoTime() rather than System.currentTimeMillis() for calculating the wind speed.  The reason for the change wasn’t specifically to gain additional precision (although that is an OK bonus). My concern is that it is possible for the time to be corrected (such as via NTP or by the admin) in the midst of doing the timing for the wind speed.  This could cause an incorrect value to be calculated, and has led to readings of wind speeds in excess of 100 mph in some other weather software packages.  Mine is most likely susceptible to this bug as well.  While this is a rare occurrence and the probability of it happening is slim, the Java recommended method for doing timing is using nanoTime(), which is based on a relative time and is only useful for comparing to other nanoTime() values.  Again, I’ll follow my data for the next few days and see  how it looks.

 

Cutting The Cord

This past weekend I noticed that my weather Web site was no longer updating with the most recent weather data from my weather station.I check the error logs on the Raspberry Pi running the weather software and saw numerous errors indicating communication problems with the weather station.  Upon closer inspection I discovered that when my son mowed the lawn he somehow managed to sever the cable between the weather server and the weather station hardware.

I promptly shut down the weather station software.  I had to let the station stay offline until I had an opportunity to purchase a new 25 foot long Cat 5 network cable.  In the mean time I made some minor coding modifications to the JavaScript code to better display when there were data errors.  The PHP code displayed the errors how I wanted, but the JavaScript code showed zeroes instead of an error indicator.

Today I had some time to stop by the computer store and purchase a new cable for $14.00.  I removed the severed cable and installed the new one.  Once I restarted the weather software everything came back up fine.  So the weather station is open for business once again.

I’ve been toying with the idea of making the weather station wireless by locating a Raspberry Pi outside with the weather station and use a wireless network to connect to the station.  Right now I have the Pi running from a USB hard drive which I would not want to locate outdoors.  The main reason for the USB hard drive is so that I don’t wear out the SD card from too many writes.  I could split up the software so that the outdoor station reads the hardware and makes the readings available over the network.  This would eliminate many of the writes to the SD card.  An indoor computer could retrieve the raw data and then process it, store it, and update any remote weather services.

The change wouldn’t be difficult to make, and is kind of interesting.  There are a lot of wireless networks in my neighborhood and lots of interference.  Wireless networking isn’t always as reliable in my house as I would like, so I’ve resorted to wired wherever I can.  I may choose to make this change some time, but for now I’m good with the wired connection.

Raspberry Pi Weather Station

It took me long enough, but I finally moved my weather station software to the Raspberry Pi.  I am happy to say that it is running smoothly, my weather database is updating, all the services I share my weather data with are receiving data, and my Web site is up to date.  I set the software up to run as a service so that I can use the typical Linux tools for managing, starting, and stopping the software.  This is no different than what I was doing on the prior server.  I installed my logrotate scripts and verified that they run by hand, but haven’t conformed that the logs are rotating automatically yet.  The log files need to grow a little more before I will see them being rotated.

I also updated my logretriever software to accept several new command line parameters.  Originally I had hard coded the server hostname, port, and the file name to store the retrieved data in.  Now they can all be specified on the command line.  I also added a usage/help display when the parameters are specified incorrectly.  The downside is that I used the commons cli jar file to do the heavy lifting for this functionality.  That means a new dependency to run the software.

I also made a few script updates.  I had to change the scripts that run logretriever to use the new command line parameters.  While I was in there I also modified the script that updates the weather data to use a loop rather than repeating the same commands multiple times.  Not sure why I never made that change.  It makes for less code to maintain, and also makes it easier to fine tune the script as far as the number of times to run and the delay between each run.

It’s only been one day so far, but everything is working correctly.  The updated software has been uploaded to the Web site.  I will continue to monitor everything closely for the next week or so to make sure there are no issues.

Get Some Raspberry Pi

My scripts have been running pretty well these last few weeks. I added some error checking in case something does go wrong. But overall they have been humming along nicely. Time to move on to the next project, which is transitioning my station computer over to one of my Raspberry Pi microcomputers. In case some of you aren’t familiar with the Raspberry Pi, it is a small, single board microcomputer that was originally designed as an educational device. While it serves that purpose well, its’ low price allowed it to be used by hobbyists for many other purposes. For more information on the Raspberry Pi you can browse to raspberrypi.org to get more information on the board.

The Raspberry Pi is similar in many ways to the GuruPlug computer I had used previously as my weather station computer.  That computer, as well as the Pi uses an SD card as its’ primary storage device.   So the operating system and any files it needs are all located on a storage device rather than a traditional hard drive.  While the access speeds are fairly fast compared to a hard drive, the SD card has limited storage and a finite number of write cycles.  This isn’t a big problem for many uses, but my weather station software writes to the file system fairly often.  And since I keep a database with my weather data the limited storage could eventually become a problem.

I had a spare external hard drive with a USB interface I could use to add storage to the Pi.  Utilizing this drive would permit the weather station software to write to a hard drive and avoid wearing out the SD card so quickly.  It also had plenty of storage if I wanted a database to store the weather data.  Sounded like a win/win type of situation.  Then I thought about taking it one step further and installing the operating system on the hard drive and having the boot loader mount the drive and load the operating system from there.  I did some quick Web searching and came up with the solution.

The page I found is The Raspberry Pi Hobbyist: Running From an External Hard Drive.  This page had detailed instructions on how to set everything up.  If you aren’t familiar with Linux the instructions will look like some arcane version of gibberish.  But to somebody familiar with installing hard drives on Linux/Unix it will make sense.  You can look at that page for details, but here are the high level steps.

  • Attach the USB drive and make sure it is recognized
  • Partition the drive as desired, making sure to at least create a partition for the OS and a swap partition.  I created an additional data partition.
  • Format the partitions as needed
  • Set up the swap partition on the hard drive
  • Copy the OS partition from the SD card to the hard drive partition.  This uses a block by block copy from the SD card to the hard drive, creating a mirror image on the hard drive.
  • Check the hard drive OS partition for errors.
  • Resize the newly copied OS data to use the entire hard drive partition which is most likely larger than the SD card partition.
  • Modify the boot file on the SD card to boot from the hard drive rather than the SD card
  • Change fstab to mount the new partitions
  • Prevent the system from using the swap partition on the SD card

I made all of these changes to my system and then rebooted.  It still requires the SD card to initiate the boot process, but then mounts up the hard drive and boots from there.  It booted up just fine and when I checked the root partition was on the USB hard drive.  So that step worked just fine.  I also set up Postgresql and configured it so that databases are created on the data partition I created on the USB hard drive.  Then I installed Java and the rxtx library so that my weather application can run.  I don’t really have a whole lot of work left to be able to swap computers and use the Pi for my weather station.  But I have several other unrelated activities that must take precedence over the weather station.  So it may be a little while before I switch over.  I’ll post an update when I make the switch.  Then I need to fix the humidity sensor by replacing the sensor chip once again.  I always seem to be able to find ways to stay busy.

Web Cam Madness Scripts

I have decided to post my Powershell scripts that extract and upload the images, and build and upload the time lapse video.  These were some of my very first Powershell scripts, so there will probably be a lot of room for improvement.  But here they are.

The first script retrieves the current image from an HTTP server and save it in a folder named with the current date. The images are given sequence numbers.

########################################################
# Create a sequence of images retrieved from a Web server
#
# Author: Seth Cohen
# Date: 02/17/2014
#
# Input: An http server to retrieve the images.
#
# Output: a folder named YYYY-MM-DD containing a set of jpeg
#       images.  The images are named nnnnn-image.jpg where
#       nnnnn is a sequential number.
#       If the folder doesn't exist, it will be created.
#       When the day changes, a new folder will be created
#               if need be.
#
#       Note: This script must be called each time a new image
#       is to be created.  It can be run from the Windows task
#       scheduler periodically.  I currently run it daily
#       repeated every minute.
#
########################################################

# set destination folder the where images will be stored
#       Use the full path to reference it
$folderPath = "f:\Work"

# set the source URL to retrieve
$url = "http://192.168.123.150:8888/out.jpg"

# create folder name to store individual images, based on the date
$folderName = get-date -format yyyy-MM-dd

# get full path to folder using the folder path specified at the top
$fullFolderPath = join-path -path $folderPath -childpath $folderName

# create folder if it doesn't already exists
if (!(Test-Path -path $fullFolderPath))
{
New-Item $fullFolderPath -type directory
}

# create file name using a sequence number.  The sequence number is
#   calculated by counting the number of jpg files in the destination folder
#   and adding one to it.
$count = (get-childitem $fullFolderPath -filter *-image.jpg).count + 1
$fileName = "{0:D5}" -f $count
# $fileName = get-date -format HHmmss
$fileName = "$fileName-image.jpg"

# Build the file name to grab
$sourceFile = join-path -path $fullFolderPath -childpath "out.jpg"

# retrieve the image from web server using HTTP
invoke-webrequest $url -outfile $sourceFile

# move and rename file
$destPath = join-path -path $fullFolderPath -childPath $fileName
move-item $sourceFile $destPath

The next script creates the time lapse video and uploads the file to an FTP site

########################################################
# Create a time lapse movie from a set of images using ffmpeg
#
# Author: Seth Cohen
# Date: 02/17/2014
#
# Input: a folder named YYYY-MM-DD containing a set of jpeg
#       images.  The images are named nnnnn-image.jpg where
#       nnnnn is a sequential number.
#
# Output: a mpeg movie with a file name of YYYY-MM-DD.mpg
#
########################################################
# set full path to the executable to run (ffmpeg)
$executable = "path_to_ffmpeg.exe_goes_here"

# set destination path where images will be stored
$folderPath = "f:\Work"

# build yesterday's folder name
$yesterday = [DateTime]::Now.AddDays(-1)
$foldername = $yesterday.toString("yyyy-MM-dd")

# get full path to folder
$fullFolderPath = join-path -path $folderPath -childpath $folderName

# make sure yesterday's folder exists
if (Test-Path -path $fullFolderPath)
{
        # we have a folder to work with, so run ffmpeg
        $commandLineArgs = "-i $fullFolderPath\%05d-image.jpg $folderPath\$folderName.mpg"
        invoke-expression "$executable $commandLineArgs"

        # after the movie is created we can perform any clean up deemed necessary
        #   like deleting the source directory.  Do this by hand for now or use
        #   another clean up script to prune old folders

        # copy the video to the  weather web site via ftp
        $server = "your_ftp_site_goes_here"
        $user = "your_username_goes_here"
        $password = "your_password_goes_here"

        echo "open $server
                user $user $password
                binary
                lcd $folderPath
                cd public_html/images
                put $folderName.mpg time-lapse.mpg
                dir time-lapse.mpg
                quit" | ftp -n -v
}

Finally, here is the script that grabs the image from the Webcam and uploads it to Weather Underground via FTP

#################################################
# Powershell script to retrieve the current webcam image
# from Yawcam and upload it to Weather Undeground
# The image is at http://localhost:8888/out.jpg
#
# Author: Seth Cohen
# Date: 02/16/2014
#
#################################################

#Import-Module PSFTP

# first retrieve the image from yawcam
invoke-webrequest http://192.168.123.150:8888/out.jpg -outfile c:\temp\wxwebcam.jpg

# if it was successful then ftp it to the remote site
$server = "webcam.wunderground.com"
$file = "wxwebcam.jpg"
$user = "wxunderground_user_name_goes_here"
$password = "wxunderground_password_goes_here"

echo "open $server
user $user $password
binary
lcd c:\temp
put $file image.jpg
dir
quit" | ftp -n -v