It seems like it’s an actually an Excel problem. The data in the CSV file exported from the HoboWare software are stored in one format – 07/02/12 06:00:00 PM – but the data in Excel are sometimes translated into Excel serial dates (which is what we want) and sometimes remain in this Hobo format.

There may be a quicker way of handling this foible, but (as ever) my first thought was to throw some VBA at it. Here’s the code for a UDF which will knock that data into shape.

Option Explicit

Function HOBO_TO_SERIAL(HoboTime As String) As Date

    ' Fix the edge case for values at midnight where no time part is brought in
    If Not (InStr(HoboTime, ":") > 0) Then
        HoboTime = HoboTime & " 00:00:00"
    End If
    
    ' Set up to produce the date value
    Dim dateVal As Date
    Dim dateArr As Variant
    dateArr = Split(Split(HoboTime)(0), "/")
    
    ' Set up to produce the time value
    Dim timeVal As Date
    Dim timeArr As Variant
    timeArr = Split(Split(HoboTime)(1), ":")
    
    ' Correct time hour values from 12 to 00 to make the maths work later
    If timeArr(0) = 12 Then timeArr(0) = "00"
    
    ' Parse the input into a date value part and a time value part
    If InStr(HoboTime, "AM") Then
        ' Put the date into the format "dd/mm/yyyy". This means adding "20" to the start of the year
        dateVal = DateValue(dateArr(0) & "/" & dateArr(1) & "/" & ("20" & dateArr(2)))
        timeVal = TimeValue(timeArr(0) & ":" & timeArr(1) & ":" & timeArr(2))
    ElseIf InStr(HoboTime, "PM") Then
        ' Put the date into the format "dd/mm/yyyy". This means adding "20" to the start of the year
        dateVal = DateValue(dateArr(0) & "/" & dateArr(1) & "/" & ("20" & dateArr(2)))
        ' Correct the time from 12 hr to 24 hour clock
        timeArr(0) = (CInt(timeArr(0)) + 12) Mod 24
        timeVal = TimeValue(timeArr(0) & ":" & timeArr(1) & ":" & timeArr(2))
    Else
        If Application.International(xlDateOrder) = 1 Then
            ' Switches the day and month for UK-style date users
            dateVal = DateValue(Format(HoboTime, "yyyy-dd-mm"))
        Else
            dateVal = DateValue(Format(HoboTime, "yyyy-mm-dd"))
        End If
        timeVal = TimeValue(Format(HoboTime, "hh:mm:ss"))
    End If
    
    ' Finally, add the two values together to get a complete Excel serial time
    HOBO_TO_SERIAL = dateVal + timeVal
End Function

As ever, hope this helps someone else out.

Wind rose charts

Do you have a home weather station and you’d like to interpret the outputs? Are you writing a renewable energy report and want to add a graphic to represent the wind resource? A structural report and you’d like to visualise the wind loading?
Then you might just need a wind rose.

Wind speeds chart

And while you’re about it, why not add in a chart of wind speeds. This is sometimes called a Weibull chart because wind speeds tend to follow a Weibull distribution.

Knowing where the wind comes from and how fast it is can tell you a lot about whether the site is suitable for a wind turbine. The faster the average wind speed the better return you’ll generate. A common rule of thumb is that if the average wind speed is less than 5 meters per second then you won’t get much benefit from a wind turbine.

This post will take you through creating charts like these.

1: Getting your wind data

Obviously the most important part of creating your wind speed graphic is the data. How you get hold of this will vary depending on what you’re using it for. These are a few typical places you might find wind speed data to use:

Weather stations

All over the world weather stations are set up recording every passing gust of wind, floating cloud, drop of rain and ray of sunshine. You can download data from thousands of Weather Underground stations around the world using our Weather Access web form.

Energy modelling weather files

You can often extract the data you need from the weather files used by building simulation engineers. These aren’t as well distributed across the world as weather stations, but they do represent a typical year rather than just a year chosen random so are likely to give a more accurate idea of wind conditions than a weather station. You can find your nearest weather files using the Weather File Finder tool from this site.

Your own weather station

If you have your own personal weather station or anemometer then you will probably have a ready source of data. If not, you can get hold of one, through one of the links at Weather Underground, or through Better Generation who produce the Power Predictor which is specifically designed to assess renewable energy potential.

2: Download and open the wind chart creator

This is an easy step. The download link is here. Just save the file to your computer, unzip the zip file and open it up in Excel.

3: Import the wind data

This step depends on the source of your data.
If it’s from our Weather Access page then you will probably have downloaded it in two six-month chunks. There’s a button on the front page of the wind charts maker to import them for you.

If it’s from an energy simulation weather file then the process may be a bit more involved. We’ll work towards making a simple import button like we have for Weather Access/Weather Underground data but for now you’ll probably have to wrangle the data yourself in Excel and paste it into the Windspeed m/sec column.

If it’s from your own weather station then the simplest route might be to join the Weather Underground network and import it from there, otherwise again it’s a case of formatting the data yourself in Excel and pasting it into the Wind Charts tool.

4: Check your charts

Your charts should now be ready in the Charts tab of the workbook. Just copy and paste them into your document and away you go!

This is a brand new tool so please do give us your feedback, either by email or in the comments here.

For lots more information on wind energy, wind turbines, and calculating the return you can expect in your location, Small Scale Wind Power Generation: A Practical Guide is available now!

There’s a load of other weird and wonderful things knocking around on the site. Have a look in the Wet Bread section. I particularly like the look of Pong in Excel!

If there’s one single site on the internet that I couldn’t live without (perhaps with the exception of Google), it would be StackOverflow from StackExchange. It’s an absolute life-saver for programmers, and as a fairly recent convert to programming, I struggle to think what it must have been like in the days before it existed. It’s on a par with how people ever found information before Google.

The amount it increases a lone coder’s productivity is just staggering.

So when I see that there is a suggestion to create a StackExchange site for building energy performance simulation I can barely control my excitement. A place where researchers can share best practice, iron out little niggles in their models, show prospective employers that they know their stuff [1], and all the other good things that StackExchange brings to the coding community.

Apologies for the evangelising but this is a really great opportunity to make a difference to researchers’ and engineers’ day-to-day lives. Please do head on over and get involved. You can help move the fledgling site out of “Area 51″ and on to the next stage of development before it can become a fully-developed StackExchange site.

Thanks to Clayton Miller at OpenRevit for the heads-up.

[1] Via the reputation system. Here’s my profile on StackOverflow. I look forward to adding building energy performance simulation to this badge.

If you want to know how to design your building you to need to know what the local weather is like. If you want to understand how your building is performing you need to know what weather it was coping with. If you want to install renewable energy – solar panels, wind turbines – you need to know what kind of weather conditions you’ll be able to harness. Weather Underground is a great source of free weather data, aggregating a multitude of independent weather stations all around the world.

I first needed to pull some data from Weather Underground after I found out that the weather station on a university campus down in Cornwall where I’m monitoring a building had been damaged during the construction of another building on the campus. And then I found out the same thing had happened to another building I’m monitoring down there. The problem was I needed months and months of data for two sites, but Weather Underground only seemed to serve up one day at a time. And so was born wunderscraper.py.

You can do some pretty cool things with weather data. This one plots temperature against heating energy use.

And this one plots the wind conditions for a location. Very useful if you’re considering installing a wind turbine. If you or someone you know are interested, Small-Scale Wind Power: A Practical Guide would make a great present – and it’s on offer this Christmas!

After that as a bit of a challenge to improve my non-VBA coding, I thought it might be a nice idea to try and put together a proper web-based API to simplify things a bit [1]. If it gets too much traffic I’ll have to take it down, hobble it, or password protect it – so be gentle!. I don’t want to take real business away from the Wunderground people, but I don’t think there’ll be enough traffic to this site to really worry them.

So, how does it work?

It’s really a very simple API. There are three URL parameters, stationID, startdate and enddate. Basically you build your URL as follows:

http://oco-carbon.com/weather/download.php?stationID=ICORNWAL14&startdate=2012-10-09&enddate=2012-10-10

Give it a try.

That downloads you a CSV file containing all the data between the two dates specified. For this example the file will be called ICORNWAL14_2012-10-09_2012-10-10.

Since my site is currently hosted in the United States (if you’re looking for hosting, check out my affiliate link [2]), the resulting file comes back with temperatures in Farenheit, rainfall in inches, wind speeds in miles per hour, etc. by default. I’ll add a parameter to specify whether you want imperial/US or metric data in the future but this will do for now. For metric data just tick the checkbox option.

As well as the API, I’ve also put together a web form that you can use to do the downloading if you’re not minded to use the API directly. It’s as easy as putting in the weather station ID which you can find from here, and the start and end date of the period you want.

Enjoy!

[1] I know, I know. They have a proper API. But in the words of a blog post I just read, “I don’t need no stinking API”, and I enjoyed the learning process.

[2] Disclosure: Apparently iPage will pay me for referrals, but I’d recommend them anyway. My partner’s site, Another Voice Translations is also hosted through iPage on my recommendation and that was before I knew about the affiliate programme. They made making the transition from a WordPress blog to a self-hosted site really very simple, and adding the scripting that powers this API was also pretty easy. This site is independently owned and the opinions expressed here are my own.

The key difference is that you have to supply a time parameter as well as the start and end location. This can either be a departure or an arrival time.

The format required for the time parameter is seconds since midnight on 1/1/1970 (the start of the UNIX epoch), while Excel times are in days since midnight on 1/1/1900. This difference meant it seemed worth writing a function to do the conversions.

Function ExcelSerialToUnixEpoch(ExcelSerialTime As Date)
' Converts a date in Excel's date format to seconds since midnight 1/1/1970
Dim secondsPerDay As Double
Dim unixEpoch As Date
Dim excelEpoch As Date
Dim daysDifference As Long
    unixEpoch = #1/1/1970#
    excelEpoch = #1/1/1900#
    secondsPerDay = 86400
    daysDifference = DateDiff("d", excelEpoch, unixEpoch)
    ExcelSerialToUnixEpoch = (ExcelSerialTime - daysDifference) * secondsPerDay
End Function

Function UnixEpochToExcelSerial(UnixEpochTime As Double)
' Converts seconds since midnight 1/1/1970 to a date in Excel's date format
Dim secondsPerDay As Double
Dim unixEpoch As Date
Dim excelEpoch As Date
Dim daysDifference As Long
    unixEpoch = #1/1/1970#
    excelEpoch = #1/1/1900#
    secondsPerDay = 86400
    daysDifference = DateDiff("d", excelEpoch, unixEpoch)
    UnixEpochToExcelSerial = (UnixEpochTime / secondsPerDay) + daysDifference
End Function

So far so good. So what next?

A day of coding an interface for Google’s experimental Time Zone API, that’s what. This is because I can’t assume that everyone’s in the UK – Max is interested in public transport in Singapore for example.

The results are the following functions for your spreadsheeting delight. They’re also fine for VBA for Excel.

=LOCALTIME(Location As String)
Returns the current local time for a given location, including daylight saving

=DST_OFFSET(Location As String)
Returns any daylight saving time offset to the time at a given given location

=UTC_OFFSET(Location As String)
Returns the time difference between UCT and the time at a given given location

=TIMEZONE_ID(Location As String)
Returns the ID of the timezone for a given location

=TIMEZONE(Location As String)
Returns the name of the timezone for a given location

These functions all ought to be pretty stable. They don’t cache results, obviously – since they’re to do with current time. But they do use recursion to keep extending a wait between repeated calls if you have reached Google’s soft limits for the API. These soft limits are where Google throttles the service if you appear to be accessing it too often in a short period of time. The module also includes a fallback to spot if you seem to have hit the hard limit for the day of 25,000 calls. In that case it will return the error message OVER_HARD_QUERY_LIMIT. If that happens then you need to go away for a while and try again the next day (or change your IP address if you’re feeling particularly sneaky).

You can download a workbook here with all of these functions.

This was all something of a distraction from the main purpose of figuring out how to access travel times for public transport. That’s the next task on the horizon in the ever-expanding set of Google API interfaces for Excel that I’m building up here. It’s actually pretty simple, it just needs tidying up so if that’s what you’re waiting for, keep checking back here over the next week or so.

I happened to be debugging a recent tool (the EPW weather file finder) and thought that it might be an idea to upload the finished version including features like results caching, error handling, URL encoding, and avoidance of localisation problems from the use of commas as decimal separators.

As before it returns latitude and longitude, but it now has three additional options for the return values and one to force the function to requery Google, for example if you think the value it has returned is incorrect.

=G_LATLNG("Trafalgar Square") returns 51.5073346,-0.1276831
=G_LATLNG("Trafalgar Square",1) also returns 51.5073346,-0.1276831
=G_LATLNG("Trafalgar Square",2) returns 51.5073346
=G_LATLNG("Trafalgar Square",3) returns -0.1276831
=G_LATLNG("Trafalgar Square",4) returns Trafalgar Square, City of Westminster, London, UK

To force the function to requery you need to set the Requery option to True like so: =G_LATLNG("Trafalgar Square",,True) (note the two commas – you can also still use the number codes as the second parameter to return latitude, longitude or address).

There’s quite a lot of code in this function, but it’s well commented so should be reasonably easy to follow.

Function G_LATLNG( _
    InputLocation As Variant, _
    Optional n As Long = 1, _
    Optional Requery As Boolean = False _
    ) As Variant
' Requires a reference to Microsoft XML, v6.0
' The parameter 'n' refers to the type of reponse
' n = 1: Returns latitude, longitude as string
' n = 2: Returns latitude as double
' n = 3: Returns longitude as double
' n = 4: Returns address as string

' Updated 30/10/2012 to
'   - return an #N/A error if an error occurs
'   - cache only if necessary
'   - check for and attempt to correct cached errors
'   - work on systems with comma as decimal separator

Dim myRequest As XMLHTTP60
Dim myDomDoc As DOMDocument60
Dim addressNode As IXMLDOMNode
Dim latNode As IXMLDOMNode
Dim lngNode As IXMLDOMNode
Dim statusNode As IXMLDOMNode
Dim CachedFile As String
Dim NoCache As Boolean
Dim V() As Variant
    On Error GoTo exitRoute
    G_LATLNG = CVErr(xlErrNA) ' Return an #N/A error in the case of any errors
    ReDim V(1 To 4)
    
    ' Check and clean inputs
    If WorksheetFunction.IsNumber(InputLocation) _
        Or IsEmpty(InputLocation) _
        Or InputLocation = "" Then GoTo exitRoute
    InputLocation = URLEncode(CStr(InputLocation), True)
    
    ' Check for existence of cached file
    CachedFile = Environ("temp") & "\" & InputLocation & "_LatLng.xml"
    NoCache = (Len(Dir(CachedFile)) = 0)
    
    Set myRequest = New XMLHTTP60
    
    If NoCache Or Requery Then ' if no cached file exists or if asked to requery then query Google
        ' Read the XML data from the Google Maps API
        myRequest.Open "GET", "http://maps.googleapis.com/maps/api/geocode/xml?address=" _
            & InputLocation & "&sensor=false", False
        myRequest.Send
    Else ' otherwise query the cached file
        myRequest.Open "GET", CachedFile
        myRequest.Send
        Set myDomDoc = New DOMDocument60
        myDomDoc.LoadXML myRequest.responseText
        ' Get the status code of the cached XML file in case of previously cached errors
        Set statusNode = myDomDoc.SelectSingleNode("//status")
        If statusNode.Text <> "OK" Then
            Call G_LATLNG(InputLocation, n, True) ' Recursive way to try to remove cached errors
        End If
    End If
    
    ' Make the XML readable using XPath
    Set myDomDoc = New DOMDocument60
    myDomDoc.LoadXML myRequest.responseText
    
    ' If statusNode is "OK" then get the values to return
    Set statusNode = myDomDoc.SelectSingleNode("//status")
    If statusNode.Text = "OK" Then
        ' Get the location as returned by Google
        Set addressNode = myDomDoc.SelectSingleNode("//result/formatted_address")
        ' Get the latitude and longitude node values
        Set latNode = myDomDoc.SelectSingleNode("//result/geometry/location/lat")
        Set lngNode = myDomDoc.SelectSingleNode("//result/geometry/location/lng")
        V(1) = latNode.Text & "," & lngNode.Text
        V(2) = Val(latNode.Text) ' Fixed for systems with comma as decimal separator
        V(3) = Val(lngNode.Text) ' Fixed for systems with comma as decimal separator
        V(4) = addressNode.Text
        G_LATLNG = V(n)
        If NoCache Then: Call CreateFile(CachedFile, myRequest.responseText) ' Cache API response if required
    End If

exitRoute:
    ' Tidy up
    Set latNode = Nothing
    Set lngNode = Nothing
    Set myDomDoc = Nothing
    Set myRequest = Nothing
End Function

This depends on the following functions:

Public Function URLEncode( _
   StringVal As String, _
   Optional SpaceAsPlus As Boolean = False _
) As String
' Function from http://www.tinyguru.com/error/qid218181.html
Dim StringLen As Long: StringLen = Len(StringVal)
Dim i As Long, CharCode As Integer
Dim Char As String, Space As String

    If StringLen > 0 Then
        ReDim result(StringLen) As String

    If SpaceAsPlus Then Space = "+" Else Space = "%20"

        For i = 1 To StringLen
            Char = Mid$(StringVal, i, 1)
            CharCode = Asc(Char)
            Select Case CharCode
                Case 97 To 122, 65 To 90, 48 To 57, _
                    45, 46, 61, 95, 123, 125, 126
                    result(i) = Char
                Case 32
                    result(i) = Space
                Case 0 To 15
                    result(i) = "%0" & Hex(CharCode)
                Case Else
                    result(i) = "%" & Hex(CharCode)
            End Select
        Next i
        URLEncode = Join(result, "")
    End If
End Function

Function CreateFile(fileName As String, Contents As String) As String
' Function from http://www.jpsoftwaretech.com/vba/create-new-text-documents-using-vba/
' creates file from string contents
Dim tempFile As String
Dim nextFileNum As Long
  nextFileNum = FreeFile
  tempFile = fileName
  Open tempFile For Output As #nextFileNum
  Print #nextFileNum, Contents
  Close #nextFileNum
  CreateFile = tempFile
End Function

This isn’t what I actually need the equation for, just a test case for making sure the equation works. If you want to try this out yourself, the data is the abalone data set from UCI. It’s a fairly simple data set which collates a whole load of data about abalone, along with the number of rings they have in their shells – an indicator of age, like tree rings.

The regression model I built is a simple one, with all the features as predictive variables and the ring count as the independent variable.

Here’s the code:

Function NormalEquation(X As Variant, y As Variant, Transpose As Boolean)
' theta = (X’ * X)^-1 * (X’ * y)
' Where n is the number of cases you have and k is the number of features:
'   X is an n rows x k columns array
'   y is an n rows x 1 column array
Dim XTy As Variant
Dim XTX As Variant
Dim theta As Variant
    XTX = Application.MMult(Application.Transpose(X), X)
    XTy = Application.MMult(Application.Transpose(X), y)
    NormalEquation = Application.MMult(Application.MInverse(XTX), XTy)
    If Transpose Then NormalEquation = Application.Transpose(NormalEquation)
End Function

To use it, set up the data, set up named ranges for X, y and theta (theta should be a 1D array of length k). In the picture below, X is red, y is orange and theta is blue.

Use this code to call the function:

Sub TestNormalEquation()
Dim X As Variant
Dim y As Variant
Dim theta As Variant
    X = Range("X").Value2
    y = Range("Y").Value2
    theta = NormalEquation(X, y, True)
    Range("theta") = theta
End Sub

To use the results, the simplest way is to enter =ROUND(SUMPRODUCT(theta,C3:K3),0) in cell M3 and fill down to the bottom of the data.

There you have it, a really powerful linear regression function ready for anything you want to throw at it.

It’s more of an issue in documents than spreadsheets. People seem to hold Excel documents to a lower standard than those in Word, but it still nags at me when I see table headings with these annoying little errors.

So this is something I’ve been meaning to write for literally years, and only just got around to completing. You may have seen my post on subscript and superscript formatting for common things like CO₂, m², m³, H₂O and so on in Word. If not, here it is.

And here’s the same thing for Excel. The macro corrects:

m2 to m²
m3 to m³
H2O, H20 to H₂O
C02, CO2 to CO₂
kwh, KWH, KWh to kWh
KG, Kg to kg

It works on protected worksheets in your workbook, so long as they are not password protected.

You’re welcome!

Sub FixFormatting()
'Exit Sub
' To fix m2, m3, CO2, kWh and kg to their proper states.
Dim c As Range
Dim StartCells As Range
Dim ws As Worksheet
Dim intPlace As Integer
Dim wsStartsProtected As Boolean

    Application.ScreenUpdating = False
    On Error GoTo errorCatch
    Set StartCells = Selection
    For Each ws In Application.Worksheets
        wsStartsProtected = ws.ProtectContents
        With ws
            .Visible = True
            .Activate
            .Unprotect
            .UsedRange.Select
        End With
        For Each c In ws.UsedRange
            With c
                .Replace What:="C02", Replacement:="CO2", LookAt:=xlPart, MatchCase:=False
                .Replace What:="H20", Replacement:="H2O", LookAt:=xlPart, MatchCase:=False
                .Replace What:="kwh", Replacement:="kWh", LookAt:=xlPart, MatchCase:=False
                .Replace What:="mwh", Replacement:="MWh", LookAt:=xlPart, MatchCase:=False
                .Replace What:="gwh", Replacement:="GWh", LookAt:=xlPart, MatchCase:=False
                .Replace What:="kg", Replacement:="kg", LookAt:=xlPart, MatchCase:=False
            End With
            intPlace = InStr(c.Value, "CO2")
            If intPlace > 0 Then
                If ActiveSheet.ProtectContents = True Then ActiveSheet.Unprotect
                c.Characters(intPlace + 2, 1).Font.Subscript = True
            End If
            intPlace = InStr(c.Value, "H2O")
            If intPlace > 0 Then
                If ActiveSheet.ProtectContents = True Then ActiveSheet.Unprotect
                c.Characters(intPlace + 1, 1).Font.Subscript = True
            End If
            intPlace = InStr(c.Value, "m2")
            If intPlace > 0 Then
                If ActiveSheet.ProtectContents = True Then ActiveSheet.Unprotect
                c.Characters(intPlace + 1, 1).Font.Superscript = True
            End If
            intPlace = InStr(c.Value, "m3")
            If intPlace > 0 Then
                If ActiveSheet.ProtectContents = True Then ActiveSheet.Unprotect
                c.Characters(intPlace + 1, 1).Font.Superscript = True
            End If
            If wsStartsProtected Then ws.Protect
        Next
    Next
    StartCells.Parent.Activate
    StartCells.Select
    Application.ScreenUpdating = True
    Exit Sub

errorCatch:
    If wsStartsProtected Then ws.Protect
    StartCells.Parent.Activate
    StartCells.Select
    Application.ScreenUpdating = True
End Sub

Ever had the problem of looking at all the weather files available from the EnergyPlus weather files site and wondering which is closest to the location you’re modelling in? The oCo Carbon Weather file finder will save you the trouble of looking them up, downloading them, opening up the statistics file to check their suitability, and once you’ve found the right site it also has links to download the file or files you need.

Each white dot on the map is the location of a weather file. You can see how hard it can be to pick the nearest one if you don’t happen to know the name of your local weather stations.

For use in building energy simulations .epw files can be read by EnergyPlus, as well as DesignBuilder and other tools which use EnergyPlus as the simulation engine, IES:VE. They can also be converted by TAS using their Weather Utility macro. They can also be used for other purposes such as understanding the influence of local climate on building design using Climate Consultant [Windows] [Mac].

The weather file finder tool looks up the latitude and longitude of an entered location. You can enter anything from a postcode/zipcode to a country name, or even a latitude/longitude pair if you know them. It then calculates the great circle distance to each weather file location (taken from the first line of each weather file on the US Department of Energy website) and creates links to download zip archive containing the .epw weather files for the closest three locations.

The tool also tells you the difference in latitude, longitude and elevation between the two sites which can help you to see whether the two locations are likely to be similar. Locations which have a large difference in elevation or latitude are likely to have quite different weather.

Finally, it lists a number of key statistics about the location including heating and cooling degree days, extreme temperature conditions and climate classification zone.

This tool can save you a lot of time, especially if you’re somewhere with a lot of nearby weather file locations (Poland, I’m looking at you here).

Download oCo Carbon Weather File Finder v 1.02 here.

* With particular thanks to Luke McGuire, Max Tillberg and Patrick Bivona for their useful comments and feedback.