For a long time now I have been advocating that building energy consumption benchmarking is not realistic without a statistically representative database. The fundamental problem with existing databases used for benchmarking (such as DOE's CBECS database) is that building use categorization is much too general and in each building use category there are an insufficient number of buildings to provide statistically meaningful comparisons. Most researchers understand the need for statistically representative data. Unfortunately, the typical response today for relying of any benchmarking database with known shortcomings is "but it's the best we have." I have always believed that it is better not to benchmark than to benchmark against a database that has not been statistically proven to be representative.
Let's look at some of the variables that certainly must be factored into any building energy consumption benchmarking analysis. Of course, the more variables there are, the larger the database that must be built to provide meaningful benchmarking data at a reasonable confidence level, and it starts right with building use. To classify building use into the four major uses, i.e., office, retail, industrial and lodging, would be to grossly oversimplify the picture as there are literally dozens of major building use categories. Each one of these major building use categories can then be further subdivided to reflect buildings with common characteristics. For example, take the office building category which is probably the simplist of building use categories to characterize. Office buildings may be constructed tall or wide, with significant square footage or relatively small square footage. There are different classes (at least three). They may be attached to other buildings or stand-alone. Buildings may have been constructed a hundred years ago or more recently (perhaps under tighter energy efficiency building codes). The buildings may be located in a warm climate or a cold climate or anything in-between. Some office buildings include retail. Some office buildings have high energy consumption facilities such as data centers. There may or may not be standby emergency generators that must be tested periodically. And all this is to say nothing about how the building is operated! For example, what are the building's hours of operation? How does the building handle maintainence and repairs? What amount of space is occupied and what amount is vacant? I can go on and on, and this is just for the "simple" office building category. While lodging has just about as many variables as office buildings, these pale against the many more variables associated with retail and industrial use!
As I said, there literally are dozens of significantly different major building uses. The entire CBECS database (which identifies only 17 different "principal" building uses) includes slightly more than 5,200 buildings to represent the entire country. In view of the wide standard deviation around energy consumption data for each building use, the number of buildings that should be included in the benchmarking database to achieve a 90% or 95% confidence level for each building use category/sub-category is significantly more than currently in the CBECS database.
What should we as an industry do about this problem? I believe that two things are a must. First, every effort must be made to develop a single building energy consumption database that will include all the major building uses (properly categorized with common characteristics) and be statistically representative for benchmarking purposes. This should be a priority of both DOE (with its CBECS database) and EPA (with its Energy Star database). Second, we must refrain from using any benchmarking database, e.g., CBECS, just because "it is the best that we currently have." By using a benchmarking database that has significant deficiencies, the results can (and will) be questioned and may do more harm than good. We as an industry must come together now on this critically important problem and deal with it once and for all.