Accepted for publication at Building Simulation 2009, Glasgow, Scotland

DAYLIGHT FACTOR SIMULATIONS HOW CLOSE DO SIMULATION

BEGINNERS REALLY GET?
Diego I. Ibarra, Christoph F. Reinhart
Graduate School of Design, Harvard University
Cambridge MA, USA
[email protected], [email protected]

ABSTRACT
The daylight factor is usually one of the first daylight
performance measures that simulation newcomers
calculate. Apart from the intrinsic limitations of the
daylight factor as a meaningful daylighting
performance metric, little work has been done in the
past as to how accurate one can actually expect
simulation novices to simulate the daylight factor
compared to an expert modeler. This paper presents
the comparison of daylight factor predictions from a
best practice model of an L-shaped perimeter
classroom to a total of 69 novice/student models. In
all cases the models were prepared in ECOTECT and
simulated in RADIANCE. The paper discusses
common mistakes that simulation beginners make
when carrying out a daylight simulation, how close
their simulation results were compared to the best
practice model, and how software developers and
educators could potentially guide users to avoid
making these mistakes. In addition, a comparison of
simulation results obtained with ECOTECTs buildin split flux method as opposed to RADIANCE is
carried out for the 69 models in order to quantify in
how far using a less reliable simulation engine
compromises the accuracy of a simulation.
Keywords:
Daylight simulation, simulation errors

INTRODUCTION
Introducing effective daylight strategies has become
an essential goal for any sustainable building.
However, since it is difficult to evaluate its quality
and quantity in non-standard spaces through simple
rules of thumb, the use of daylight simulations has
considerably increased as a necessary step to
accurately evaluate daylight in buildings. This has
been a conclusion of two recent surveys: the first
survey focused on simulation experts and their
specific use of daylight simulations (Reinhart and
Fitz 2008), the second survey addressed more
globally how green building design teams are
currently implementing daylighting in their projects.
Both groups reported that they routinely use
simulations especially at the design development
stage (Galasiu and Reinhart 2008). The simulation
experts mostly modeled work plane illuminances and
daylight factor (DF) whereas the green building

community was also concerned with the overall

energy implications of daylighting and solar gains.
In North America, the growing use of and interest in
daylight simulations tools can be attributed to
building standard and green building rating systems
alike. E.g. ASHRAEs Standard 90.1-2007
(ASHRAE 2007), which serves as a building energy
code requirement in several US states, now stipulates
the use of daylight simulation in order to determine
the energy saving potential of photocell controlled
diming systems in a particular space. At the same
time, the US Green Building Councils widely used
LEED Green Building Rating System promotes
daylight simulations as one of the compliance paths
to earn daylighting credit 8.1 (USGBC 2006).
The rising interest in daylight simulations has largely
happened in parallel with a rising interest in building
energy modeling. But, one important difference
between the two types of simulation programs is that
while there already exists an ASHRAE/ANSI
standard that specifies test procedures for evaluating
the technical capabilities [...] of computer programs
that calculate the thermal performance of
buildings(ASHRAE 2007), there is no such
standardized testing procedure in place for
daylighting software. This caveat notwithstanding, a
number of validation studies using measured indoor
illuminance have been completed over the past 15
years. Especially the RADIANCE backward raytracer (Ward and Rubinstein 1988) has been studied
systematically over the last decade (Mardaljevic
1995, Reinhart and Walkenhorst 2001, Reinhart and
Andersen 2006). These validation studies have
shown that RADIANCE can be combined with a
daylight coefficient approach to reliably model the
changing levels of daylight in a building over the
course of a year. Work plane illuminances can be
typically modeled with an accuracy of about 20%
which can be considered as sufficient for most design
purposes given that the human eyes itself is a
logarithmic sensor. A very recent validation study
suggests that the Mental Ray raytracer under 3ds
Max Design 2009 can also model indoor
illuminances with an accuracy comparable to
Daysim/Radiance (Reinhart and Breton 2009).
One could in principle conclude from these studies
that validated daylight simulation programs can now
be recommended to design teams interested in
implementing daylighting into their projects and that

Figure 1: Left view of the room; left - floor plan of the room.
the use of these programs will yield reliable results if
the investigated buildings are of comparable
complexity to the ones investigated in the validation
studies. But, given that previous validation studies
were carried out by a handful of simulation experts
one might actually wonder how accurate one can
expect the results of simulation novices to be?
Few studies have focused on the impact of the user in
the accuracy of simulation results by analyzing the
output of multiple users modeling the same
simulation case. Kummert (Bradley, Kummert and
McDowell 2004) compared the difference in
simulation results when three expert users applied
ANSI/ASHRAE Std 140-2001 to the TRNSYS
simulation program (TRNSYS 2009). The users
where categorized as a developer, a user/developer,
and an expert user. The study concluded that there is
a great leeway within a given software package to
make widely varying assumptions and yet still fall
well within the range of acceptably accurate results.
The study concluded that despite this modeling
latitude, knowledgeable users can still be confident
that their results will not dramatically vary from
those of other expert users (Kummert et al. 2004).
This conclusion might be reasonable for expert users
who understand the underlying assumptions and
limitations of a simulation program. But, are these
conclusions also valid for novice users? How large is
the error margin introduced by typical simulation
newcomers? Can common mistakes be identified and
potentially be avoided in the future?
To answer these questions this paper compares
daylight factor simulation results for a best practice
model of an L-shaped perimeter classroom to a total
of 69 novice/student models. The objectives of this
work are to indentify common mistakes that
simulation beginners make and to develop guidance
for software developers of how they could help users
to avoid making these mistakes.
In addition, a comparison of simulation results
obtained with ECOTECTs build-in split flux method
as opposed to RADIANCE is carried out for the 69
models in order to quantify in how far using a less

reliable simulation engine compromises the accuracy

of a simulation

METHODOLOGY
During the Fall 2005 and 2006 Terms the second
author asked a total of 87 students at the McGill
School of Architecture to model the daylight factor
distribution in one of the schools L shaped crit
rooms (Figure 1). The task was assigned as part of
the deliverables for an introductory course in lighting
and daylighting for 3rd year Bachelor of Architecture
students. The room is located on the ground floor of
the Macdonald-Harrington Building in Montreal,
Canada (Latitude: 4550N, Longitude: 7370W).
The building was designed by Sir Andrew Taylor and
built between 1896 and 1897. The room is daylit
through four windows and is characterized through
over 800mm thick walls and a 4350mm high ceiling.
The floor area is about 64 m2 and the room
dimensions in the widest sections are 11.95m along
the east-west axis and 8.17m wide along the northsouth axis. The optical characteristics of all walls and
windows were measured using a reference white
surface and a Hagner luminance meter1. The resulting
material properties are listed in Table 1.
Table 1. Building material optical properties
Ceiling

Diffuse reflectance 80%

Wall

Diffuse reflectance 80%

Floor

Diffuse reflectance 8%

Windows

Visual transmittance 65%

The students were requested to model the space in

ECOTECT and report the daylight factor at selected
positions within the space. The students were not
given any floor plans or sections of the space and
accordingly had to measure all dimensions
themselves, build a model in ECOTECT, assign
adequate material properties and run a daylight factor
simulation. In class they had previously learnt how to
do a daylight factor calculation using either the build1

http://www.hagnerlightmeters.com/products.htm

in split-flux method in ECOTECT or the export

function to Radiance. For the assignment they were
given the choice to use either program. It is
interesting to highlight that none of the students
carried out the simulation using RADIANCE. One
may infer from this information that most simulation
newcomers, whether students or practitioners, would
probably make the same choice given that the
limitations of the split-flux method are not widely
known. For this reason and in order to quantify the
difference in simulation results that one gets from
using either ECOTECT (v5.6) split-flux or
RADIANCE, all models were run using both
programs with RADIANCE (v3.8) acting as the
benchmark case for ECOTECT.
Another motivation for running the simulations in
RADIANCE was that the authors hope that the use of
the split-flux method in ECOTECT will ultimately be
phased out so that users have to use RADIANCE or
another validated engine, by default. The main focus
of this study is to understand what type of modeling
mistakes simulation newcomers make and - while a
simulation program such as ECOTECT can easily be
changed to another engine - it will always be up to
the user to enter building geometry data correctly and
to assign meaningful material properties.
Model Analysis
The model sample was narrowed down as follows:
Out of the 87 students who were enrolled in the fall
2005 and 2006 terms 9 did not submit their
ECOTECT files leaving 78 ECOTECT models for
the authors to review. In addition, even thought the
assignment had to be submitted individually, students
ended up working in groups. To accurately account
for the modeling mistakes done by each individual
user or user group, the model sample was narrowed
down to consider only one model per group. A total
of 9 models were discarded during this process
leaving 69 independent models. It is worthwhile
mentioning that it is unclear at this point whether
students who submitted the same ECOTECT model
had actually worked in a group or whether a student
had simply copied and submitted a peers file without
actually working on it.
Best Practice Model
A best practice model was created by the authors
using ECOTECT. Additional elements, such as a 2D
floor plan and trees, were created in a separate CAD
tool and imported individually in .3ds format into the
ECOTECT model. Interior surfaces were modeled
according to Table 1. The ground albedo was set to
20% and the outside trees were modeled as
transparent surfaces assuming a visual transmittance
of 30%. The diffuse reflectance of all neighboring
buildings was set to 50% and all window shades were
assumed to be fully opened. Figure 2 shows an
ECOTECT visualization of the best practice model.
A 32x40 horizontal grid with a total of 802 upward
facing sensors was set at a height of 800mm above

the floor, which corresponds to the height at which

the students were supposed to model the sensor grid.

Figure 2. Best practice model

As explained above, the practice model as well as all
student models were simulated using both the
ECOTECTs built in split-flux method as well as
RADIANCE. For the best practice model the
ECOTECT simulations were run using the Full
Precision setting in ECOTECT. For the student
models the ECOTECT simulations were rerun using
the lighting analysis setting that the students had
originally chosen. Table 2 lists the RADIANCE
simulation parameters that were used for all
simulations, ambient bounces (ab), ambient divisions
(ad), ambient sampling (as), ambient accuracy (aa)
and ambient resolution (ar). These simulation
parameters were chosen based on recommended
values from earlier Daysim validation studies and
correspond to a scene of high complexity as defined
in the Daysim tutorial (Reinhart 2006).
Table 2. RADIANCE simulation parameters
ab
6

ad
1500

as
100

aa
0.05

ar
300

Since RADIANCE simulations using the best

practice model served as the benchmark case against
which the student models were analyzed, a relative
error band of 10% was assumed for the RADIANCE
daylight factor simulations of the best practice
model. The 10% error band was derived from
Reinhart and Andersen (2006).
Analysis of the Model Sample
The analysis of the student models was carried out in
two steps. Initially, a qualitative overview of the 69
models was performed during which the authors
manually went through all of the models and
identified the most common simulation mistakes that
were done by the students. Based on this initial
screening the authors selected fifteen independent
model inputs that could be quantified for all models
(Table 3). These fifteen model inputs were grouped
into four categories: General, geometry, materials
and simulation settings.

Table 3. List of model inputs that were used to characterize the 69 student models.
Category

Question

PossibleAnswers

ErrorFrequency

General
Geometry

Q1: In which semester was the model built?

Q2: Was the model built within ECOTECT? Imported from a third
party unsuccessfully (i.e. not exploded or incomplete envelope)?
Or Imported successfully (i.e. with glazing or added in
ECOTECT)?

Fall 05 / Fall 06

2005 - 2006

Q3: Are interior room dimensions modeled accurately?

Q4: Are the window dimensions (size and position of window
openings) modeled accurately?
Q5: At what thickness are the walls modeled?
Q6: Are neighboring buildings modeled?
Materials

Q7: Are adjacent trees modeled?

Q8: Did the model use the customized NRC material library (as
opposed to the ECOTECT default library; Fall 06 only)?

24

0= Yes | 1= No

67

Q11: What was the modeled wall reflectance?

reflectance in % [target value 80%]

44
30
13

69
55
69

Q12: What was the modeled floor reflectance?

reflectance in % [target value 8%]

69

Q13: Were the sensor positions correctly (correct sensor height)?

Q14: Were the sensor modeled correctly (grid sufficiently fine and
inside envelope boundary)?

grid height in mm.[target 800mm]

18

0= Yes | 1= No

14

The first category (Q1) differentiated between the

school year in which a model was build. A difference
in the accuracy of simulation results between 2005
and 2006 was expected because for the fall 2006 term
the instructor provided (a) a few modeling tips for the
students such as remember that real walls have a
thickness and consider surrounding objects
(buildings, trees, etc.)2 and (b) encouraged students
to work with a customized material database for
ECOTECT3, which provided more realistic material
descriptions for lighting than the original ECOTECT
material library.
The second category was concerned with model
geometry. The initial screening revealed dramatic
geometry errors in nearly all of the models. These
errors were the results of several modeling
shortcoming. The building geometry editor in
ECOTECT actually builds all walls and ceiling as
single surfaces, i.e. with zero thickness. As a result
typical models build in ECOTECT tend to
substantially overestimate indoor daylight factor
levels especially in spaces such as the investigated
classroom room that had a wall thickness of 980 mm
(Q5). Another, more surprising, common error was
that many students modeled the window head of all
classroom windows as a straight line and not rounded
(Q4). A large number of students were unsatisfied
with the limited three-dimensional modeling
capabilities of ECOTECT and instead used another
CAD tool and imported their scene geometry into
ECOTECT for further analysis (Q2). While this is
Wording taken form original assignment.
http://irc.nrccnrc.gc.ca/ie/lighting/daylight/daysim/docs/NRC_Lig
htingLibrarySetup.exe (last accessed February 2009)
3

16

0= Yes | 1= No

Q10: What was the modeled ceiling reflectance?

Q15: What was the selected simulation precision in ECOTECT:

26

0=Yes | 1=wrong depth |

3=wrong height | 4=wrong depth
and height
0= Yes
1= No
Thickness in mm [target value
980mm]
0= Yes | 1= No

visual transmittance in % [target

value 65%]
reflectance in % [target value 80%]

Q9: What was the modeled glazing transmittance?

Simulation
Settings

0= Imported unsuccessfully | 1=
Built within ECOTECT | 2=
Imported Successfully

1= lo | 2= medium | 3= high |
4=very high | 5=full

might - in principle - have been a good idea it often

resulted in grossly erroneous models with typical
importing errors being that the whole scene was
imported in a single group with a fixed material
property, planes that were imported only as
construction lines resulting in missing walls and
ceilings, no planes in fenestrations, and out of scale
models. Two other frequent model shortcomings
were that external trees and neighboring buildings
were not included in the model. The thresholds for
identifying accurate space dimensions (Q3) were
established based on a study on error impact over
simulation results carried out by Thanachareonkit
(2008). Thanachareonkits study, concluded that for a
sidelit space simulated under a CIE overcast sky, a
variation of over 5cm on a sensor point will result in
a 10% illuminance variation for sensors located at
0,2, 3.2 and 6.2m from the window, compared to real
measurements. Assuming that a modeling error of
more than 5cm in window dimensions under a CIE
Overcast Sky will equally result in more than a 10%
error in simulation accuracy, a maximum of 10%
error on the space dimensions was allowed. In
conjunction with a 20x24 sensor grid (301 sensor
points for the space geometry) this corresponds to a
maximum of 5cm sensor displacement along the X
and Y axes. The threshold for accurate window
dimensions (Q4) was equally set to a 5cm from the
target level. The wall thickness was verified in
absolute values in relation to the 980 mm target value
(Q5). Considering the surrounding buildings and
trees (Q6 and Q7) were evaluated on a yes or no
basis.
For the material category the fact whether a student
had used the customized NRC material database for
ECOTECT (Q8) as well as the assigned surface
reflectances and transmittances (Q9 to Q12) were

verified. The simulation settings verified whether the

sensor grid was set to the correct height above the
floor (Q13) as well as whether all sensor grids were
properly positioned within the room (Q14). As
explained above the simulation precision (Q15) was
only used for the ECOTECT split-flux simulations.
RESULTS
In this section the simulation results of the best
practice model as well as the 69 student models are
compared between ECOTECT split-flux and
RADIANCE.
Afterwards
the
RADIANCE
simulations of the 69 student models are compared to
the best case results. Through the results section the
mean daylight factor (DFmean) and the percentage of
sensor above a 2% DF (Area>2%) are used as
daylighting metrics for the models.
Comparison of ECOTECT and RADIANCE
The DFmean and Area>2% for the best practice model
simulated in ECOTECT and RADIANCE were
0.55% and 0% for ECOTECT compared to 2.59%
and 41.65% for RADIANCE. Figures 4 and 5
visualize these drastic discrepancies between the two
programs. The differences can be attributed to the
fact that the ECOTECT split-flux method does not
take multiple lighting reflectances into account which
leads to a gross under-prediction of interior lighting
levels in the presence of thick walls, light-shelves,
overhangs or other elements that are commonly
found in daylit spaces.
Table 3. Comparison of RADIANCE and ECOTECT
simulation results for Best Practice Model
ECOTECT

Average DF
1.53

Area Above 2% DF
6.89

RADIANCE

2.59

41.65

Figure 4. ECOTECT Daylight Factor Simulation

Figure 5. RADIANCE Daylight Factor Simulation

Figure 6 shows ECOTECT and RADIANCE
simulations of DFmean for the 69 student models. The
benchmark simulation result of the Radiance best
practice model is shown was will with an error band
of 10%. Comparing the ECOTECT and Radiance
results for each student model individually, one finds
for 60 out of the 69 student models RADIANCE
predicts higher daylight factors than ECOTECT: In
2005 the DFmean was 5.9% for the ECOTECT
simulations compared to 8.3% for RADIANCE. In
2006 the numbers were 4.5% and 8.2%, respectively.
When analyzed by year, the 2005 ECOTECT student
models tend to their repsective RADIANCE models
than in 2006. The reason for this is that in 2005 only
5 out of 39 students considered modeling wall
thicknesses whereas all 30 student models in 2006
had non-zero wall thickness4. When wall thicknesses
are modeled, daylight penetration resulting from
reflected light is much more relevant than direct sky
contributions which means that ECOTECT tends to
predicts lower levels than RADIANCE. The only
excetion to this rule occurs if a student used low
simulation precision settings. In that case ECOTECT
might accidentally predict higher levels than
RADIANCE.
The fact that the ECOTECT simulations are
sometimes higher and sometimes lower than
Radiance underlines that that using the ECOTECT
split-flux method cannot even be interpreted as a
worst case scenario for a space but that it may both
grossly over and under predict real daylight factor
levels in a space. The authors therefore believe that
this simulation engine should not be used at all. This
topic is revisited below.
Comparison of Student Models and Best Practice
Results
A comparison of the RADIANCE simulations in
Figure 6 shows that in both years the results from the
student models were significantly higher than for the
best practice model. In 2006, two models (17 and 25)
show extreme values of 32.92% and 48.60%. In both
4

A direct consequence of the modeling tips given by

the instructor.

Fall2005 ECOTECT&RADIANCEAverageDaylightFactorSimulation
12

PercentageDF[%]

10
8
6
4
2
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

ECOTECTSimulation

RADIANCESimulation

Fall2006 ECOTECT&RADIANCEAverageDaylightFactorSimulation
26.0

12

32.9

48.6

PercentageDF[%]

10
8
6
4
2
0
1

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

ECOTECTSimulation

RADIANCESimulation

Figure 6. Comparison of ECOTECT and RADIANCE average daylight factor results for the 2005and 2006
models. The simulation results are compared to the best practice RADIANCE model for which a 10% error
band is plotted as well.
cases these high results stem from and unsuccessful
import of the scene geometry from another CAD
tools into ECOTECT: In model number 17 the
ceiling plane was imported as construction lines only.
Similarly, in model 25 most of the perimeter walls
were imported as construction lines.
Figure 7 shows a frequency distribution of the
RADIANCE results from Figure 6 binned into 1%
slots for 2005 and 2006. Contrary to what one would
expect, the two distributions do not approximate a
normal distribution but the results lie by around
200% to 300% to the 400% over the best practice
model results. In fact in 2005 only one models was in
either the 2% or 3% bins that can be interpreted as
the acceptable result range. In 2006 the number of
acceptable models grew to six that still only
corresponds to 16% of the submitted student models.
As will be shown in the following the reason for the
different types of models submitted in 2006 can be
largely attributed to the simulation tips provided by
the instructor in the second year.

Figure 7. Frequency distribution of RADIANCE

student models for 2005 and 2006.
Figure 8 shows the frequency distribution of the
relative error of DFmean grouped by how the scene
geometry was build (Q2). Each scene was either built
within ECOTECT or imported into ECOTECT from

RelativeErrorbyModelSource
14
12

Frequency

10
8

9
8
7
6
5
4
3
2
1
0

RelativeErrorofECOTECTModelsbyYear

1 0.5

6
4

0 0.5 1 1.5 2 2.5 3 3.5

ModelbuiltwithinEcotect 2006
ModelbuiltwithinEcotect 2005

Figure 9. Frequency distribution of ECOTECT

models only by year.

2
0
1 0.5 0 0.5 1
ModelImportedWrong
ModelImportedok

1.5

2 2.5 3 3.5 4
ModelcreatedinEcotect

Figure 8. Frequency distribution of the relative error

of DFmean by model source.
One could expect that the models that have been
imported correctly would generated the most reliable
results since they take wall thickness into account
and generally model the room dimensions in greater
accuracy than ECOTECT models due to the limited
3D modeling capabilities in ECOTECT. Surprisingly,
this is not the case since the models that were
imported correctly do seem to be systematically
better than the other models. The reason for this
finding is that all students models were somehow
seriously flawed. Even if they properly imported the
model geometry students forgot to explode the
imported scene and assign meaningful material
properties and in many instance window glazings
were somehow forgotten. In some instance scene
elements were simply imported as construction
lines5.
Figure 9 shows a frequency distribution of the
models that were built in ECOTECT only separated
by year. While the 2005 models area scattered across
a large range the 2006 model are actually much
closer together. The reason for this is that in 2006 all
ECOTECT models had non-vanishing wall
thicknesses. Despite of the general improvements of
the models build in ECOTECT in 2006 over 2005 all
models still experiences some shortcoming which is
why they still lay a factor of 2 over the best practice
model.

Frequency

another CAD tool. As explained above, in many

instances this import leads to corrupted models which
is why the imported models are divided into Model
imported OK and Model imported wrong.

In this context it is important to note that the

ECOTECT 5.6 import function while greatly
improved over previous version- is still rather prone
to errors. In Some instances the ECOTECT models
with missing planes but when exported to RADINCE
the planes were included in the RADIANCE model.

DISCUSSION
The previous section presented the simulation results
for all 69 models and compared them to a best
practice model results. The sobering results of the
students models reveal that even simple mature
workflows have to be taught in greater detail for
novices to obtain accurate simulation results. Part of
the problem can be attributed to the limited 3D
modeling capabilities of the ECOTECT GUI.
Modeling three-dimensional spaces using zero
thickness walls does not lead to acceptable results.
One may question if the test space was
unrepresentatively difficult for ECOTECT to model
due to the uncommonly high wall thicknesses of 980
mm. The authors believe not because while modern
buildings tend to have thinner walls they come with
advanced faade features (overhangs, shading
devices etc.) which also have to be geometrically
modeled.
At the same time, import workflows from other
programs have to be further streamlined and properly
explained to software users; as many as 72% of the
users who tried importing geometry were
unsuccessful. Material properties, on the other hand,
had a relatively minor effect on the model quality in
this study mainly because the scene geometries were
modeled inadequately.
Users were surprisingly careless when modeling
space geometries, something unexpected from
architecture students. The study demonstrates that
students pay attention to simulation tips, which is a
real opportunity for instructors and a good sign for
the adoption of modeling guidelines.
There is currently a push toward moving away from
the daylight factor as a performance metric for
daylighting and using climate-based metrics instead
(Reinhart, Rogers and Mardaljevic 2006). While this
study used the daylight factor to determine the
quality of a simulation one should assume that the
results would have been largely the same if a climate
based metric such as daylight autonomy or useful
daylight illuminance had been used instead. The
nature of the simulation errors lay in basic model

properties, such as scene geometry and material

properties, which are of equal importance for static
and dynamic daylighting metrics.

CONCLUSION
When comparing the simulation results reported by
ECOTECTs built-in engine and RADIANCE for the
best practice model, ECOTECT reported a dramatic
79% lower daylight factor and a reduction in the
Area above>2% DF from 41% to a 0%. For the 69
student models, ECOTECT simulations reported on
average a 36% lower result and a 72% lower MBE
than the same simulations run in RADIANCE.
Furthermore individual ECOTECT models both
grossly over- and under predicted daylight factor
levels according to RADIANCE. This finding
suggests that ECOTECT-based daylight factor
predictions cannot be considered to be worst case
assumptions and that RADIANCE should always be
used instead of the build-in ECOTECT daylighting
engine.
Comparing the quality of the student models
submitted in 2005 and 2006 suggests that if the
instructor provides simulation tips they are being
followed. This finding puts an additional emphasis
on the importance of high quality teaching material
to complement simulation workflows: Offering
simple simulation tips in the 2006 version of the class
considerably improved the accuracy of the simulation
results. Conversely, when no explicit modeling
guidelines were provided in 2005, students tend to
make dramatic errors, especially in relation to
geometry input. If only a few parameters are
addressed, users tended to overlook the impact of
other variables, and continued to obtain inaccurate
simulation results.
The authors conclude that even a simple set of
modeling guidelines is generally required to
complement any simulation workflow, however
simple it may be, in order to ensure that simulation
novices can follow it accurately.

ACKNOWLEDGEMENT
The authors would like to thank Lia Ruccolo for
helping with some of the initial student model
analysis.

REFERENCES
ASHRAE, ed. ASHRAE/ IESNA 90.1 Standard 2007 - Energy Standard for Buildings Except
Low-Rise
Residential
Buildings.
2007,
American Society of Heating Refrigerating and
Air-conditioning Engineers
Bradley, E., Kummert, M., McDowell T., 2004.
Experiences with and Interpretation of Standard
Test Methods of Building Energy Analysis

Tools. The Bi-Annual Conference on IBPSACanada, eSIM 2004, Vancouver.

Galasiu A D, Reinhart CF, Current Daylighting
Design Practice: A Survey, Building Research
& Information 36:2 pp. 159 174, 2008.
Mardaljevic J. 1995. Validation of a Lighting
Simulation Program under Real Sky Conditions.
Lighting Research & Technology 27(4): 181188.
Reinhart C F, Breton P-F. 2009. Experimental
Validation of Autodesk 3ds Max Design 2009
and Daysim 3.0, NRC-IRC report B3241.1,
(www.autodesk.com/us/3dsmaxdesign/B3241.M
entalRayValidation_v3.pdf ).
Reinhart C F and Andersen M. 2006. Development
and validation of a Radiance model for a
translucent panel. Energy & Buildings 38(7):
890-904.
Reinhart C F and Fitz A. 2006. Findings from a
survey on the current use of daylight simulations
in building design. Energy & Buildings 38(7):
824-835.
Reinhart C F, Mardaljevic J and Rogers Z. 2006.
Dynamic Daylight Performance Metrics for
Sustainable Building Design. Leukos 3(1):1-20.
Reinhart C F and Walkenhorst O. 2001. Dynamic
RADIANCE-based Daylight Simulations for a
full-scale Test Office with outer Venetian
Blinds. Energy & Buildings 33(7): 683-697.
Thanachareonkit, A. 2008 Comparing Physical and
Virtual Methods for Daylight Performance
Modelling Including Complex Fenestration
Systems, Ecole Polytechnique Federale de
Lausanne, Switzerland.
TRNSYS, The Transient Energy Simulation Tool.
http://www.trnsys.com/ (last visited February
20th, 2009).
US-DOE (US Department of Energy). 2007.
www.eere.energy.gov/buildings/tools_directory/
USGBC, 2006. LEED for New Construction Ver 2.2,
US Green Building Council.
Ward, G and Rubinstein F. 1988. A New Technique
for Computer Simulation of Illuminated Spaces.
Journal of the Illuminating Engineering Society
1: 80-91: