See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/258136062

Factorial and Criterion Validity of Scores of a Measure of Belonging in Youth

Development Programs

Article  in  Educational and Psychological Measurement · October 2002

DOI: 10.1177/001316402236882

CITATIONS READS
106 3,403

2 authors, including:

Dawn Anderson-Butcher
The Ohio State University
153 PUBLICATIONS   2,843 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

OCCMSI View project

All content following this page was uploaded by Dawn Anderson-Butcher on 12 October 2017.

The user has requested enhancement of the downloaded file.

 10.1177/001316402236882
EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT
ANDERSON-BUTCHER AND CONROY

FACTORIAL AND CRITERION VALIDITY OF SCORES

OF A MEASURE OF BELONGING IN
YOUTH DEVELOPMENT PROGRAMS

DAWN ANDERSON-BUTCHER
Ohio State University

DAVID E. CONROY
Pennsylvania State University

Many youth development programs, including the Boys & Girls Clubs of America, fea-
ture belonging as a central piece in their theories of change. From a psychometric per-
spective, little is known about measures of belonging. This research examined the
factorial validity, internal consistency, and predictive validity of scores from one mea-
sure of belonging to an after-school youth development program. Confirmatory factor
analysis yielded a five-item measure from a calibration analysis that demonstrated
“tight” cross validity in a cross-validation sample as well as factorial invariance between
females and males. Internal consistency estimates for this 5-item scale exceeded .90 in
both samples. Belonging scores were positively related to actual program attendance
over a 6-month period, self-reported attendance in the last week, and protective factors
found in communities. Belonging scores were moderately and negatively related to com-
munity-based risk factors.

Youth development programs are known to reduce problem behaviors

(e.g., academic failure, substance abuse, juvenile delinquency, gang involve-
ment, early pregnancy) and enhance social competence (e.g., enhanced self-
esteem, interpersonal skills, initiative, bonding and identity development,

The authors would like to acknowledge Bob Dunn, executive director of the Boys & Girls
Clubs of Murray and Midvale, Utah, for his assistance and support of this research. Both authors
contributed equally to the study. Correspondence concerning this article should be directed to
Dawn Anderson-Butcher, 325D Stillman Hall, 1947 College Road, The Ohio State University,
43210-1162; e-mail: [email protected].
Educational and Psychological Measurement, Vol. 62 No. 5, October 2002 857-
DOI: 10.1177/001316402236882
© 2002 Sage Publications
857
858 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

peer and family relationships).1 A pivotal mechanism in the theories of

change for many popular youth development programs involves developing a
sense of belonging in youth (cf. Boys & Girls Club of America, 1998; Burt,
Resnick, & Novick, 1998; National Association of Elementary School Prin-
cipals, 1999). The present research examined the psychometric properties of
scores on one popular and easily administered self-report measure of youths’
sense of belonging to an after-school youth development program.

The Role of Belonging in Youth Development Programs

Developing belonging among participants is a central task in many youth

development programs. For instance, guidelines suggest that after-school
programs should make youth feel welcome, comfortable, and accepted
(National Association of Elementary School Principals, 1999). Similarly,
agencies such as Big Brother/Big Sisters, Inc. and the Boys & Girls Clubs of
America have discussed the importance of fostering relationships and con-
nections between mentors, staff, and youth participants (Boys & Girls Clubs
of America, 1998; Burt et al., 1998). Many authors suggest that youth get
more from their involvement when participation includes having a connec-
tion or bond with a particular program, when youth are committed to the
agency’s rules and norms, and/or when youth feel as if they are contributing
members (Anderson-Butcher, 2000; Brown, 1988; Heath, 1999; Holland &
Andre, 1987; Larson, 2000; Youniss, Yates, & Su, 1997). From a program
leaders’ perspective, there is more to participation in these programs than
mere attendance, and research suggests that the positive effects of participa-
tion may be amplified when youth adopt a sense of belonging and connection
to the agency or program.
Several theories also suggest the importance of attachment, commitment,
and bonding in promoting prosocial behaviors among children and youth.
For instance, the notion that belonging, expressed commitment, bonding, and
attachment with prosocially oriented individuals or organizations/institu-
tions will positively enhance youth development is implicit in social learning
theory (Catalano & Hawkins, 1996; Sutherland & Cressey, 1978), social con-
trol theory (Hirschi, 1969), the social development model (Catalano &
Hawkins, 1996; Hawkins, Catalano, Kosterman, Abbott, & Hill, 1999), and
social identity theory (Tajfel & Turner, 1985; Turner, 1982). Therefore,
developing such a sense of belonging in youth development programs may be
an important mechanism for youth development.
Paradoxically, researchers and practitioners have traditionally limited
their assessment of participation to self-report or objective measures of atten-
dance. To borrow a common phrase from youth development specialists,
“kids vote with their feet.” Only recently have we begun to explore the protec-
tive influences of one’s sense of belonging, bonding, and attachment to these
 ANDERSON-BUTCHER AND CONROY 859

agencies (Anderson-Butcher & Fink, 2001; Hawkins & Catalano, 1995;

Hawkins et al., 1999; McLaughlin, Irby, & Langman, 1994; O’Donnell,
Michalak, & Ames, 1997; Resnick et al., 1997).

The Nature of Belonging

Belonging appears to be a fundamental psychological need for humans

(Baumeister & Leary, 1995; Ryan & Deci, 2000). Attachment is a similar
construct that has been investigated in great detail as an individual difference
variable (Ainsworth, Blehar, Waters, & Wall, 1978; Bowlby, 1982). Like-
wise, Somers (1999) explored environmental models that include
belongingness with family, friends, work/school, and the neighborhood.
More limited efforts have been made to understand belonging as the nexus of
the individual and organization (i.e., program or agency specific). Clarifying
definitions from work on youths’ commitment, engagement, and
connectedness to schools and related programs may inform our understand-
ing of this important concept.
Commitment refers to the combination of the perceived importance of
one’s work, level of interest in the activity, personal enjoyment, effort, and
attendance (M. Arthur, Pollard, Hawkins, & Catalano, 1997). Commitment
also has been used to describe being connected to or buying into the program
(Martinek, 1999). Engagement refers to youths’beliefs about whether school
or related organizations are fun and exciting or if youth look forward to learn-
ing new things (Antonovsky, 1983; Rosenfeld, Richman, & Bowen, 2000) as
well as how often youth come to class prepared and how often they feel bored
(Lee & Smith, 1993). Finally, connectedness reflects youths’ perceptions of
whether they are treated fairly by their teachers, how close they feel to people
at school, whether they feel like they are a part of the school, and how satis-
fied they feel in the school (Blum & Rinehart, 1998; Rosenfeld et al., 2000;
Small, 1991). Collectively, these constructs span similar conceptual space as
the construct of belonging.
In reflection, the domain of belonging has aspects of commitment,
engagement, and connectedness. The present study, however, searches for
the common, shared space between and among these domains. It is this
shared domain that we believe constitutes the construct of belonging. Fur-
thermore, the study also aims to develop a brief scale that can be readily used
at the program level, as opposed to more general, lengthy scales that are sys-
tem specific (see Social Research Institute, 1997; Somers, 1999).

Measuring a Sense of Belonging

Several youth development agencies are currently using a 10-item mea-

sure of belonging initially designed for use in a chartered Boys & Girls Club.
860 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Six questions related to comfort and inclusion in a program (Items 1, 2, 3, 4,

5, and 7) were drawn with permission from a sense of belonging subscale
developed by the Arizona Prevention Research Center for use in the Boys &
Girls Club of East Valley (B. Arthur, 1997).2 Four additional items represent-
ing other indices of belonging were added to the instrument. Items on the
final scale explored the domains of commitment (Items 5, 8), engagement
(Items 2, 4, 9), and connectedness (Items 1, 3, 6, 7, 10). The scale items are
presented in Table 1. Although the use of this instrument in Boys & Girls
Clubs and other youth development agencies has grown, little is known about
the psychometric properties of this instrument. Furthermore, the present
instrument samples a variety of domains related to belonging. It is important
to evaluate whether the commitment, engagement, and connectedness items
relate to a common domain of belonging. If they do not, removing items that
do not appear to sample the common domain will clarify the nature of this
belonging measure.

Validity of Inferences About Score Meaning

The impact of program evaluation research may be limited by the validity

of the score interpretations for measures used in the evaluation. In the unified
theory of validity, Messick (1995) outlined several types of evidence for
score validity. We focus here on factorial, convergent, and predictive validity
because of their relevance in the assessment of belonging. Factorial validity
describes how well an underlying dimension (or set of dimensions) can be
used to summarize relationships between item responses. Evidence of facto-
rial validity is critical in the early stages of instrument development because
it provides a basis for calculating scale scores used in all other tests of score
validity. Convergent validity provides evidence that scores are related to the-
oretically related measures (e.g., belonging scores should be at least moder-
ately related to program attendance). Predictive validity refers to whether the
measure is associated with theoretically related outcomes or criteria. For
instance, youth who feel a sense of belonging to an after-school program
should be more likely to attend that after-school program than youth who do
not feel such a sense of belonging.
The first purpose of this research was to investigate the factorial validity of
belonging scores. Confirmatory factor analysis was the principal analytic
tool for these analyses because the theoretical underpinnings of the belong-
ing construct guided instrument development and item selection/removal
(Thompson, 1997; Thompson & Daniel, 1996). If the a priori model did not
fit well, an iterative process of model modification was undertaken to develop
a well-fitting model. Item content and theoretical considerations were deter-
mining factors in all modifications, but empirical indices (e.g., parameter
estimates, standardized residuals, squared multiple correlations [SMCs])
 Table 1
Descriptive Statistics for the Sense of Belonging Scale Items in the Calibration/Cross-Validation Samples

Item N M SD Median Skewness Kurtosis

1. I don’t have many friends at the program. 296/168 2.34/2.20 1.20/1.23 2/2 0.15/0.38 –1.53/–1.49
2. I feel comfortable at the program. 298/165 2.93/2.72 1.12/1.18 3/3 –0.70/–0.41 –0.89/–1.34
3. The leaders at the program make me feel
wanted and accepted. 297/166 2.96/2.77 1.13/1.52 3/3 –0.68/–0.43 –0.96/–1.26
4. I feel like I am an important member of
the program. 294/157 2.77/2.67 1.14/1.46 3/3 –0.39/–0.31 –1.27/–1.33
5. I wish I were not a part of the program. 297/165 1.69/1.85 1.00/1.05 1/1 1.26/0.94 0.32/–0.43
6. I am disliked by kids at the program. 293/159 1.88/1.76 0.97/0.94 2/1 0.84/1.06 –0.35/0.12
7. I am a part of the program. 292/165 2.96/2.58 1.21/1.26 3/3 –0.69/–0.10 –1.14/–1.65
8. I am committed to the program. 287/160 2.75/2.54 1.16/1.22 3/3 –0.32/–0.11 –1.36/–1.56
9. I am supported at the program. 291/163 2.87/2.60 1.13/1.24 3/3 –0.61/–0.18 –1.03/–1.60
10. I am accepted at the program. 287/163 3.01/2.67 1.11/1.21 3/3 –0.83/0.30 –0.70/–1.49

Note. Calibration sample statistics are presented first, followed by cross-validation sample data. In the calibration sample, SE skewness was 0.14 for all items, and SE kurtosis ranged from
0.28 to 0.29. In the cross-validation sample, SE skewness was 0.19 for all items, and SE kurtosis ranged from 0.38 to 0.39.

861
862 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

also informed modifications. Such a strategy carries the risk of capitalizing

on chance in the modified model (MacCallum, Rosnowski, & Necowitz,
1992). Assessing the generalizability or cross validity of modified solutions
is essential.
Different levels of cross validity exist (Bentler, 1980; MacCallum, Ros-
nowski, Mar, & Reith, 1994). “Loose” cross validation occurs when a model
replicates in an independent sample with no constraints on the parameters
used to estimate the model in the second sample. “Partial” cross validation
involves reproducing the observed covariances in a second sample using one
or more sets of parameter estimates generated in the first sample. Invariance
analyses using chi-square difference tests provide a tool for testing the effects
of parameter constraints (cf. Jöreskog & Sörbom, 1996). “Tight” cross valid-
ity exists when all of the parameters used to estimate a model are invariant
between samples (i.e., when parameters estimated in one sample can repro-
duce observed covariances in an independent sample).
Measurement equivalence (i.e., invariance) between groups is a require-
ment if meaningful group comparisons are to be made (Drasgow & Kanfer,
1985). Previous research identified gender differences in risk and protective
factors and related problem behaviors among youth (McGee, Feehan, Wil-
liams, & Anderson, 1992; Moffitt, Caspi, Dickson, Silva, & Stanton, 1996;
Parke & Slaby, 1983). Thus, a measure of belonging that might be used in
such research should demonstrate equivalence for females and males. The
present study used invariance analyses to assess the equivalence of the
belonging score model for females’ and males’ responses. It also is important
to examine the predictive validity of the belonging scores (i.e., whether the
belonging scores were associated with theoretically related outcomes or cri-
teria). Therefore, the present study explored the relationships among belong-
ing scores, attendance at the program, and various community risk and pro-
tective factors.

Purpose

To summarize, little is known about the factorial and predictive validity of

the 10-item model of belonging scores used by several Boys & Girls Clubs
and related youth development agencies. Specific goals of this research
included (a) evaluating the factorial validity of the a priori 10-item model of
belonging responses, (b) modifying the a priori belonging model as needed to
improve the model, (c) establishing the cross validity of the modified model
in an independent sample, and (d) testing the invariance of the belonging
model across samples of females and males. In addition, this research esti-
mated the internal consistency of belonging responses and evaluated the con-
vergent and predictive validity of belonging scores. Belonging was hypothe-
sized to be (a) positively related to attendance patterns among youth and
 ANDERSON-BUTCHER AND CONROY 863

community rewards and opportunities for prosocial involvement and (b) neg-
atively related to laws and norms favorable to drug use, low neighborhood
attachment, and high community disorganization.

Method

Participants

The calibration and cross-validation samples comprised 309 and 185 par-
ticipants, respectively. After listwise deletion of cases with missing data,
these samples contained 271 and 146 cases, respectively. Participants in the
calibration sample were recruited from a Boys & Girls Club located in a large
western city. Female (n = 140) and male (n = 139) participants ranged in age
from 9 to 16 years (M = 11.3 years, SD = 2.23). Thirty participants did not
report gender. This calibration sample was culturally diverse in that it com-
prised Caucasians (45%), Hispanics (36%), African Americans (7%), Native
Americans (7%), and other racial groups (5%).
The cross-validation sample was recruited through local United Way
agencies such as Boys & Girls Clubs, Big Brothers/Big Sisters, Boy Scouts,
and local neighborhood settlement houses and community centers. Female (n
= 63) and male (n = 80) participants ranged in age from 9 to 18 years (M =
13.60, SD = 2.53). Forty-two participants did not report gender. Within this
sample, participants represented Caucasians (54%), Hispanics (19%), Afri-
can Americans (7%), Native Americans (7%), Asian Americans/Pacific
Islanders (7%), and other racial/ethnic groups (6%).

Instrumentation

Belonging. The 10-item Belonging scale described in the introduction and

in Table 1 was developed by the first author for use in a chartered Boys &
Girls Club located in a large western city. Responses were made on a 4-point
Likert-type scale (NO! = 1, no = 2, yes = 3, and YES! = 4) modeled after M.
Arthur et al. (1997). These response items were chosen based on the first
author’s experiences collecting data with similar populations. Children and
youth completing instruments using these response items have reported ade-
quate comprehension and differentiation (Anderson-Butcher, 2000; Social
Research Institute, 1997).

Attendance. Participants reported how many times in the past 7 days they
attended the program on a 7-point ordinal scale composed of the following
choices: None, 1, 2, 3, 4 to 5, 6 to 10, and 11 or more. This ordinal scale format
was used to assist the youth with the recording of their behaviors. It was
believed by the researchers that a scale format, as opposed to an open-ended
864 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

format, would decrease the amount of missing data reported. Program

records also were examined to calculate a frequency score for total atten-
dance (in days) during the past 6 months.

Risk and protective factors. Community risk and protective factors among
youth were measured with 27 items from the following scales on the Utah
Division of Substance Abuse Needs Assessment Survey (O’Donovan, 1996;
Social Research Institute, 1997): Low Neighborhood Attachment (α = .84),
High Community Disorganization (α = .80), Laws and Norms Favorable to
Drug Use (α = .86), Community Rewards for Prosocial Involvement (α =
.71), and Community Opportunities for Prosocial Involvement (α = .63). The
response scale for all items except community opportunities for prosocial
involvement included NO! (1), no (2), yes (3), and YES! (4). Community
opportunities were rated either Yes (1) or No (2). Possible scores on each
subscale ranged from 1 to 4. Scores on the Risk and Protective Factor
subscales are predictive of substance use and antisocial behavior among
youth (O’Donovan, 1996).

Procedures

The belonging questionnaire and the community risk and protective factor
measure were administered as part of a larger battery of instruments that
required approximately 40 minutes to complete. The order of administration
was not randomized; the belonging questionnaire and the community risk
and protective factors were administered in the middle section of the battery.
In light of the length of the test battery and the potential for younger partici-
pants to become fatigued, bored, or lose their concentration, participants
younger than age 12 (n = 70) were administered the questionnaires in no
more than three sessions within a 1-week period. As such, it is possible that
youth were administered the measures used within this study during separate
measurement settings.

Data Analysis

Covariance matrices were generated using PRELIS 2.2 (Scientific Soft-

ware International, Chicago) for input to LISREL 8.3 (Scientific Software
International, Chicago).3 Parameters were estimated using the maximum
likelihood algorithm, a standard estimation technique with ordered-categori-
cal data (Hoyle & Panter, 1995). A variety of absolute and relative (or incre-
mental) fit indices were consulted to assess model fit. Absolute fit indices
included the χ2 statistic and goodness-of-fit index. Relative fit indices
included the nonnormed fit index (NNFI) (Bentler & Bonett, 1980) and the
comparative fit index (CFI) (Bentler, 1990). Finally, the root mean square
 ANDERSON-BUTCHER AND CONROY 865

error of approximation and standardized root mean residual (SRMR) were

consulted. Misspecified models are less likely to be accepted if models are
rejected when the (a) NNFI or CFI is less than 0.95, and (b) SRMR is greater
than 0.09 (Hu & Bentler, 1999).
To evaluate sources of error in the model and inform model respecifica-
tions, parameter estimates, SMCs, and standardized residuals were exam-
ined. Items were considered for removal when they (a) were not significantly
related to the latent factor (p < .05), (b) shared less than half of their variance
with the latent factor (SMC < .50), or (c) involved multiple statistically sig-
nificant standardized residuals (i.e., > ± 2.0).

Results
Table 1 presents descriptive statistics for each scale item in the calibration
and cross-validation samples. The sizes of the calibration and cross-valida-
tion samples were sufficient because the ratio of sample size to variables in
the models tested with each sample was greater than 20:1 and the ratio of
indicators to factors was always greater than or equal to 5:1 (Loehlin, 1992;
Marsh, Hau, Balla, & Grayson, 1998; Tanaka, 1987). The multivariate nor-
mality of data was examined using PRELIS 2.20 (Scientific Software Inter-
national, Inc., Chicago). In the calibration sample, Mardia’s (1970, 1974)
coefficients (normalized values) for multivariate skewness and kurtosis were
estimated to be 21.54 and 14.04, respectively. In the cross-validation sample,
these estimates were 16.37 and 10.88, respectively. Although these values are
statistically significant, it is known that large samples positively bias
Mardia’s coefficient (Bollen, 1989; Mardia, 1970, 1974). These values are
comparable to those found in other research using similar parameter estima-
tion techniques, so they are reported primarily for descriptive purposes
(Boomsma, 2000).
A principal-axis factor analysis was performed to test whether one factor
was likely to underlie responses as predicted by the theory used to develop the
instrument. The first eigenvalue from this analysis in the calibration sample
was 5.46; the second and third largest eigenvalues were 1.42 and 0.86. The
first factor accounted for 55% of the variance in responses, and the second
factor was not readily interpretable. We concluded that responses to the 10
items represented a single domain and that a model with only one factor was
appropriate for further analysis.
In the confirmatory factor analysis, all 10 items were used to model a sin-
gle latent variable. One pattern coefficient was fixed at 1.0 to establish a met-
ric for the latent variable, and the factor variance was freely estimated.
Uniquenesses were estimated for each item, and no covariances between
uniquenesses were modeled.
866 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Table 2
Fit Indices for Confirmatory Factor Analyses on Calibration Sample Data

2
Model df χ RMSEA (90% CI) SRMR GFI NNFI CFI

A priori 10-item model 35 177.83 .13 (.11 – .15) .06 .88 .90 .92
Removed Item 5 27 129.70 .12 (.10 – .15) .04 .90 .92 .94
Removed Items 5
and 1 20 118.41 .14 (.12 – .17) .03 .89 .92 .94
Removed Items 5, 1,
and 6 14 115.74 .17 (.15 – .20) .04 .88 .91 .94
Removed Items 5, 1,
6, and 3 9 59.50 .14 (.10 – .17) .03 .94 .94 .96
Removed Items 5, 1,
6, 3, and 4 5 11.74 .07 (0.0 – .12) .01 .98 .99 .99

Note. RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized
root mean residual; GFI = goodness-of-fit index; NNFI = nonnormed fit index; CFI = comparative fit index.

Model Modifications

As seen in Table 2, the fit of the a priori 10-item, one-factor model was
suboptimal. All parameter estimates were statistically significant (p < .05),
but three items had SMCs less than the operationally defined criterion of .50.
Item 5 had the lowest SMC (.00) and was removed. The resulting model had
slightly improved fit, but 2 items still had unacceptable SMCs. Item 1
(SMC = .04) was removed, and the resulting model demonstrated a slightly
improved fit. One item (Item 6, SMC = .08) still did not appear to tap the
latent variable to the same extent as the other items (mean SMC = .72, SD =
.04). When Item 6 was removed, all items had SMC values in excess of .65;
however, Item 3 had the single largest standardized residual (4.99, with Item 4)
as well as the most standardized residuals greater than ±4.0 (n = 3), indicating
that this item introduced a notable amount of error variance. When Item 3 was
removed, model fit improved substantially. Item 4 had the single largest stan-
dardized residual in the resulting model (5.36, with Item 2) and was the only
item with three statistically significant standardized residuals. This item was
removed, and the resulting 5-item model (Items 2, 7, 8, 9, and 10) showed
excellent fit to the data in the calibration sample. The internal consistency of
responses to these items in the calibration sample was .93.

Cross Validation

As seen in Table 3, the 5-item Belonging scale replicated in the cross-vali-

dation sample, providing support for loose cross-validity of the model.
Testing the partial cross validity of the model required establishing the equiv-
alence of (a) covariance matrices and (b) factor structures. As seen in Table 3,
both tests were successful. As seen from the chi-square difference tests in
Table 3, progressive constraints on item pattern/structure coefficients, factor
 Table 3
Fit Indices for Cross-Validation and Invariance Analyses

Model df χ2 RMSEA (90% CI) SRMR GFI NNFI CFI χ2 Difference df Difference p

Loose cross validation 5 15.60 .12 (.05 – .19) .02 .96 .97 .99
Equal covariance matrices 15 21.09 .04 (0.0 – .08) .11 .06 1.00 1.00
Fixed factor structure 10 27.34 .09 (.05 – .13) .02 .96 .98 .99
Partial cross validation:
fixed structure and item
pattern/structure coefficients 14 28.29 .07 (.03 – .10) .03 .96 .99 .99
Partial cross validation: fixed
structure, item pattern/
structure coefficients, and
factor variance 15 30.57 .07 (.03 – .10) .12 .96 .99 .99
Tight cross validation: fixed
item pattern/structure
coefficients, factor variance,
and uniquenesses 20 42.29 .07 (.04 – .10) .11 .93 .99 .99
Model comparisons
Item pattern/structure
coefficients—structure 0.95 4 > .05
Factor variance—item pattern/
structure coefficients 2.28 1 > .05
Uniquenesses—factor variance 11.72 5 > .05

Note. RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean residual; GFI = goodness-of-fit index; NNFI = nonnormed fit index; CFI
= comparative fit index.

867
868 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Table 4
Standardized Invariant Parameter Estimates Demonstrating Tight Cross-Validation—
Common Metric Completely Standardized Solution

Standardized Item Squared

Item Pattern/Structure Coefficients Uniqueness Multiple Correlation

Item 2 0.78 .39 .61

Item 7 0.87 .25 .75
Item 8 0.90 .20 .80
Item 9 0.89 .21 .79
Item 10 0.90 .19 .81

Note. All parameters were statistically significant at p < .05.

variances, and item uniquenesses between samples did not significantly

decrease model fit. Thus, the model demonstrated tight cross validity. Table 4
presents the common metric completely standardized solution for this five-
item model of belonging scores. Coefficient alpha for this model was esti-
mated at .96 in the cross-validation sample.

Gender Invariance

For females, Mardia’s (1970, 1974) coefficients (normalized values) for

multivariate skewness and kurtosis were estimated to be 20.02 and 12.11,
respectively. In the male sample, these estimates were 17.65 and 11.17,
respectively. Again, these values are statistically significant, but it is known
that large samples positively bias Mardia’s coefficient (Bollen, 1989;
Mardia, 1970, 1974). These values are comparable to those found in other
research using similar parameter estimation techniques, so they are reported
primarily for descriptive purposes (Boomsma, 2000). As seen in Table 5, the
five-item model fit data from separate samples of females and males well.
The covariance matrices for females and males were equivalent as was the
factor structure. As indicated by the chi-square difference tests, additional
equality constraints on item pattern/structure coefficients and the factor vari-
ance did not significantly decrease model fit. Model fit did decrease signifi-
cantly when uniquenesses were constrained to invariance across samples.
Thus, the five-item belonging model demonstrated gender invariance at the
levels of item pattern/structure coefficients and factor variance but not at the
level of uniquenesses. This finding satisfied the minimal criterion for con-
cluding that a measure demonstrates invariance (Bollen, 1989).

Predictive Validity

Predictive validity was examined by exploring the relationship between

scores on the five-item Belonging scale and self-reported attendance in the
 Table 5
Fit Indices for Gender Invariance Analyses

Model df χ2 RMSEA (90% CI) SRMR GFI NNFI CFI χ2 Difference df Difference p

Females 5 15.75 .10 (.05 – .17) .02 .97 .97 .99

Males 5 22.71 .14 (.08 – .19) .02 .95 .96 .98
Equal covariance matrices 15 33.30 .08 (.04 – .11) .02 .97 .99 .99
Fixed factor structure 10 38.46 .12 (.08 – .16) .02 .95 .97 .98
Fixed structure and item
pattern/structure coefficients 14 39.15 .10 (.06 – .13) .02 .95 .98 .99
Fixed structure, item pattern/
structure coefficients and
factor variance 15 39.15 .09 (.06 – .13) .02 .95 .98 .99
Fixed structure, item pattern/
structure coefficients, factor
variance, and uniquenesses 20 57.53 .10 (.07 – .13) .02 .94 .98 .98
Model comparisons
Item pattern/structure
coefficients 0.69 4 > .05
Factor variance—item
pattern/structure coefficients 0.00 1 > .05
Uniquenesses—factor variance 18.38 5 < .05

Note. RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean residual; GFI = goodness-of-fit index; NNFI = nonnormed fit index; CFI
= comparative fit index.

869
870 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

past 7 days (Md = 4.00, Mo = 1.00, SD = 2.16); 6-month attendance (M =

22.55, SD = 30.80, community rewards for prosocial involvement (M = 2.51,
SD = .93), community opportunities for prosocial involvement (M = 2.74,
SD = .78), neighborhood attachment (M = 2.01, SD = .90), community disor-
ganization (M = 1.70, SD= .68), and laws and norms favorable to drug use
(M = 1.74, SD = .68).
Belonging scores on the five-item measure were strongly related to self-
reported 7-day attendance, Spearman’s ρ(272) = .59, p < .01, and moderately
related to actual 6-month attendance, r(277) = .37, p < .01.4 The Spearman
rank correlation is a special case of the Pearson product-moment correlation
(Cohen & Cohen, 1983, p. 40), so it was possible to test the hypothesis that
their difference was zero using a test for differences in dependent correlations
(Cohen & Cohen, 1983, pp. 56-57), t(269) = 5.02, p < .01. Scores on the
Belonging scale also were significantly correlated with various community
risk and protective factors. Specifically, belonging was associated with
increased protection as demonstrated by its positive association with
community rewards for prosocial involvement, r(275) = .18, p < .01, and
neighborhood attachment, r(276) = .24, p < .01. Belonging was related to
decreased risk as demonstrated by its negative relation to community disor-
ganization, r(275) = –.31, p < .01, and laws and norms favorable to drugs use,
r(275) = –.32, p < .01.

Discussion
Five items were removed from the original 10-item Belonging scale in the
course of model modifications. The first item to be removed (Item 5) tapped
individuals’ fantasies (i.e., wishes) instead of their perceptions of reality like
the nine other items. Item 5 and the next two items to be removed (i.e., Items 1
and 6) also were negatively worded. Previous research established that nega-
tive wording can introduce extraneous variance that detracts from the fit of
confirmatory factor analysis models (Marsh, 1996; Motl, Conroy, & Horan,
2000; Tomàs & Oliver, 1999). The minimal relationship of these three items
to the latent factor suggested that these items were not measuring the latent
Belonging factor. Mixing items with small and large SMCs can distort the
location of centroids for latent variables in multivariate space (Little, Linden-
berger, & Nesselroade, 1999), so these three items were removed.
Items 1, 3, and 6 tapped relationships with peers and program leaders.
Although these variables may influence one’s participation, they were incon-
sistent with other scale items that focused on perceptions of self. The last item
to be removed, Item 4, asked participants if they felt like “an important mem-
ber of the program.” Perceptions of importance or value to the program may
be different from a sense of belonging. In fact, it is quite likely that children
 ANDERSON-BUTCHER AND CONROY 871

and youth can feel like they belong in a setting without believing that they are
necessarily an important part of that setting. Perceptions of importance also
are relative judgments that may depend, to an extent, on the sources of
comparison.
The remaining five items all tapped a common domain involving security
(i.e., support, acceptance, comfort) and membership (being part of and com-
mitted to the program) aspects of youths’perceptions related to their program
involvement. This Belonging scale does not sample feelings related to enjoy-
ment and affect related to participation (like the satisfaction domain would by
itself), nor does the Belonging scale tap interest levels (i.e., intrinsic motiva-
tion) or the amount of effort displayed in preparation for or within the pro-
gram (like the commitment and engagement domains would by themselves).
Furthermore, this Belonging scale does not explore youths’ relationships
with key figures in the activity setting (e.g., peers, program leaders).
The resulting measure of belonging between an individual and an organi-
zation bears some similarity to the individual difference construct of attach-
ment security. Attachment security refers to an individual’s belief that the
world is a safe place to explore and that caregivers will be available and
responsive to the individual’s needs (Ainsworth et al., 1978; Bowlby, 1982).
Belonging is similar because it represents a feeling of safety and acceptance
within a program.
Findings also revealed that belonging scores had a weak albeit positive
relationship with protective factors and a moderate negative relationship with
risk factors. As hypothesized, belonging to the youth development program
was positively related to community opportunities and rewards for prosocial
involvement and negatively related to low neighborhood attachment, high
community disorganization, and laws and norms favorable to drug use. Youth
missing a strong sense of belonging to a youth development organization are
at greater risk for poor social outcomes than youth with a strong sense of
belonging. This finding is consistent with previous assertions that belonging
has a positive influence on youth development (Anderson-Butcher & Fink,
2001; Blum & Rinehart, 1998; Resnick et al., 1997).

Future Research

Future research should focus on gathering evidence to evaluate the con-

vergent validity of belonging scores. From a theoretical perspective, belong-
ing scores should be positively related to youths’enjoyment of a program and
its activities, availability of and time spent in programming, intrinsic motiva-
tion for program activities, friendships with peers at the program, relation-
ships with program leaders, sense of community, and effort in participation.
Belonging scores should be strongly related to scores for commitment to,
872 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

engagement with, and satisfaction with the program. It will be important to

evaluate whether belonging scores and scores for these other variables are
collinear.5 Extending predictive validity evidence to establish the relation-
ship between belonging and behavioral outcomes such as gang involvement,
alcohol and drug use, and future involvement in program activities (i.e.,
retention and dropout) also will be important.
The different effect sizes in belonging score correlations with self-
reported past-week attendance and actual 6-month attendance during the pre-
vious 6 months suggests that it will be important to establish the stability of
belonging scores over time. Youths’sense of belonging may be very sensitive
to recent attendance or social influences at the program (a recency hypothe-
sis). Support for this hypothesis would indicate that the social environment
should be more completely considered when assessing belonging. For exam-
ple, dynamic friendship networks among children and youth may confound
interpretations of belonging scores in this case.6 Alternatively, belonging
may distort the accuracy of self-reported attendance such that participants
who have a sense of belonging report attending more frequently than they
actually did (a perceptual distortion hypothesis). Support for the perceptual
distortion hypothesis would point to important methodological issues related
to the measurement of participation. Research should be designed to test
these (as well as other) plausible explanations for the different relationships
of belonging scores to the two attendance measures.

Youth Development Applications for a Measure of Belonging

The five-item Belonging scale evaluated in this study may be useful for
both researchers and practitioners in the areas of youth development. Belong-
ing scores may be useful for predicting attendance patterns, an important out-
come within youth programming. Furthermore, it may be especially useful
for identifying individuals who are not likely to attend or are likely to drop out
from the program. These individuals could then be targeted for enhanced
retention efforts.
The instrument also may be useful in guiding practice within youth devel-
opment agencies. The National Longitudinal Study on Adolescent Health
found that youth with strong connections to school displayed lower levels of
violence and emotional distress and had less frequent cigarette, alcohol, and
marijuana use (Blum & Rinehart, 1998; Resnick et al., 1997). We expect that
a sense of belonging to youth development agencies may have similar effects
because youth who feel supported and accepted at the program, believe they
are part of the program, are committed to the program, and feel comfortable
at the program are more likely to participate in the program. In fact, it could
be argued that many programs promote positive youth development by devel-
oping a sense of belonging that facilitates internalization of program values.
 ANDERSON-BUTCHER AND CONROY 873

Notes
1. A detailed review of research documenting these effects is presented elsewhere (Ander-
son-Butcher, 2000; Lawson & Anderson-Butcher, 2001).
2. The seven remaining items on B. Arthur’s (1997) sense of belonging subscale sampled
concepts unrelated to the definition of belonging adopted in this article (e.g., introversion, qual-
ity of friendships, and personal affect); these items were not included with the present belonging
instrument because they were inconsistent with the theory of belonging developed here and were
expected to introduce extraneous variance.
3. The variance-covariance matrices used in these analyses are available on request from the
first author.
4. The Spearman rank correlation coefficient between self-reported attendance in the past
week and actual attendance over the past 6 months (n = 300) was .62, p < .01.
5. We thank two anonymous reviewers for calling our attention to this important priority for
future research.
6. We thank the anonymous reviewer who offered this plausible rival hypothesis.

References
Ainsworth, M. D., Blehar, M. C., Waters, E., & Wall, S. (1978). Patterns of attachment: A psy-
chological study of the strange situation and at home. Hillsdale, NJ: Lawrence Erlbaum.
Anderson-Butcher, D. (2000). Exploring the relationships among participation in youth devel-
opment programs, risk and protective factors, and problem behaviors. Unpublished doctoral
dissertation, University of Utah, Salt Lake City.
Anderson-Butcher, D., & Fink, J. S. (2001). The importance of a sense of belonging to youth ser-
vice agencies: A risk and protective factor analysis. Manuscript submitted for publication.
Antonovsky, A. (1983). The sense of coherence: Development of a research instrument. News-
letter and Research Reports, 1, 1-11.
Arthur, B. (1997). Boys & Girls Club of East Valley’s member survey: Subscale on sense of be-
longing. Tempe: Arizona Prevention Research Center.
Arthur, M., Pollard, J., Hawkins, J., & Catalano, R. F. (1997). Student survey of risk and protec-
tive factors. Seattle, WA: Developmental Research and Programs.
Baumeister, R., & Leary, M. R. (1995). The need to belong: Desire for interpersonal attachments
as a fundamental human motivation. Psychological Bulletin, 117, 497-529.
Bentler, P. M. (1980). Multivariate analysis with latent variables: Causal modeling. Annual Re-
view of Psychology, 31, 419-456.
Bentler, P. M. (1990). Comparative fit indices in structural equation models. Psychological Bul-
letin, 107, 238-246.
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of
covariance structures. Psychological Bulletin, 88, 588-606.
Blum, R. W., & Rinehart, P. M. (1998). Reducing the risk: Connection that make a difference in
the lives of youth. Minneapolis: Division of General Pediatrics and Adolescent Health, Uni-
versity of Minnesota.
Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley.
Boomsma, A. (2000). Reporting analyses of covariance structures. Structural Equation
Modeling, 7, 461-483.
Bowlby, J. (1982). Attachment and loss: Vol. 1 Attachment. New York: Basic Books.
Boys & Girls Clubs of America. (1998). Youth development: The foundation for the future. At-
lanta, GA: Author.
Brown, B. B. (1988). The vital agenda for research on extracurricular influences: A reply to Hol-
land and Andre. Review of Educational Research, 58, 107-111.
874 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Burt, M. R., Resnick, G., & Novick, E. R. (1998). Building supportive communities for at-risk
adolescents: It takes more this. Washington, DC: American Psychological Association.
Catalano, R. F., & Hawkins, J. D. (1996). The social development model: A theory of antisocial
behavior. In J. D. Hawkins (Ed.), Delinquency and crime: Current theories. New York: Cam-
bridge University Press.
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behav-
ioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Drasgow, F., & Kanfer, R. (1985). Equivalence of psychological measurement in heterogeneous
populations. Journal of Applied Psychology, 70, 662-680.
Hawkins, J. D., & Catalano, R. F. (1995). Risk-focused prevention: Using the social development
strategy. Seattle, WA: Developmental Research and Programs.
Hawkins, J., Catalano, R., Kosterman, R., Abbott, R., & Hill, K. (1999). Preventing adolescent
health-risk behaviors by strengthening protection during childhood. Archives of Pediatric
Adolescent Medicine, 153, 226-234.
Heath, S. B. (1999). Dimensions of language development: Lessons from older children. In A. S.
Masten (Ed.), Cultural processes in child development: The Minnesota symposium on child
psychology (pp. 59-75). Mahwah, NJ: Lawrence Erlbaum.
Hirschi, T. (1969). Causes of delinquency. Berkeley: University of California Press.
Holland, A., & Andre, T. (1987). Participation in extracurricular activities in secondary school:
What is known, what needs to be known? Review of Educational Research, 57, 437-460.
Hoyle, R. H., & Panter, A. T. (1995). Writing about structural equation models. In R. H. Hoyle
(Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks,
CA: Sage.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
Jöreskog, K. G., & Sörbom, D. (1996). LISREL 8: User’s reference guide. Chicago: Scientific
Software International.
Larson, R. W. (2000). Toward a psychology of positive youth development. American Psycholo-
gist, 55, 170-183.
Lawson, H. A., & Anderson-Butcher, D. (2001). In the best interests of the child: Youth develop-
ment as a child welfare support and resources. In A. L. Sallee, H. A. Lawson, & K. Briar-
Lawson (Eds.), Innovative practices with vulnerable children and families. Dubuque, IA:
Eddie Bowers.
Lee, V. E., & Smith, J. B. (1993). Effects of school restructuring on the achievement and engage-
ment of middle-grade students. Sociology of Education, 66, 164-187.
Little, T. D., Lindenberger, U., & Nesselroade, J. R. (1999). On selecting indicators for
multivariate measurement and modeling with latent variables: When “good” indicators are
bad and “bad” indicators are good. Psychological Methods, 4, 192-211.
Loehlin, J. (1992). Latent variable models: An introduction to factor, path, and structural analy-
sis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
MacCallum, R., Rosnowski, M., Mar, C., & Reith, J. (1994). Alternative strategies for cross-val-
idation of covariance structure models. Multivariate Behavioral Research, 29, 1-32.
MacCallum, R., Rosnowski, M., & Necowitz, L. B. (1992). Model modifications in covariance
structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111,
490-504.
Mardia, K. V. (1970). Measures of multivariate skewness and kurtosis with application.
Biometrika, 57, 519-530.
Mardia, K. V. (1974). Applications of some measures of multivariate skewness and kurtosis in
testing normality and robustness studies. Sankhya B, 36, 115-128.
Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively-meaningful dis-
tinction or artifactors? Journal of Personality and Social Psychology, 70, 810-819.
 ANDERSON-BUTCHER AND CONROY 875

Marsh, H. W., Hau, K. T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number
of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research,
33, 181-220.
Martinek, T. (1999). Evaluation of Project Effort. Greensboro: University of North Carolina at
Greensboro.
McGee, R., Feehan, M., Williams, S., & Anderson, J. (1992). DSM-III disorders from age 11 to
age 15 years. Journal of the American Academy of Child and Adolescent Psychiatry, 31, 50-
59.
McLaughlin, M. W., Irby, M. A., & Langman, J. (1994). Urban sanctuaries: Neighborhood or-
ganizations in the lives and futures of inner-city youth. San Francisco: Jossey-Bass.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from per-
sons’ responses and performances as scientific inquiry into score meaning. American Psy-
chologist, 50, 741-749.
Moffitt, T. E., Caspi, A., Dickson, N., Silva, P., & Stanton, W. (1996). Childhood-onset versus
adolescent-onset antisocial conduct problems in males: Natural history from ages 3 to 18
years. Development and Psychopathology, 8, 399-424.
Motl, R. W., Conroy, D. E., & Horan, P. (2000). The Social Physique Anxiety Scale: An example
of the potential consequences of negatively-worded items. Journal of Applied Measurement,
1, 327-345.
National Association of Elementary School Principals. (1999). After-school programs & the K-8
principal: Standards for quality school-age child care. Alexandria, VA: Author.
O’Donnell, J., Michalak, E. A., & Ames, E. B. (1997). Inner-city youths helping children: After-
school programs to promote bonding and reduce risk. Social Work in Education, 19, 231-
241.
O’Donovan, A. (1996). Six state consortium for prevention needs assessment studies: Measure-
ment validation results and conclusions. Seattle: Social Development Research Group, Uni-
versity of Washington.
Parke, R. D., & Slaby, R. G. (1983). The development of aggression. In E. M. Hetherington
(Ed.), Handbook of child psychology: Vol. 4. Socialization, personality, and social develop-
ment (pp. 557-641). New York: John Wiley.
Resnick, M., Bearman, P. S., Blum, R. W., Bauman, K. E., Harris, K. M., Jones, J., et al. (1997).
Protecting adolescents from harm: Findings from the National Longitudinal Study on Ado-
lescent Health. Journal of the American Medical Association, 278, 823-834.
Rosenfeld, L. B., Richman, J. M., & Bowen, G. L. (2000). Social support networks and school
outcomes: The centrality of the teacher. Child and Adolescent Social Work Journal, 17(3),
205-226.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic mo-
tivation, social development, and well-being. American Psychologist, 55, 68-78.
Small, S. A. (1991). Teenage assessment project question bank. Madison: University of Wiscon-
sin–Madison, School of Family Resources and Consumer Services.
Social Research Institute. (1997). Utah Substance Abuse Prevention Needs Assessment Survey.
Salt Lake City, UT: Author.
Somers, M. D. (1999). Development and preliminary validation of a measure of belonging. Un-
published doctoral dissertation, Temple University, Philadelphia.
Sutherland, E. H., & Cressey, D. R. (1978). Principles of criminology. Philadelphia: J. B.
Lippincott.
Tajfel, H., & Turner, J. C. (1985). The social identity theory of intergroup behavior. In S. Worchel
& W. G. Austin (Eds.), Psychology of intergroup relations. Chicago: Nelson-Hall.
Tanaka, J. S. (1987). “How big is big enough?” Sample size and goodness of fit in structural
equation models with latent variables. Child Development, 58, 134-146.
Thompson, B. (1997). The importance of structure coefficients in structural equation modeling
confirmatory factor analysis. Educational and Psychological Measurement, 57, 5-19.
 876 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of
scores: A historical overview and some guidelines. Educational and Psychological Mea-
surement, 56, 197-208.
Tomàs, J. M., & Oliver, A. (1999). Rosenberg’s Self-Esteem Scale: Two factors or method ef-
fects. Structural Equation Modeling, 6, 84-98.
Turner, J. C. (1982). Towards a cognitive redefinition of the social group. In H. Tajfel (Ed.), Self
identity and intergroup relations. Cambridge, UK: Cambridge University Press.
Youniss, J., Yates, M., & Su, Y. (1997). Social integration: Community service and marijuana
use in high school seniors. Journal of Adolescent Research, 12, 245-262.

View publication stats